Read file above 4 GB in size

[Farabi]

Anyone know how to read a file chunk by chunk?

[clive]

ReadFile, GetFileSize and SetFilePointer all support reading files to 4GB and beyond (might be problems on Win9x due to file systems)

You can read the whole file in 32KB chunks.

-Clive

[qWord]

here an example using ReadFile:

Code Select Expand

    .data?
        align 8
        dwRead      LABEL DWORD
        dqFileSize  dq ?
        cbRest      dd ?
        nBlocks     dd ?
        hFile       dd ?
        pBuffer     dd ?
    .code
    BLOCK_SIZE  EQU (1024*1024) ; 2^20
    POW         EQU 20

    .if ASM(mov hFile,rv(CreateFile,"C:\test.file",GENERIC_READ,FILE_SHARE_READ,0,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0)) != INVALID_HANDLE_VALUE
        invoke GetFileSizeEx,hFile,OFFSET dqFileSize
        mov eax,DWORD ptr dqFileSize
        mov edx,DWORD ptr dqFileSize+4
        mov ecx,eax
        shrd eax,edx,POW        ; = x  /  BLOCK_SIZE
        and ecx,BLOCK_SIZE-1    ; = x MOD BLOCK_SIZE
        mov nBlocks,eax
        mov cbRest,ecx     
        .if ASM(mov pBuffer,rv(GlobalAlloc,GPTR,BLOCK_SIZE))
            .while nBlocks
                .break .if !rv(ReadFile,hFile,pBuffer,BLOCK_SIZE,OFFSET dwRead,0)
               
                ; process blocks
               
                dec nBlocks
            .endw
            .if dwRead && cbRest && rv(ReadFile,hFile,pBuffer,cbRest,OFFSET dwRead,0)
               
                ; process remaining  bytes (n < BLOCK_SIZE)

            .elseif cbRest || !dwRead
@@:             invoke GlobalFree,pBuffer
                jmp @F
            .endif
            invoke CloseHandle,hFile
            invoke GlobalFree,pBuffer
        .else
@@:         invoke CloseHandle,hFile
            jmp @F
        .endif
    .else
@@:     ;error     
    .endif

EDIT:  bug fixed

[Farabi]

 :U Thanks.

[ecube]

qWord thanks but I heard that was painfully slow, isn't mapped files a better approach? i'm abit weary to test myself for files that exceed the size of my ram( > 2 gigs) even though I don't think it should panic and cause problems.

[clive]

Unless a file is only a couple of megabytes, you should probably code your application to only load the pieces you need and adapt to the memory on the system. What you don't directly hold in memory will be cached by the system. If you assume memory is finite, the application will run better on more systems. Generally you should assume you can't load the whole file, even if it will fit in virtual memory (page file), you have a better understanding of your file and it's structure, and are thus best placed to design/manage how the file/data are handled.

Large files often have a structure to permit navigation (archives, databases), or are streams of data (video, audio) where blocks are read/processed/discarded. Even video files tend to have some structure, or external indexes, to permit rapid scanning of frames or index points (fast forward/reverse).