Anyone know how to read a file chunk by chunk?
ReadFile, GetFileSize and SetFilePointer all support reading files to 4GB and beyond (might be problems on Win9x due to file systems)
You can read the whole file in 32KB chunks.
-Clive
here an example using ReadFile:
.data?
align 8
dwRead LABEL DWORD
dqFileSize dq ?
cbRest dd ?
nBlocks dd ?
hFile dd ?
pBuffer dd ?
.code
BLOCK_SIZE EQU (1024*1024) ; 2^20
POW EQU 20
.if ASM(mov hFile,rv(CreateFile,"C:\test.file",GENERIC_READ,FILE_SHARE_READ,0,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0)) != INVALID_HANDLE_VALUE
invoke GetFileSizeEx,hFile,OFFSET dqFileSize
mov eax,DWORD ptr dqFileSize
mov edx,DWORD ptr dqFileSize+4
mov ecx,eax
shrd eax,edx,POW ; = x / BLOCK_SIZE
and ecx,BLOCK_SIZE-1 ; = x MOD BLOCK_SIZE
mov nBlocks,eax
mov cbRest,ecx
.if ASM(mov pBuffer,rv(GlobalAlloc,GPTR,BLOCK_SIZE))
.while nBlocks
.break .if !rv(ReadFile,hFile,pBuffer,BLOCK_SIZE,OFFSET dwRead,0)
; process blocks
dec nBlocks
.endw
.if dwRead && cbRest && rv(ReadFile,hFile,pBuffer,cbRest,OFFSET dwRead,0)
; process remaining bytes (n < BLOCK_SIZE)
.elseif cbRest || !dwRead
@@: invoke GlobalFree,pBuffer
jmp @F
.endif
invoke CloseHandle,hFile
invoke GlobalFree,pBuffer
.else
@@: invoke CloseHandle,hFile
jmp @F
.endif
.else
@@: ;error
.endif
EDIT: bug fixed
:U Thanks.
qWord thanks but I heard that was painfully slow, isn't mapped files a better approach? i'm abit weary to test myself for files that exceed the size of my ram( > 2 gigs) even though I don't think it should panic and cause problems.
Unless a file is only a couple of megabytes, you should probably code your application to only load the pieces you need and adapt to the memory on the system. What you don't directly hold in memory will be cached by the system. If you assume memory is finite, the application will run better on more systems. Generally you should assume you can't load the whole file, even if it will fit in virtual memory (page file), you have a better understanding of your file and it's structure, and are thus best placed to design/manage how the file/data are handled.
Large files often have a structure to permit navigation (archives, databases), or are streams of data (video, audio) where blocks are read/processed/discarded. Even video files tend to have some structure, or external indexes, to permit rapid scanning of frames or index points (fast forward/reverse).