Loading a file in memory 1 byte at a time...

[ThoughtCriminal]

I know how to use memory mapped files and CreateFile - Readfile.  Readfile has a field for how many bytes to read.  Reading one byte at a time sounds like a lot of calls to Readfile.  What I want to do is read in a file with a bytescanner looking for a #include to include antoher file.  Eventually have all files combined in memory in order.  Is there a better way or some sort of streaming method?

Thanks.

[donkey]

You can load the file in blocks and scan each block, say load a file 1 page at a time then scan the memory where you loaded it for #include.

But the best way to do it is with memory mapped files, you can quickly scan it for any byte sequence.

If speed is not too much of an issue I use a callback function to read a text file line by line...

ReadFileLines:
   Sequentially reads a file line by line and passes the NULL
   terminated strings to a callback routine
   Parameters:
      pszFile = Pointer to a fully qualified file name
      pCallback = Pointer to a callback function
      
      Format for callback:
      ReadFileLinesProc:
         pLineText = Pointer to a buffer containing the text
         cbLine = Count of characters in pLineText
         nLine = Line number in file
         pLine = Offset of the first character in the file
      The callback returns FALSE to stop the enumeration

   Returns 0, -1 in case of error. Use GetLastError for more information
   If the callback returns FALSE the enumeration is stopped and the function
   returns -1, GetLastError will return ERROR_CANCELLED

Code Select Expand

ReadFileLines FRAME pszFile, pCallback
uses edi,esi,ebx
LOCAL fSizeLo :D
LOCAL fSizeHi :D
LOCAL pBuffer :D
LOCAL hHeap :D
LOCAL LineCount :D
LOCAL StripSize :D
LOCAL hFile :D
LOCAL hFileMapping :D
LOCAL pFileMapping :D

invoke IsBadCodePtr,[pCallback]
or eax,eax
jz >
invoke SetLastError, 5 ;ERROR_ACCESS_DENIED
xor eax,eax
dec eax
ret
:

mov D[LineCount],0

invoke GetProcessHeap
mov [hHeap], eax

xor ebx,ebx
; Open and map the file
invoke CreateFileA,[pszFile],GENERIC_READ,FILE_SHARE_READ,ebx,OPEN_EXISTING,ebx,ebx
mov [hFile], eax
cmp eax,INVALID_HANDLE_VALUE
je >>.NOFILEHANDLE
invoke CreateFileMappingA,[hFile],ebx,PAGE_READONLY,ebx,ebx,ebx
mov [hFileMapping],eax
or eax,eax
jz >>.NOMAPPING
invoke MapViewOfFile,[hFileMapping],FILE_MAP_READ,ebx,ebx,ebx
mov [pFileMapping],eax
or eax,eax
jz >>.NOMAPVIEW
invoke GetFileSize,[hFile],offset fSizeHi
mov [fSizeLo],eax
or eax,eax
jz >>.ERROR

mov esi,[pFileMapping]
mov edi,[pFileMapping]
mov ebx,[fSizeLo]
mov ecx,ebx
mov eax,0Ah

L1:
repne scasb
push ecx,eax

mov eax,edi
sub eax,esi
push eax
add eax,0Fh
and eax,0FFFFFFF0h
invoke HeapAlloc,[hHeap],HEAP_ZERO_MEMORY,eax
mov [pBuffer],eax
pop ecx
sub ecx,2
jns >
mov ecx,0
:
inc D[LineCount]
invoke copymemory,eax,esi,ecx

sub esi,[pFileMapping]
invoke [pCallback],[pBuffer],ecx,[LineCount],esi
or eax,eax
jz >.EARLYEXIT

invoke HeapFree,[hHeap],0,[pBuffer]
mov esi,edi

pop eax,ecx
or ecx,ecx
jnz <L1

invoke UnmapViewOfFile,[pFileMapping]
invoke CloseHandle,[hFileMapping]
invoke CloseHandle,[hFile]
xor eax,eax
RET

.EARLYEXIT
invoke SetLastError,1223
invoke HeapFree,[hHeap],0,[pBuffer]

; ERROR returns
.ERROR
invoke UnmapViewOfFile,[pFileMapping]

.NOMAPVIEW
invoke CloseHandle,[hFileMapping]

.NOMAPPING
invoke CloseHandle,[hFile]

.NOFILEHANDLE
xor eax,eax
dec eax
RET

ENDF

copymemory FRAME lpDest,lpSource,nBytes
uses edi,esi,ecx
cld
mov edi,[lpDest]
mov esi,[lpSource]
mov ecx,[nBytes]

; do the evenly divisible ones
shr ecx,2
rep movsd

; do the remainder
mov ecx,[nBytes]
and ecx,3
rep movsb

RET
ENDF

[Tedd]

I suppose one way to achieve that would be to read in the first file (all of it; or memory map it) and then as you scan through it, write the contents to your new output file (or the memory it happens to be in) but each time you reach an 'include' line, recursively call this same function to read in that file (and output it in-place in the output file.) When you reach the end of a file, you simply return and it'll carry on with the rest of the previous file that included it.
You will have to keep track of which files you've already included though (not necessarily in the same file, but indirectly through others), otherwise you could end up looping forever :lol

[ThoughtCriminal]

Thanks. I guess the best what I want is to just include 1 file in another using a hint (ie. #include) to specify what file to include.