memory mapped files , what if i need to map 100 files?

[supercoollee]

in Iczelion's tutorial, he taught the way to read file into memory. but he only mentioned the case that 1 file was read into memory.

what if i need 100 files read into memory?
should each file has a "hFileRead HANDLE" "pMemory DWORD" "hMapFile HANDLE"? which makes 300 dwords in memory ?
can i use these 3 dwords to read all the files?
like this:
read file1 , get handles(3 dwords) , get memory address for the content of file1 , free handles
read file2 , get handles(3 dwords) , get memory address for the content of file2 , free handles
read file3 , get handles(3 dwords) , get memory address for the content of file3 , free handles
......

in this way i save 200 dwords of memory space. can i ?
PS: i only need to read files, i don't write to them.

[ecube]

icezlions tut on memory mapped files isn't the best cause he has you map  a view of the entire thing at once which isn't practical for very large files. a cache of the file is kept in mem too, so lots of files being mapped would clog up your systems memory even after your program is closed ;\, here's a much more intelligent approach, this memory maps files in typically 64kilobyte parts and it disables caching, you can work with files up 18 exabytes in size(the 64bit hardware limit) as a result, and is great for multiple files.

this function calls a callback of your choice with the current memory it has a view mapped off and its size

Code Select Expand


include masm32rt.inc

MyCallBack proto :DWORD,:DWORD
MapLargeFile proto :DWORD,:DWORD

.data
thefile db "yourfile.txt",0

.code
start:
invoke MapLargeFile,addr thefile,addr MyCallBack
invoke ExitProcess,0

MapLargeFile proc iFile:DWORD,iFunc:DWORD
LOCAL bAlign:SYSTEM_INFO
LOCAL hFile:DWORD
LOCAL pbFile:DWORD
LOCAL iMapFile:DWORD
LOCAL qwFileSize:LARGE_INTEGER
LOCAL qwFileOffset:LARGE_INTEGER
LOCAL dwBytesInBlock:DWORD
invoke GetSystemInfo,addr bAlign

mov qwFileSize.LowPart,0
mov qwFileSize.HighPart,0

mov qwFileOffset.LowPart,0
mov qwFileOffset.HighPart,0

invoke CreateFile,iFile,GENERIC_READ,FILE_SHARE_READ,NULL,OPEN_EXISTING,FILE_ATTRIBUTE_ARCHIVE or FILE_FLAG_NO_BUFFERING or FILE_FLAG_OVERLAPPED,0
cmp eax,INVALID_HANDLE_VALUE
je @Error
mov hFile,eax

invoke CreateFileMapping,hFile,NULL,PAGE_READONLY,0,0,NULL
mov iMapFile,eax


invoke GetFileSize,hFile, addr qwFileSize.HighPart
mov qwFileSize.LowPart,eax

.while TRUE
                ;if the file is larger than AllocationGranularity(usually 64kb) then use AllocationGranuarity to start
                ;otherwise use the file size
mov eax,qwFileSize.HighPart
        .if eax == 0
            mov eax, qwFileSize.LowPart
            mov edx, bAlign.dwAllocationGranularity
            .if eax < edx
                mov dwBytesInBlock,eax
            .else
                mov dwBytesInBlock,edx
            .endif
        .else
            mov edx,bAlign.dwAllocationGranularity
            mov dwBytesInBlock,edx
        .endif
       
        invoke MapViewOfFile,iMapFile,FILE_MAP_READ,qwFileOffset.HighPart,qwFileOffset.LowPart,dwBytesInBlock
        mov pbFile,eax
       
push dwBytesInBlock
push pbFile
call iFunc

;we unmap the current view and mapview the next AllocationGranularity(usually 64kbs)
    invoke UnmapViewOfFile,pbFile
   
   ;Increment Offset quadword
        mov ecx,qwFileOffset.LowPart
        mov edx,qwFileOffset.HighPart
mov eax,dwBytesInBlock
mov ebx,0
add ecx,eax
adc edx,ebx
        mov qwFileOffset.LowPart,ecx
        mov qwFileOffset.HighPart,edx
       
        ;Decrement Size quadword
        mov ecx,qwFileSize.LowPart
        mov edx,qwFileSize.HighPart
        mov eax,dwBytesInBlock
        mov ebx,0
        sub ecx,eax
        sbb edx,ebx
        mov qwFileSize.LowPart,ecx
        mov qwFileSize.HighPart,edx
       
        .if edx == 0
            .if ecx == 0
                .break
            .endif
        .endif
.endw
invoke CloseHandle,iMapFile
invoke CloseHandle,hFile

@Error:
ret
MapLargeFile endp

MyCallBack proc iMem:DWORD,iLen:DWORD
;do whatever with mem
ret
MyCallBack endp
end start


[ecube]

keep in mind memory mapped files may not be the best choice here either, for top to bottom reading, ReadFile is much faster, for random access memory map is better, a big problem with memory mapped files too is they use aton of CPU.

[Tedd]

Unless you need all 100 files open at once, you can just open them one at a time, and re-use the variables for holding the handles.

I wouldn't recommend using memory-mapping unless the files are potentially quite large (at least a few MB) and you require random access. If they're not too big and you just need to read them start to end, allocate a block of memory (GlobalAlloc, or HeapAlloc), ReadFile into that, do your stuff, and then re-use the memory for the next file.

[supercoollee]

i need to read all files at once, and close them all at once when the program ends. and the files are not partially read, always in a whole. the file sizes may vary from xxKB to xxMB.
then which function is better?

[supercoollee]

icezlions tut on memory mapped files isn't the best cause he has you map  a view of the entire thing at once which isn't practical for very large files. a cache of the file is kept in mem too, so lots of files being mapped would clog up your systems memory even after your program is closed ;\, here's a much more intelligent approach, this memory maps files in typically 64kilobyte parts and it disables caching, you can work with files up 18 exabytes in size(the 64bit hardware limit) as a result, and is great for multiple files.

this function calls a callback of your choice with the current memory it has a view mapped off and its size

Code Select Expand


include masm32rt.inc

MyCallBack proto :DWORD,:DWORD
MapLargeFile proto :DWORD,:DWORD
...
...
MyCallBack endp
end start


my program is quite hard coded when it comes to loading files into memory, file names are all set, load modes are all set (it's a game requiring textures, sounds, etc) .
i have a hard time understanding these codes.  :eek  and i like simple codes, since flexibility is not required.

[supercoollee]

i just got to know about HeapAlloc() and Fopen() , and for OPENGL texture , there is AuxDIBImageLoad() .
there are so many ways to open a file or load file into memory, which is best?

[oex]

Easiest is probaly to use the InputFile Macro on smaller files

mov eax, InputFile(chr$("c:\test.txt"))

(ecx contains File Length)

[Tedd]

For textures, you don't need the files to be open (and it only takes up extra memory) - you open them, load them into memory, extract/convert the data, close the file, and use the data to create the texture.

You can use LoadBitmap, and then GetDIBits to get the data, then you can delete the bmp returned by LoadBitmap, and return the pixel data. The pixel data can then be used directly with glTexImage2D.

Also, to avoid loading so many texture files, you should put related textures into the same image (separate them by two pixels) and fit the texture co-ordinates accordingly. And if your game has any notion of levels/sections, only load the ones for the current section at the beginning of that section - that should keep memory usage and load time down (which also means you can use bigger textures for each.)

[supercoollee]

there are so many ways to do one thing so i would stay with one way for now, then try other ways later on.
let's say now i only need 1 TGA with size of 128x128, format 32bit, BGRA , and there is only 1 level in my game (i have just started the project)
i have successfully loaded all data of the TGA file into memory. say address = 0x07000000-0x07010030 ,  pMapFile dd   contains the address 0x07000000 . the data in memory are like these:
x x x x x x x x x x x x x x x x x x 00 00 ff ff 00 ff 00 ff ff 00 00 ff ...... (header 18 bytes, color of each pixel follows)

i am following NEHE's OGL tutorial:
http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=06  (loading bmp image)
http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=24  (loading TGA image)
i tried to convert the codes into ASM (omitting some functions such as: test the file exist or not, test the format of image, etc). but the outcome is wrong.

my code for creating GL texture:
=============================
loadtexture proc
    invoke  glGenTextures,1,gltexture[0]
    invoke  glBindTexture,GL_TEXTURE_2D,gltexture[0]
    invoke  glTexParameteri,GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_LINEAR
    invoke  glTexParameteri,GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_LINEAR
    mov eax,pMapFile
    add eax,18
    invoke  glTexImage2D,GL_TEXTURE_2D,0,4,128,128,0,GL_BGRA_EXT,GL_UNSIGNED_BYTE,eax
    ret
loadtexture endp
=============================
drawscene  proc
...
    invoke  glBindTexture,GL_TEXTURE_2D,gltexture[0]
    invoke  glBegin,GL_QUADS
    invoke  glTexCoord2i,0,0
    invoke  glVertex3f,0bf800000h,0bf800000h,3f800000h  ; ---- note that 0bf800000h means float value -1.0
    invoke  glTexCoord2i,0,127
    invoke  glVertex3f,3f800000h,0bf800000h,3f800000h
    invoke  glTexCoord2i,127,127
    invoke  glVertex3f,3f800000h,3f800000h,3f800000h
    invoke  glTexCoord2i,127,0
    invoke  glVertex3f,0bf800000h,3f800000h,3f800000h
    invoke  glEnd
...
drawscene endp
=============================
what is wrong ?

i don't understand the purpose of:
glGenTexture(1,&texture[0].texID)
glBindTexture(GL_TEXTURE_2D,texture[0].texID)
glTexImage2D(...... , texture[0].imagedata)

what are " &texture[0].texID" "texture[0].texID" "texture[0].imagedata"
are they memory addresses? or just ID numbers?
how do they relate to the memory address 0x07000000 ?

in my program, the polygon only shows red/black strips , while the TGA contains red/green/blue colors.