Loading Large File

[Don57]

I am trying to load 2 large files into an encryption program. (both larger than 1G). The second load fails because of lack of RAM. I need a way to load the source files, and write the output file in more manageable blocks. ::)

[Vortex]

Memory mapped files?

http://win32assembly.online.fr/tut13.html

[Don57]

Thank you. I missed that tutorial in my reading.  :P

[jj2007]

Memory mapped or not, you will probably need to stream the files in. The difficult part is often how to handle the end of the streaming window, i.e. avoiding or handling access violations.

[Don57]

I don't want to seem lazy, but i've been looking on the internet for file streaming and can't find a code example. It's been 15 years since I worked in assembler and I'm use to low level disk reads and writes. If you could point me in the right direction it would be greatly appreciiated. I live in the country and only get into the city for internet access a couple times a month, so I try to get enough info to keep me going. Thanks.

[qWord]

hi,
the question is, if your encryption function can handle the file block-wise or if a continued byte/word/...-stream is needed?

[Don57]

My encryption function is a simple xor. The key file is xor'ed over the source file and the source buffer is rewritten to the disk using a different name. It's a whole 7 lines of code, I should be able to nest it anywhere.

[clive]

Well presuming that your source file, key file and destination file are all the same size, is there a strong reason to hold them in memory completely at all? The use of the data seems quite temporal, in DOS or Embedded circles where you couldn't possibly hold all/any of the files totally in memory, you learn to split things into usable chunks. 32K fits well with a lot of media blocking sizes (sectors, clusters, pages, etc)

A quick, off-the-cuff, pseudo code would look something like this

BufferSize = 32KB
Allocate : SourceBuffer[BufferSize]
Allocate : KeyBuffer[BufferSize]
CreateFile : Source, READ
CreateFile : Key, READ
CreateFile : Destination, WRITE
GetFileSize : Source
while(SourceSize)
{
  BlockSize = min(SourceSize, BufferSize)
  ReadFile : Source, SourceBuffer, BlockSize
  ReadFile : Key, KeyBuffer, BlockSize
  XORBlock : SourceBuffer, KeyBuffer, BlockSize
  WriteFile : Destination, SourceBuffer, BlockSize
  SourceSize -= BlockSize ; Reduce remaining data by the amount consumed, repeat until all consumed
}
CloseHandle : Destination
CloseHandle : Source
CloseHandle : Key
Free : SourceBuffer
Free : KeyBuffer

[qWord]

hi,
in the attachment an example using a file mapping.
(key.txt and data.txt can have any size)

[Don57]

Thank you everybody for your help I'm just finishing the code and it came out looking alot like Clive's.  :cheekygreen:

[Don57]

Well here's the first writting of the code. It writes to the buffers using a bottom checking loop, and EOF is no problem, but it will not load the memory map with the next 64M
chunk of code. I get err 0005 access denied. I tried flushing the buffers but that didn't work. Any help would be appreciated.

[dedndave]

that usually occurs when you try to write to an illegal memory address
for me, it is quite often address 00000000   :P

it is hard to see by looking at the proc, alone
it would be easier to find with the test code and files

however, that may not be necessary
when the program crashes into Dr Watson, it may tell you the address of your problem

[Don57]

Probably didn't explain myself well enough. The problem is even after I unmap the data I cannot map the next 64M chunk of data to the map created by the CreateFile Map call.
64M (04000000h) is added to  dwFileViewStartLow every iteration untill EOF.

The program file contains bitmapped graphics and therefore excedes the max file size for posts

[dedndave]

my statements still apply   :bg

you can create a listing of the program by adding the /Fl switch to the MASM commandline
then, get the address of the error from the crash report
it may help you troubleshoot the problem

[Don57]

I'll give it a shot. Thanks