Insert data into file

[Neil]

I have a binary data file which consists of a large number of 10 byte entries. My problem is how to insert a new entry into the middle of the file. The method I have come up with is to load the file into memory, find the insertion point, move the rest of the file 10 bytes forward then overwrite the data at the insertion point, then resave the larger updated file.

If anyone has any better ideas please suggest them. I don't want the code just the method.

[MichaelW]

How about read the file into memory, find the insertion point, write the data up to the insertion point to the output file, write the new data to the output file, then write the data after the insertion point to the output file. Without testing I don't know how the file operations would compare in speed to the move operation, but I think this should be at least a little faster than what you are proposing. For more speed you could try file mapping.

[Neil]

Thanks Michael, any suggestions are welcome & I'll look into them.

[Jimg]

If you're really brave, you can read in the file starting at the insertion point, write the new data directly to the input file followed by writing the old data.  Screw up and your original file is lost.

[Neil]

Jimg,
I don't think I'd try that one without plenty of backups, it's a thought though, might be worth playing around with.

[Mark Jones]

Question, is this data that will be modified frequently, or infrequently?  If it is holding something like mathematical calculations which are updated millions of times per second, then all that reading and writing would make short order of the hard disk.

If there are many update operations happening quickly, consider buffering the updates -- i.e., read the source data into buffer 1. Create one additional ram buffer (buffer 2) as large as the first, and one (buffer 3) twice the size of the first. When an insert is requested, save both the offset of the insert and its value into buffer 2 -- i.e., a DWORD offset and 10 bytes of data, ad nauseum. Continue to save these insert value requests into buffer 2 until either it nears being full, or after a set amount of time has passed. Then, read each entry in buffer 2 (DWORD offset, 10 bytes of DATA) and perform each operation while copying data from buffer 1 to buffer 3. When all of the copy operations are complete, save buffer 3 to disk, and make buffer 3 the new buffer 1. Resize buffer 3 (no need to clear it as it will just be overwritten anyways -- same with buffer 2.)

If the data grows quickly, it might be a good idea to every-once-in-a-while free all the buffers, delete their handles, and start the whole process anew. This is to eliminate memory fragmentation.

[Neil]

Thanks Mark,
I had to read your reply three or four times before I could get my head around what you were saying. :dazzled:
The file will be updated on an infrequent basis, perhaps once per session, but all your suggestions make interesting reading & give me food for thought. I haven't implemented any of them yet, I'm a Top Down sort of person & like to make plenty of notes before I embark on the actual code, I still make (silly) mistakes but not half as many if I didn't have A 'plan'.