Hi.
There are times when i have no idea how much memory might be needed . - Is there any way to
handle this kind of thing ? how is this situation usuallly handled ? - for example the buffer I've allocated might get used up & i may need more.
can i extend the memory of a buffer in between, whenever i need to after allocating a fixed amount already in the beginning ?
Thanks
-
Basically YES but you need to execise some caution. You need to set up the emory allocation result in a variable (pointer) and each memory reference must use that pointer variable to write data to the allocated memory. If you exercise this little bit of caution you can allocate a small amount then repeatedly REallocate it larger as you need more.
Most of the API based memory allocation strategies have matching functions ALLOC, REALLOC and FREE so you can do things like this.
The reason why you need to maintain a pointer to the start of the memory block is that when you reallocate memory to a larger size it usually changes the start address as well.
you allocate a buffer like this:
invoke CoTaskMemAlloc,BufferSize
.if eax==NULL
;handle error
.else
mov pBuffer,eax
;do what do you want with this buffer
..
...
....
;now you need new bigger buffer
invoke CoTaskMemAlloc,BiggerBufferSize
.if eax==NULL
;do error handling
.else
mov pNewBuffer,eax
;copy old buffer to new buffer
push esx
mov esi,pBuffer
mov edx,pNewBuffer
mov ecx,BufferSize
.while ecx!=0
mov al,[esi]
mov [edx],al
dec ecx
inc esi
inc edx
.endw
pop esi
invoke CoTaskMemFree,pBuffer
mov eax,pNewBuffer
mov pBuffer,eax
;do you what do you want
..
....
.........
.endif
the above code need many optimizations
i hope this helps you
Here is a complete sample on how doing that
http://www.masm32.com/board/index.php?topic=2253.msg17776#msg17776
osama, that example was very helpfull.
I need to do some more reading on the memory stuff. I reinstalled my OS & have to
install the Win SDK again.
thx for the tips hutch.
toutenMasm, thanks for that sample. - ..I think I'll understand the sample more
if i do so some reading from the SDK side by side with it.
I'll reinstall the win-api SDK today & do some reading from it.
If i have more questions as i go on, I'll post them in this thread.
..appreciate all the help..!
Rainstorm
-
what's the usual way to check if a memory buffer is full ?
thanks!
Quote from: Rainstorm on January 03, 2008, 07:26:13 PM
what's the usual way to check if a memory buffer is full ?
thanks!
There is no way to test a memory buffer to tell if it is full, you must keep track of how much data is moved into the buffer and judge how much of it is still available. Normally you would do this by keeping track with a pointer to the current top of data. For example if you allocate a buffer you would create at least 2 pointers, one for the base of the buffer and one for the current top of data...
invoke VirtualAlloc,NULL,[cbBuffer],MEM_COMMIT,PAGE_READWRITE
mov [pBase], eax
mov [pTop], eax
// Move some data
invoke CopySomeDataFunc, [pTop], offset Data, [cbData]
mov eax, [cbData]
add [pTop], eax
// Calculate free space, free space in EAX
mov eax, [pTop]
sub eax, [pBase]
sub eax, [cbBuffer]Donkey
Donkey, I kinda got the idea....that was very helpful.
just have a few more questions & I'll post my code in the next post.
- what does 'cb' mean actually ? (like in [cbData] etc)
- when the buffer is initialised to '0' in the Alloc functions with HEAP_ZERO_MEMORY is that
the ASCII-null 0 or the ASCII(48) '0' ?
- on 32 bit windows the memory pages are 4K each so does that mean,..the smallest amount
actually allocated is 4Kbytes ? - like even if I allocate 256bytes,.. 4Kilobytes is still allocated ?
Thankyou.
-
Quote from: Rainstorm on January 04, 2008, 08:35:51 AM
Donkey, I kinda got the idea....that was very helpful.
just have a few more questions & I'll post my code in the next post.
- what does 'cb' mean actually ? (like in [cbData] etc)
- when the buffer is initialised to '0' in the Alloc functions with HEAP_ZERO_MEMORY is that
the ASCII-null 0 or the ASCII(48) '0' ?
- on 32 bit windows the memory pages are 4K each so does that mean,..the smallest amount
actually allocated is 4Kbytes ? - like even if I allocate 256bytes,.. 4Kilobytes is still allocated ?
Thankyou.
-
- cb is just a naming convention I use, and Microsoft also, it means count bytes
- Initialization to zero is to 0 (or NULL if you prefer), why would it mean initialization to 48 ?
- Yes, when using VirtualAlloc the size is always rounded up to the next page boundary, also the start address if specified is rounded down to the nearest boundary. So, if you have a buffer of 4097 bytes, 8192 bytes will be allocated since the last byte crosses to a second page. A memory page is normally 4KB, you should do a check system granularity before you assume that, it is subject to change. If you are allocating large buffers you should always use multiples of system granularity. You can obtain the page size using...
LOCAL si:SYSTEM_INFO
invoke GetSystemInfo, offset si
mov eax, [si.dwPageSize]Donkey
Hi,
here is the code where i've used HeapAlloc to allocate & reallocate memory.
It searches for lines with a character count equal to & above the requested amount, stores the references to them (line numbers & how many chars each line contains) in 2 buffers & then displays the results.
Donkey, thanks for clearing those earlier questions..
- It seems to assemble & work okay..not tested it much.., however with a high number of matches (around 550 something like that ?) windows terminates it, dunno why this happens.. or if the console can't handle all the output.
- have a few more questions but will ask them later.
..would appreciate any feedback.. & also if there are any mistakes in it.
Thanks.
.const
HEAP_CREATE_ENABLE_EXECUTE equ 262144
.data?
.data
pfname dd 0
flen dd 0
fpattern db "text files",0,"*.txt",0,0
total_characters_in_line dd 0
pfbuff dd 0
line_char_limit dd 0
pline_char_limit dd 0
hInstance dd 0
hHeap_line dd 0
hHeap_linecharcount dd 0
p_linecount_buff dd 0
p_linecharcount_buff dd 0
ptop_linecount_buff dd 0
ptop_linecharcount_buff dd 0
linebuffer_usedbytes dd 0
charbuffer_usedbytes dd 0
linebuff_size dd 0
line_charcountbuff_size dd 0
total_matchinglines_found dd 0
total_lines dd 0
temp dd 0
; -----------------------------------------------------------------------
.code
start:
invoke GetModuleHandle, NULL ; get the handle of this program
mov hInstance, eax
mov pfname, OpenFileDlg(NULL,hInstance,"Select File",addr fpattern)
cmp BYTE PTR [eax], 0
je exit_
invoke read_disk_file,pfname,addr pfbuff, addr flen
cmp eax, 0
je exit_
print "returned file length - "
print ustr$(flen),13,10
; ------- // Input the character limit // ---------
mov pline_char_limit, input("enter the max character count of line",58)
mov line_char_limit, uval(pline_char_limit)
print "pause.. - character Limit - : "
print ustr$(line_char_limit),13,10
; mov eax, pfbuff
; mov byte ptr [eax+flen-1], 0 ; add a NULL at the eof in the buffer
; ------ // Create the memory heap & Allocate Memory // --------
; ----------------------------------
; create buffer for line numbers
; ----------------------------------
invoke HeapCreate, HEAP_CREATE_ENABLE_EXECUTE or HEAP_NO_SERIALIZE, 0, 0
cmp eax, 0
je heap_creation_error
mov hHeap_line, eax ; handle to memory object
invoke HeapAlloc, hHeap_line, HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY, 512
cmp eax, 0
je heap_allocation_error
mov p_linecount_buff, eax ; pointer to allocated block
mov ptop_linecount_buff, eax
invoke HeapSize, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff
cmp eax, 4294967295 ; test if the size function succeeded
je heap_size_error
mov linebuff_size, eax ; store the size
; ----------------------------------------------
; create buffer for the character count of the lines
; ----------------------------------------------
invoke HeapCreate, HEAP_CREATE_ENABLE_EXECUTE or HEAP_NO_SERIALIZE, 0, 0
cmp eax, 0
je heap_creation_error
mov hHeap_linecharcount, eax ; handle to memory object
invoke HeapAlloc, hHeap_linecharcount, HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY, 512
cmp eax, 0
je heap_allocation_error
mov p_linecharcount_buff, eax ; pointer to allocated block
mov ptop_linecharcount_buff, eax
invoke HeapSize, hHeap_linecharcount, HEAP_NO_SERIALIZE, p_linecharcount_buff
cmp eax, 4294967295
je heap_size_error
mov line_charcountbuff_size, eax
; -----------------------------------------------
mov esi, pfbuff ; move address of the file buffer into esi
xor ecx, ecx
xor ebx, ebx
jmp count_characters
pre_count:
xor ecx, ecx ; reset count of chars in the line
add esi, 1
count_characters:
cmp byte ptr [esi], 0
je display_results
cmp byte ptr [esi], 10
je newline
inc esi ; pointer to file buffer
add ecx, 1 ; count of characters in the line
jmp count_characters
newline:
; add ecx, 1 ; would count the 'LF' too
add total_lines, 1 ; total line count
mov total_characters_in_line, ecx
cmp ecx, line_char_limit ; cmp the total chars in the line with the requested limit
jb pre_count
add total_matchinglines_found, 1
comment * - - - - - - - - - - - - - - - - -
else check to see if the memory buffer is full
if so, reallocate more memory
else add the character count & line number to the buffer
- - - - - - - - - - - - - - - - - - *
; ---- / check if there's enough space in the buffers / -----
mov eax, ptop_linecount_buff
sub eax, p_linecount_buff ; subtract base addr from top addr
mov linebuffer_usedbytes, eax
mov edx, linebuff_size
sub edx, 16 ; check at 16 bytes ahead of time
cmp edx, eax ; cmp no. of used bytes with the buffer size
jbe reallocate_linecount_buff
call store_linenumber
check_line_charcount_buffer:
mov eax, ptop_linecharcount_buff
sub eax, p_linecharcount_buff
mov charbuffer_usedbytes, eax
mov edx, line_charcountbuff_size
sub edx, 16 ; reallocate 16 bytes ahead of time
cmp edx, eax
jbe reallocate_charcount_buff
call store_line_charcount
jmp pre_count
; ---- / add the line number & character count of the line to the buffers / -----
store_linenumber:
mov eax, total_lines ; the line number
mov edx, ptop_linecount_buff
mov [edx], eax ; move the count of chars into the buffer
add ptop_linecount_buff, 4 ; track the top of the buffer
ret
store_line_charcount:
mov ecx, total_characters_in_line
mov edx, ptop_linecharcount_buff
mov [edx], ecx ; move the line count in the buffer
add ptop_linecharcount_buff, 4 ; track the top of the linecount buffer
ret
; ----------- // Reallocate Memory // ---------------
reallocate_linecount_buff:
invoke HeapReAlloc, hHeap_line,HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY,
p_linecount_buff, 256
cmp eax, 0
je heap_reallocation_error
mov p_linecount_buff, eax ; pointer to reallocated block
invoke HeapSize, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff
cmp eax, 4294967295 ; test if the size function succeeded
je heap_size_error
mov linebuff_size, eax
mov eax, linebuffer_usedbytes
add eax, p_linecount_buff
mov ptop_linecount_buff, eax
call store_linenumber
jmp check_line_charcount_buffer
;---------------------------------
reallocate_charcount_buff:
invoke HeapReAlloc, hHeap_linecharcount,HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY,
p_linecharcount_buff, 256
cmp eax, 0
je heap_reallocation_error
mov p_linecharcount_buff, eax ; pointer to reallocated block
invoke HeapSize, hHeap_linecharcount, HEAP_NO_SERIALIZE, p_linecharcount_buff
cmp eax, 4294967295 ; test if the size function succeeded
je heap_size_error
mov line_charcountbuff_size, eax ; store the size
mov eax, charbuffer_usedbytes
add eax, p_linecharcount_buff ; bring the pointer in the new block..
mov ptop_linecharcount_buff, eax ; to where it was in the data before..
call store_line_charcount ; reallocation.
jmp pre_count
;---------------------------------------------------------------------
; display the results
; ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
display_results:
print " ",13,10,
print "Filename : "
print pfname,13,10
print "Total Lines: "
print ustr$(total_lines),13,10
inkey
print " ",13,10,
print "Lines with a character count equal to or above "
print ustr$(line_char_limit)
print " = "
print ustr$(total_matchinglines_found),13,10
print "=============================================",13,10
mov edi, p_linecount_buff
mov edx, ptop_linecount_buff
mov dword ptr [edx+1], 0 ; add a 0 at the top of the buffer
; --// display the line numbers //
display_results1:
mov ebx, dword ptr [edi]
print ustr$(ebx)
print ", "
add edi, 4
cmp dword ptr [edi], 0
jne display_results1
print " ",13,10,
print "=============================================",13,10
print " ",13,10,
inkey
print " ",13,10,13,10
print "lines with corresponding character counts equal\
to or above the requested limit",13,10
print "=============================================",13,10
mov edi, p_linecount_buff
mov edx, ptop_linecount_buff ; add a 0 at the top of the buffer
mov dword ptr [edx+1], 0
mov esi, p_linecharcount_buff
mov edx, ptop_linecharcount_buff
mov dword ptr [edx+1], 0
comment * -------------------------------------------------------
Display the line numbers with the corresponding char
counts of the lines
------------------------------------------------------- *
display_results2:
mov ebx, [edi]
print ustr$(ebx)
print " - "
mov ebx, [esi]
print ustr$(ebx),13,10
add edi, 4
add esi, 4
cmp dword ptr [edi], 0
jne display_results2
print " ",13,10,
print "=============================================",13,10
print "finished",13,10,13,10
jmp free_memory
; ----------- // Errors //------------------------------------------
heap_creation_error:
print "memory error: memory heap couldn't be created",10,13
jmp exit_
heap_allocation_error:
print "memory error: memory couldn't be allocated",10,13
jmp exit_
heap_reallocation_error:
print "memory error: memory couldn't be reallocated",10,13
jmp free_memory
heap_size_error:
print "memoryerror: heap size error",10,13
jmp exit_
; ---------------------------------------------------------------------
free_memory:
free pfbuff
free pfname
invoke HeapFree, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff
cmp eax, 0
jne @F
print "HeapFree error !",13,10
@@:
invoke HeapFree, hHeap_linecharcount,HEAP_NO_SERIALIZE,p_linecharcount_buff
cmp eax, 0
jne @F
print "HeapFree error !",13,10
jmp exit_
@@:
print "Memory successfully freed",13,10
exit_:
inkey "press any key to exit..."
exit
end start
in the above code if i try to add a null at the end of the buffer into which the file is read, using invoke read_disk_file , .. I get the windows error of having to shut down the application..- why is that ?
I've commented the 2 lines out in the code in the previous post
mov eax, pfbuff
mov byte ptr [eax+flen+1], 0 ; add a NULL at the eof in the buffer
- If i specify the option HEAP_CREATE_ENABLE_EXECUTE while creating the heap & a processor doesn't support that option.. then are there any side effects ?\
- Suppose i free a heap or any other memory.. that hasn't been allocated in the first place.. what happens then, are there any effects ?
many thanks
Rainstorm
-
any one have any idea about or can help clarify the above statements ?
mov eax, [pfbuff]
add eax, [flen]
mov byte ptr [eax+1], 0 ; add a NULL at the eof in the buffer