any way to extend a memory buffer ?

[Rainstorm]

Hi.

There are times when i have no idea how much memory might be needed . - Is there any way to
handle this kind of thing ? how is this situation usuallly handled ? - for example the buffer I've allocated might get used up & i may need more.
can i extend the memory of a buffer in between, whenever i need to after allocating a fixed amount already in the beginning ?

Thanks

-

[hutch--]

Basically YES but you need to execise some caution. You need to set up the emory allocation result in a variable (pointer) and each memory reference must use that pointer variable to write data to the allocated memory. If you exercise this little bit of caution you can allocate a small amount then repeatedly REallocate it larger as you need more.

Most of the API based memory allocation strategies have matching functions ALLOC, REALLOC and FREE so you can do things like this.

The reason why you need to maintain a pointer to the start of the memory block is that when you reallocate memory to a larger size it usually changes the start address as well.

[ossama]

you allocate a buffer like this:

Code Select Expand


invoke CoTaskMemAlloc,BufferSize
.if eax==NULL
    ;handle error
.else
    mov pBuffer,eax
    ;do what do you want with this buffer
    ..
   ...
   ....
   ;now you need new bigger buffer
   invoke CoTaskMemAlloc,BiggerBufferSize
   .if eax==NULL
      ;do error handling
  .else
      mov pNewBuffer,eax
      ;copy old buffer to new buffer
                push esx
      mov esi,pBuffer
mov edx,pNewBuffer
mov ecx,BufferSize
.while ecx!=0
mov al,[esi]
mov [edx],al
dec ecx
inc esi
inc edx
.endw
                pop esi

   invoke CoTaskMemFree,pBuffer
   mov eax,pNewBuffer
   mov pBuffer,eax

     ;do you what do you want
..
....
.........
.endif


the above code need many optimizations
i hope this helps you

[ToutEnMasm]

Here is a complete sample on how doing that
http://www.masm32.com/board/index.php?topic=2253.msg17776#msg17776

[Rainstorm]

osama, that example was very helpfull.

I need to do some more reading on the memory stuff. I reinstalled my OS & have to
install the Win SDK again.

thx for the tips hutch.

toutenMasm, thanks for that sample. - ..I think I'll understand the sample more
if i do so some reading from the SDK side by side with it.
I'll reinstall the win-api SDK today & do some reading from it.
If i have more questions as i go on, I'll post them in this thread.

..appreciate all the help..!

Rainstorm

-

[Rainstorm]

what's the usual way to check if a memory buffer is full ?

thanks!

[donkey]

what's the usual way to check if a memory buffer is full ?

thanks!

There is no way to test a memory buffer to tell if it is full, you must keep track of how much data is moved into the buffer and judge how much of it is still available. Normally you would do this by keeping track with a pointer to the current top of data. For example if you allocate a buffer you would create at least 2 pointers, one for the base of the buffer and one for the current top of data...

Code Select Expand

invoke VirtualAlloc,NULL,[cbBuffer],MEM_COMMIT,PAGE_READWRITE
mov [pBase], eax
mov [pTop], eax

// Move some data
invoke CopySomeDataFunc, [pTop], offset Data, [cbData]
mov eax, [cbData]
add [pTop], eax

// Calculate free space, free space in EAX
mov eax, [pTop]
sub eax, [pBase]
sub eax, [cbBuffer]

Donkey

[Rainstorm]

Donkey, I kinda got the idea....that was very helpful.

just have a few more questions & I'll post my code in the next post.

-  what does 'cb' mean actually ?  (like in [cbData]  etc)
-  when the buffer is initialised to '0' in the Alloc functions with HEAP_ZERO_MEMORY is that
   the ASCII-null 0 or the ASCII(48)  '0'    ?
-  on 32 bit windows the memory pages are 4K each so does that mean,..the smallest amount
   actually allocated is 4Kbytes ? - like even if I allocate 256bytes,.. 4Kilobytes is still allocated ?

Thankyou.
-

[donkey]

Donkey, I kinda got the idea....that was very helpful.

just have a few more questions & I'll post my code in the next post.

-  what does 'cb' mean actually ?  (like in [cbData]  etc)
-  when the buffer is initialised to '0' in the Alloc functions with HEAP_ZERO_MEMORY is that
   the ASCII-null 0 or the ASCII(48)  '0'    ?
-  on 32 bit windows the memory pages are 4K each so does that mean,..the smallest amount
   actually allocated is 4Kbytes ? - like even if I allocate 256bytes,.. 4Kilobytes is still allocated ?

Thankyou.
-

• cb is just a naming convention I use, and Microsoft also, it means count bytes
  
• Initialization to zero is  to 0 (or NULL if you prefer), why would it mean initialization to 48 ?
  
• Yes, when using VirtualAlloc the size is always rounded up to the next page boundary, also the start address if specified is rounded down to the nearest boundary. So, if you have a buffer of 4097 bytes, 8192 bytes will be allocated since the last byte crosses to a second page. A memory page is normally 4KB, you should do a check system granularity before you assume that, it is subject to change. If you are allocating large buffers you should always use multiples of system granularity. You can obtain the page size using...

Code Select Expand

LOCAL si:SYSTEM_INFO

invoke GetSystemInfo, offset si
mov eax, [si.dwPageSize]

Donkey

[Rainstorm]

Hi,
here is the code where i've used HeapAlloc to allocate & reallocate memory.
It searches for lines with a character count equal to & above the requested amount, stores the references to them (line numbers & how many chars each line contains) in 2 buffers & then displays the results.
Donkey, thanks for clearing those earlier questions..

- It seems to assemble &  work okay..not tested it much.., however with a high number of matches (around 550 something like that ?) windows terminates it, dunno why this happens.. or if the console can't handle all the output.
- have a few more questions but will ask them later.
..would appreciate any feedback.. & also if there are any mistakes in it.

Thanks.

Code Select Expand

    .const
        HEAP_CREATE_ENABLE_EXECUTE      equ     262144

    .data?

    .data
        pfname                      dd      0
        flen                        dd      0
        fpattern                    db      "text files",0,"*.txt",0,0
        total_characters_in_line    dd      0
        pfbuff                      dd      0
        line_char_limit             dd      0
        pline_char_limit            dd      0
        hInstance                   dd      0
       
        hHeap_line                  dd      0
        hHeap_linecharcount         dd      0
        p_linecount_buff            dd      0
        p_linecharcount_buff        dd      0

        ptop_linecount_buff         dd      0
        ptop_linecharcount_buff     dd      0
        linebuffer_usedbytes        dd      0
        charbuffer_usedbytes        dd      0
        linebuff_size               dd      0
        line_charcountbuff_size     dd      0
        total_matchinglines_found   dd      0       
        total_lines                 dd      0
        temp                        dd      0


; -----------------------------------------------------------------------


    .code

start:

    invoke GetModuleHandle, NULL      ; get the handle of this program
    mov hInstance, eax

    mov pfname,  OpenFileDlg(NULL,hInstance,"Select File",addr fpattern)
    cmp BYTE PTR [eax], 0
    je exit_

   
    invoke read_disk_file,pfname,addr pfbuff, addr flen 
    cmp eax, 0
    je exit_

    print "returned file length - "
    print ustr$(flen),13,10

   ; ------- // Input the character limit // ---------

    mov pline_char_limit, input("enter the max character count of line",58)
   
    mov line_char_limit, uval(pline_char_limit)
    print "pause.. - character Limit - : "
    print ustr$(line_char_limit),13,10

;      mov eax, pfbuff
;      mov byte ptr [eax+flen-1], 0          ; add a NULL at the eof in the buffer



;     ------ //  Create the memory heap & Allocate Memory  // --------

    ; ----------------------------------
    ;  create buffer for line numbers
    ; ----------------------------------
   
      invoke HeapCreate, HEAP_CREATE_ENABLE_EXECUTE or HEAP_NO_SERIALIZE, 0, 0
        cmp eax, 0
        je heap_creation_error
        mov hHeap_line, eax                 ; handle to memory object

      invoke HeapAlloc, hHeap_line, HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY, 512
        cmp eax, 0
        je heap_allocation_error
        mov p_linecount_buff, eax           ; pointer to allocated block
        mov ptop_linecount_buff, eax
       
      invoke HeapSize, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff

        cmp eax, 4294967295                 ; test if the size function succeeded
        je heap_size_error
        mov linebuff_size, eax              ; store the size


    ; ----------------------------------------------
    ;  create buffer for the character count of the lines
    ; ----------------------------------------------

      invoke HeapCreate, HEAP_CREATE_ENABLE_EXECUTE or HEAP_NO_SERIALIZE, 0, 0
        cmp eax, 0
        je heap_creation_error
        mov hHeap_linecharcount, eax             ; handle to memory object

      invoke HeapAlloc, hHeap_linecharcount, HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY, 512
        cmp eax, 0
        je heap_allocation_error
        mov p_linecharcount_buff, eax            ; pointer to allocated block
        mov ptop_linecharcount_buff, eax

      invoke HeapSize, hHeap_linecharcount, HEAP_NO_SERIALIZE, p_linecharcount_buff

        cmp eax, 4294967295
        je heap_size_error
        mov line_charcountbuff_size, eax
   

   ; -----------------------------------------------


        mov esi, pfbuff                     ; move address of the file buffer into esi
        xor ecx, ecx
        xor ebx, ebx
        jmp count_characters
 
    pre_count:
         xor ecx, ecx                        ; reset count of chars in the line
         add esi, 1
       
      count_characters:
          cmp byte ptr [esi], 0
          je display_results       
          cmp byte ptr [esi], 10
          je newline
          inc esi                           ; pointer to file buffer
          add ecx, 1                        ; count of characters in the line
          jmp count_characters

    newline:
;         add ecx, 1                        ; would count the 'LF' too
          add total_lines, 1                ; total line count
          mov total_characters_in_line, ecx
          cmp ecx, line_char_limit    ; cmp the total chars in the line with the requested limit
          jb pre_count
          add total_matchinglines_found, 1


comment * -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  - 
          else check to see if the memory buffer is full
          if so, reallocate more memory
          else add the character count & line number to the buffer
        -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  - *


    ; ---- / check if there's enough space in the buffers / -----

          mov eax, ptop_linecount_buff
          sub eax, p_linecount_buff         ; subtract base addr from top addr
          mov linebuffer_usedbytes, eax
          mov edx, linebuff_size
          sub edx, 16                       ; check at 16 bytes ahead of time
          cmp edx, eax                      ; cmp no. of used bytes with the buffer size
          jbe reallocate_linecount_buff
          call store_linenumber


       check_line_charcount_buffer:

          mov eax, ptop_linecharcount_buff
          sub eax, p_linecharcount_buff
          mov charbuffer_usedbytes, eax
          mov edx, line_charcountbuff_size
          sub edx, 16                       ; reallocate 16 bytes ahead of time
          cmp edx, eax
          jbe reallocate_charcount_buff
          call store_line_charcount
          jmp pre_count

         

    ; ---- / add the line number & character count of the line to the buffers / -----


       store_linenumber:

          mov eax, total_lines                  ; the line number
          mov edx, ptop_linecount_buff
          mov [edx], eax                        ; move the count of chars into the buffer
          add ptop_linecount_buff, 4            ; track the top of the buffer

         ret

       store_line_charcount:
       
          mov ecx, total_characters_in_line
          mov edx, ptop_linecharcount_buff
          mov [edx], ecx                         ; move the line count in the buffer
          add ptop_linecharcount_buff, 4         ; track the top of the linecount buffer
         ret 


    ; ----------- //  Reallocate Memory  // ---------------

    reallocate_linecount_buff:
     
      invoke HeapReAlloc, hHeap_line,HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY,
                          p_linecount_buff, 256

        cmp eax, 0
        je heap_reallocation_error
        mov p_linecount_buff, eax              ; pointer to reallocated block


      invoke HeapSize, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff

        cmp eax, 4294967295                    ; test if the size function succeeded
        je heap_size_error
        mov linebuff_size, eax   

        mov eax, linebuffer_usedbytes
        add eax, p_linecount_buff
        mov ptop_linecount_buff, eax
        call store_linenumber
        jmp check_line_charcount_buffer

    ;---------------------------------

    reallocate_charcount_buff:   
      invoke HeapReAlloc, hHeap_linecharcount,HEAP_NO_SERIALIZE or HEAP_ZERO_MEMORY,
                          p_linecharcount_buff, 256

        cmp eax, 0
        je heap_reallocation_error
        mov p_linecharcount_buff, eax       ; pointer to reallocated block

      invoke HeapSize, hHeap_linecharcount, HEAP_NO_SERIALIZE, p_linecharcount_buff

        cmp eax, 4294967295                 ; test if the size function succeeded
        je heap_size_error
        mov line_charcountbuff_size, eax              ; store the size


        mov eax, charbuffer_usedbytes       
        add eax, p_linecharcount_buff       ; bring the pointer in the new block..
        mov ptop_linecharcount_buff, eax    ; to where it was in the data before..
        call store_line_charcount           ; reallocation.
        jmp pre_count



  ;---------------------------------------------------------------------   
  ;                display the results
  ;         ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

        display_results:
          print " ",13,10,
          print "Filename : "
          print pfname,13,10
          print "Total Lines: "
          print ustr$(total_lines),13,10
          inkey
          print " ",13,10,
          print "Lines with a character count equal to or above "
          print ustr$(line_char_limit)
          print "  =  "
          print ustr$(total_matchinglines_found),13,10
          print "=============================================",13,10

          mov edi, p_linecount_buff
          mov edx, ptop_linecount_buff     
          mov dword ptr [edx+1], 0            ; add a 0 at the top of the buffer

;    --//  display the line numbers //
           
        display_results1:
          mov ebx, dword ptr [edi]
          print ustr$(ebx)
          print ", "
          add edi, 4
          cmp dword ptr [edi], 0
          jne display_results1

          print " ",13,10,
          print "=============================================",13,10
          print " ",13,10,
      inkey
      print " ",13,10,13,10
      print "lines with corresponding character counts equal\
             to or above the requested limit",13,10

      print "=============================================",13,10

          mov edi, p_linecount_buff
          mov edx, ptop_linecount_buff     ; add a 0 at the top of the buffer
          mov dword ptr [edx+1], 0

          mov esi, p_linecharcount_buff
          mov edx, ptop_linecharcount_buff
          mov dword ptr [edx+1], 0

comment * -------------------------------------------------------   
            Display the line numbers with the corresponding char
            counts of the lines
          ------------------------------------------------------- *   

        display_results2:
          mov ebx, [edi]
          print ustr$(ebx)
          print " -  "
          mov ebx, [esi]
          print ustr$(ebx),13,10
          add edi, 4
          add esi, 4
          cmp dword ptr [edi], 0
          jne display_results2
         
      print " ",13,10,
      print "=============================================",13,10

      print "finished",13,10,13,10
      jmp free_memory


   ; ----------- //  Errors  //------------------------------------------

    heap_creation_error:
         print "memory error: memory heap couldn't be created",10,13
         jmp exit_

    heap_allocation_error:
         print "memory error: memory couldn't be allocated",10,13
         jmp exit_
         
    heap_reallocation_error:
         print "memory error: memory couldn't be reallocated",10,13
         jmp free_memory
         
    heap_size_error:
         print "memoryerror: heap size error",10,13
         jmp exit_

   ; ---------------------------------------------------------------------

    free_memory:

        free pfbuff
        free pfname
       
        invoke HeapFree, hHeap_line, HEAP_NO_SERIALIZE, p_linecount_buff
            cmp eax, 0
            jne @F
        print "HeapFree error !",13,10

      @@:
        invoke HeapFree, hHeap_linecharcount,HEAP_NO_SERIALIZE,p_linecharcount_buff
            cmp eax, 0
            jne @F
        print "HeapFree error !",13,10
            jmp exit_


     @@:
        print "Memory successfully freed",13,10


    exit_:
         inkey "press any key to exit..."
         exit


end start


[Rainstorm]

in the above code if i try to add a null at the end of the buffer into which the file is read, using invoke read_disk_file , .. I get the windows error of having to shut down the application..- why is that ?
I've commented the 2 lines out in the code in the previous post

Code Select Expand

      mov eax, pfbuff
      mov byte ptr [eax+flen+1], 0          ; add a NULL at the eof in the buffer


- If i specify the option HEAP_CREATE_ENABLE_EXECUTE while creating the heap & a processor doesn't support that option.. then are there any side effects ?\

- Suppose i free a heap or any other memory.. that hasn't been allocated in the first place.. what happens then, are there any effects ?

many thanks

Rainstorm

-

[Rainstorm]

any one have any idea about or can help clarify the above statements ?

[xmetal]

Code Select Expand


mov eax, [pfbuff]
add eax, [flen]
mov byte ptr [eax+1], 0          ; add a NULL at the eof in the buffer