GlobalReAlloc fails, with debugger all ok!!! what wrong

[denise_amiga]

I have been playing a little with the donkey code about strings, and everything works well except for the routine sa_Expand, that fails, when I examine with debugger (ollydbg) the routine works correctly, reserve the new memory an return the new ptr, without debugger return 0, that I am making bad?

[attachment deleted by admin]

[bozo]

denise_amiga

I encountered a smiliar problem with GlobalReAlloc some time ago, and i'm still not sure what the problem is.

After much time, searching the net, asking in forums and not getting response, I was forced into an alternative
method of memory allocation which actually works quite well.

Rather than "re-allocate" memory using GlobalReAlloc, and similarly with HeapReAlloc, LocalReAlloc
which also seem to fail after too many calls.

You first free the old memory using GlobalFree, (if it is handle of course), then using the old size + new size, use GlobalAlloc
to allocate your new space.

This is the only way i could figure out how to do it, and it seemed to work fine for me.

The old data stored in the memory handle returned by GlobalAlloc first time around, wasn't destroyed in my trials.

I will try post some code later to give you example, but right now i can't.

[chep]

You first free the old memory using GlobalFree, (if it is handle of course), then using the old size + new size, use GlobalAlloc
to allocate your new space.
[...]
The old data stored in the memory handle returned by GlobalAlloc first time around, wasn't destroyed in my trials.

Kernel_Gaddafi,

Under heavy memory load and/or in multithreaded context you can't rely on this method to preserve the original memory contents!

Simplest case: multithreading
A second thread could call GlobalAlloc while the first thread is "between" GlobalFree and GlobalAlloc, giving the just freed location to the second thread. The first thread will then get another location when it calls GlobalAlloc, and obviously the original memory contents are lost.

Second case
ReAlloc functions use more or less the following algorithm:

Code Select Expand

if ( (new_size<=old_size) or enough_space) then
    extend_or_reduce_area
else
  alloc new_area
  copy old_area into new_area
  free old_area
endif

Clearly, if there isn't enough space to fit the new size, the handle of the new area will not be the same as the old one. With the method you describe the copy step obviously doesn't occur, again resulting in data loss.


I can't emphasize enough on the fact that even if there is one chance over 10 billions that your technique may fail, you can be sure it will eventually happen at least twice a day... :wink

[bozo]

chep

I understand and agree with the point you are making, but this doesn't address the issue of GlobalReAlloc failing for myself
or denise_amiga.

With 512 MB of RAM and only requiring a space of about 2-3 KB of reallocated memory, GlobalReAlloc shouldn't end up returning zero,
which is what looks to me what is happening with denise_amigas code.

I could be wrong of course, but i'm just giving some ideas in how to overcome the problem, not a complete solution.

Maybe the failure is a problem with windows operating system? ;-)

Why not try out a bit of code for yourself first of all, and tell me how you get on.

That is, allocate about 32KB of memory, and keep adding 32KB to the new size using GlobalReAlloc, and see what happens ;-)

BTW, if you believe it is another process/thread accessing memory "globally", should this still happen with HeapReAlloc/LocalReAlloc too?

[bozo]

Code Select Expand


comment ~

the test shows the message:

           "GlobalReAlloc called 1 times to allocate 64000 bytes of memory"

And the system error returned is:

           "Not enough storage is available to process this command.

This is on a system with 80GB HDD and 512 MB RAM...

~
.586
.model flat,stdcall

include <\masm32\include\kernel32.inc>
includelib <\masm32\lib\kernel32.lib>

include <\masm32\include\msvcrt.inc>
includelib <\masm32\include\msvcrt.lib>

include <\masm32\include\windows.inc>
include <\masm32\macros\macros.asm>

THIRTY_TWO_KB      EQU      32000

.data?

hMem      DWORD      ?
dwBytes    DWORD      ?

nTimes    DWORD      ?

.code

start:
      mov   [dwBytes], THIRTY_TWO_KB
      mov   [nTimes], 0

invoke GlobalAlloc,GMEM_FIXED,[dwBytes]
      mov   [hMem], eax

.IF ( [hMem] != NULL )
           
            .WHILE( TRUE )
                  invoke GlobalReAlloc,[hMem],[dwBytes],GMEM_FIXED
                  mov   [hMem], eax
                 
                  .IF( [hMem] == NULL)
                       .BREAK
                  .ELSE
                       inc   dword ptr [nTimes]
                       add   dword ptr [dwBytes], THIRTY_TWO_KB
                  .ENDIF
            .ENDW

            invoke crt_printf,
                   chr$(10,'GlobalReAlloc called %d time(s) to allocate %ld bytes of memory.'),
                   [nTimes],
                   [dwBytes],
                   eax
.ELSE
            invoke crt_printf,chr$(10,'GlobalAlloc() -> Could not allocate memory')
      .ENDIF
     
      exit

end start


comment ~

not solution, but atleast an idea to think about.

i've come to the conclusion, that it doesn't matter if you first free the memory, and then use GlobalAlloc again but with a different size, simply because they are all pointers to memory anyway, and would/should not be modified unless you specify GlobalAlloc to do so with GMEM_ZEROINIT

in fact, if you understand Virtual Memory, isn't it impossible for any
other process/external thread on the system to affect the memory of another process anyway?
As all processes theoritically have access to 2 GB of memory on windows system without interfering with other processes/threads.

Imagine you have memory labeled 1-6

1 - unused
2 - unused
3 - unused
4 - unused
5 - unused
6 - unused

and we allocate 2 blocks, now 1 + 2 are in *used* state, if somewhere else api called to alloc memory for just 1 block, then 3 is in *used* state, if we then realloc after freeing first 2 blocks, requesting an extra 2, the operating system should simply skip returning the 3rd pointer to memory..and return 1 + 2 + 4 + 5..leaving 3 alone.

am i wrong here?

I am not 100% sure, but this should work fine.

just don't zero initialise it.

Code Select Expand


      mov   ebx, 16       ; size of 1 structure???

      .WHILE(TRUE)
            .IF([pArray] != NULL)
                 invoke GlobalSize,[pArray]

                 .IF(eax == NULL)
                        .BREAK
                 .ELSE
                        add   eax, ebx                ; calculate new size of mem
                        invoke GlobalFree,[pArray]    ; free old mem pointer
            .ENDIF
                                                 
                 invoke GlobalAlloc,GMEM_FIXED,ebx    ; allocate memory
           
                 .BREAK    ; exit while(TRUE) / return result of GlobalAlloc
.ENDW
ret


I honestly don't know what i've been doing wrong when trying to use GlobalReAlloc,
but it has never worked as i'd expect it to, ..whats the problem?..the solution?

denise_amiga, your code in sa_expand procedure.
   

Code Select Expand


     mov    edx, [pArray]
     mov    ecx, [edx+_size]           ; i assume this is the old size of memory
     
     push   eax

     add    eax, 4                     ; add 4
     shl    eax, 2                     ; multiply by 4
     invoke GlobalReAlloc,edx,eax,GMEM_ZEROINIT
     pop   ecx


i don't understand why you multiply total_size by 4, surely this must be atleast part of the problem?

~

[chep]

I understand and agree with the point you are making, but this doesn't address the issue of GlobalReAlloc failing for myself
or denise_amiga.

You're right, my previous comment didn't intend to provide a solution, only to explain why yours isn't reliable IMHO. :wink

Anyway, I managed to reproduce the problem using the code you posted.

Imagine you have memory labeled 1-6

1 - unused
2 - unused
3 - unused
4 - unused
5 - unused
6 - unused

and we allocate 2 blocks, now 1 + 2 are in *used* state, if somewhere else api called to alloc memory for just 1 block, then 3 is in *used* state, if we then realloc after freeing first 2 blocks, requesting an extra 2, the operating system should simply skip returning the 3rd pointer to memory..and return 1 + 2 + 4 + 5..leaving 3 alone.

am i wrong here?

Indeed I believe that's where you're wrong. Anyone correct me if I say a mistake, but Global(Re)Alloc returns *consecutive* blocks. So in your example :

1) Alloc 2 blocks => (1,2) are used
2) Alloc 1 more => (1,2) & (3) are used
3) Free the first handle => (3) is used
4) Alloc 4 blocks => can't fit in (1,2) ; (3) is already used ; so using (4,5,6,7) !!!


As a side note:

If GlobalReAlloc fails, the original memory is not freed, and the original handle and pointer are still valid.

So you still have to use GlobalFree if GlobalReAlloc fails! (which you didn't call in your example)


The only workaround that comes to my mind for now is to create a wrapper around GlobalReAlloc :
1) try GlobalReAlloc, if it is successful then no problem !
2) else, create a new block with GlobalAlloc, copy the old data into the new block, and free the old block

Code Select Expand

SafeReAlloc PROC hOldMem, dwNewSize
  LOCAL   hNewMem:DWORD, dwSize:DWORD

  invoke  GlobalReAlloc, hOldMem, dwNewSize, 0 ; try GlobalAlloc
  test    eax, eax
  jnz     SafeReAlloc_exit ; success ! return hNewMem

  invoke  GlobalSize, hOldMem ; retrieve the original block's size
  test    eax, eax
  jz      SafeReAlloc_exit ; failure ! return 0

  cmp     eax, dwNewSize ; dwSize = MIN(dwNewSize, dwOldSize)
  jb      @F
  mov     eax, dwNewSize
@@:
  mov     dwSize, eax

  invoke  GlobalAlloc, GMEM_FIXED, dwNewSize ; alloc a new block
  test    eax, eax
  jz      SafeReAlloc_exit ; failure ! return 0
  mov     hNewMem, eax

  invoke  crt_memcpy, hNewMem, hOldMem, dwSize ; copy MIN(dwNewSize,dwOldSize) from hOldMem to hNewMem
  invoke  GlobalFree, hOldMem ; free hOldMem

  mov     eax, hNewMem ; return hNewMem
SafeReAlloc_exit:
ret
SafeReAlloc ENDP

It works fine, & the original data is *guaranteed* to be correctly preserved. I managed to run up to 500 iterations, ie. 16032000 bytes (it didn't stop because it failed, but because I limited the .WHILE loop to max. 500 to avoid an infinite loop).

Anyway, we still don't know why GlobalReAlloc doesn't work as expected...... :(

[bozo]

You da man! :)

[roticv]

My guess is the following:

Say you have the following memory:

0x0700000 - Used
0x0700004 - Unused
0x0700008 - Unused
0x070000C - Used

Say when you allocate 8bytes, you get 0x0700004. When you want to increase the size to say 16bytes, you call GlobalReAlloc. Since you can't increase the size, the function fails.

[chep]

Since you can't increase the size, the function fails.

Indeed! I was pretty sure GlobalReAlloc worked just like HeapReAlloc (which I use regularly), but as usual I'd better carefully RTFM! :red

HeapReAlloc's default behaviour is to move & copy the memory block to a new location if there isn't enough room to extend the memory block in place, unless you use the HEAP_REALLOC_IN_PLACE_ONLY flag:

There can be no movement when reallocating a memory block. If this value is not specified, the function may move the block to a new location. If this value is specified and the block cannot be resized without moving, the function fails, leaving the original memory block unchanged.

But GlobalReAlloc behaviour is just the opposite! You need to use the GMEM_MOVEABLE flag to allow the memory block to be copied to a new location if it can't be extended in place :

If GMEM_MODIFY is also specified, GMEM_MOVEABLE changes a fixed memory object to a movable memory object. [This is off topic]
If GMEM_MODIFY is not specified, then GMEM_MOVEABLE allows a locked GMEM_MOVEABLE memory block or a GMEM_FIXED memory block to be moved to a new fixed location. [This is what we want]
If neither GMEM_MODIFY nor GMEM_MOVEABLE is set, then fixed memory blocks and locked movable memory blocks will only be reallocated in place. [This was the problematic behaviour]

So, in Kernel_Gaddafi's example, just replace

Code Select Expand

.WHILE( TRUE )
  invoke GlobalReAlloc,[hMem],[dwBytes],GMEM_FIXED
  ...
with

Code Select Expand

.WHILE( [nTimes] < 100 ) ; just to be sure we don't go in an endless loop ;)
  invoke GlobalReAlloc,[hMem],[dwBytes],GMEM_MOVEABLE
  ...
and it will do the trick (tested, it works).


denise_amiga,
The same applies to your sa_Expand routine. Currently you are using

Code Select Expand

invoke GlobalReAlloc, edx, eax, GMEM_ZEROINIT
but it should be

Code Select Expand

invoke GlobalReAlloc, edx, eax, GMEM_ZEROINIT OR GMEM_MOVEABLE


As simple as that (well, it's always simple once you have found the solution :bg).
Thanks roticv for pointing this out!

[bozo]

So, in Kernel_Gaddafi's example, just replace

Code:
.WHILE( TRUE )
  invoke GlobalReAlloc,[hMem],[dwBytes],GMEM_FIXED
  ...with

Code:
.WHILE( [nTimes] < 100 ) ; just to be sure we don't go in an endless loop ;)
  invoke GlobalReAlloc,[hMem],[dwBytes],GMEM_MOVEABLE
  ...and it will do the trick (tested, it works).

The whole point of WHILE(TRUE) was to have an endless loop.
It continues until GlobalReAlloc fails, thats the point..never bloody mind.

Anyway, glad i finally found out what the problem was from this thread.

[chep]

The whole point of WHILE(TRUE) was to have an endless loop.
It continues until GlobalReAlloc fails, thats the point..never bloody mind.

I agree :wink but once it works correctly the whole point from my point of view is to avoid having to kill the process or hanging the machine :P :toothy
LOL
(I have *very* limited hardware here for now, it barely supports XP + firewall + AV + IE and anything more is a disaster :'()

[Manos]

denise_amiga

I encountered a smiliar problem with GlobalReAlloc some time ago, and i'm still not sure what the problem is.

After much time, searching the net, asking in forums and not getting response, I was forced into an alternative
method of memory allocation which actually works quite well.

Rather than "re-allocate" memory using GlobalReAlloc, and similarly with HeapReAlloc, LocalReAlloc
which also seem to fail after too many calls.

You first free the old memory using GlobalFree, (if it is handle of course), then using the old size + new size, use GlobalAlloc
to allocate your new space.

This is the only way i could figure out how to do it, and it seemed to work fine for me.

The old data stored in the memory handle returned by GlobalAlloc first time around, wasn't destroyed in my trials.

I will try post some code later to give you example, but right now i can't.

I have the same problem with HeapReAlloc.
Manos.