Growable memory

[guga]

Hi guys.....i´m facing some small problems


1) i´m trying to make a growable memory using virtuallloc/virtualfree instead heapalloc/heapcreate/heapfree functions. This is because heapfree is crashing whenever i try to free the memory heap (A stack problem inside this api). Does anyone know how to make a growable memory using virtualalloc?

I´m trying to do this:
* allocate XXX memory in virtualloc
* manipulate the data to make the buffer contains what i want.
* When the data reachs the buffer limit (say... 1000 bytes), i want that this buffer be extended to allocate more 1000 bytes (Now it will have 2000 bytes)
* When the manipulation data is finished. I mean, i finish fill the buffer (that now contains 2000 bytes) ii want it to correctly free it (with virtualfree)

2) Does anyone knows how to get a module name once it is loaded in memory through a debugger ? I suceeded to grab some module names, but others simply don´t show up (all i got is the base address and PID)


Can someone help ?


Best Regards,

guga
____________________________________________________
Btw: I updated Rosasm


Debugger [Version 2.3] - Gustavo Trigueiros (Beyond2000!)
RosAsm 2.053a [Debugger V2.3]
* [Bugfix] UPdated DEBUG_EVENT structure
* Implemented: Added a popmenu to view and dump memory. Now it is able to dump memory on each segment when clicking on the Address Space Tab on the debugger. Todo>> dump full module memory
* Implemented: Identify when a loaded module is packed. It will display it as "packed dll". - Todo >> Try to force find the correct name of the modules.
* added &HEAP_NO_SERIALIZE when creating a growable heap at DebugRsrcManagerThread call 'KERNEL32.HeapCreate' &HEAP_NO_SERIALIZE, 4096, 0
Debugger_OnUnloadDll call 'KERNEL32.HeapFree' D$ModuleNameHeap, &HEAP_NO_SERIALIZE, D$edi+ME_Name
* full review of ScanPEHeader
* developed GetModuleSize function
* updated: CreateNewForm_AddressSpaceForm
AddressSpaceFormProc
AddressSpaceForm_OnTreeNavigate
AddressSpace_HandleMouseProc
LogFormProc
* implemented: AddressSpaceDialog_OnCreate
AddressSpaceDialog_OnSize
AddressSpaceForm_OnCommand
AddressSpace_CreatePopupMenu

[jj2007]

Guga,

See the self-expanding buffer thread - using exceptions, i.e. for lazy programmers.

However, it works also fine with this snippet that uses HeapAlloc under the hood:
include \masm32\MasmBasic\MasmBasic.inc   ; download
   Init
   Dim My$(1)
   For_ n=0 To 999
      Let My$(n)=Str$("This is string #%i", n)
   Next
   Inkey "bye"
   Exit
end start

> This is because heapfree is crashing whenever i try to free the memory heap (A stack problem inside this api)
Can you post an example that demonstrates your problem with HeapXXX?

[guga]

This is kinda hard to reproduce the problem because i´m coding inside rosasm main debugger functions...but, i´ll try to post the ones related to the heapxxx (or later i´ll try to make a small app that uses those functions and reproduce the error)

The function grows a array of structures whose members holds information about the debugged module. The heapfree problem happens after it unloaded a given module.

1st step

Code Select Expand



; The 'ModuleList' points at a list of module information entries that are currently
; loaded and mapped into the address space of the debuggee.

; Entry in module list
[ME_Base 0          ; The base VA of the mapped module
ME_Size 4          ; size in bytes
ME_Name 8          ; pointer to filename
ME_CodeBase 12     ; base VA of code section
ME_CodeSize 16     ; size in bytes of code section
ME_ExportBase 20   ; base RVA of export section
ME_ExportSize 24]  ; size in bytes of export section
[SizeOf_ModuleEntry 32]

[ModuleList: ? NumModules: ? ModuleNameHeap: ?]
DebugRsrcManagerThread:
(...)
    If D$ModuleNameHeap <> 0
        call 'KERNEL32.HeapDestroy' D$ModuleNameHeap | mov D$ModuleNameHeap 0
    End_If

    call 'KERNEL32.HeapCreate' &HEAP_NO_SERIALIZE, 4096, 0 ; Create a growable heap.
    mov D$ModuleNameHeap eax

    VirtualAlloc ModuleList 4096
mov D$NumModules 0

  ; Create the thread that creates and observes the debuggee.
    call 'Kernel32.CreateThread' &NULL, 0, DebugThread, 0,
                                &THREAD_PRIORITY_NORMAL+&CREATE_SUSPENDED, DebugThreadId
(...)
  ; Cleanup
    VirtualFree D$ModuleList

    call 'KERNEL32.HeapDestroy' D$ModuleNameHeap | mov D$ModuleNameHeap 0
    VirtualFree D$ThreadIDHandleTable

    call 'Kernel32.CloseHandle' D$PI.hThread
call 'Kernel32.CloseHandle' D$PI.hProcess

2nd step

Code Select Expand


Proc DebugThread:
    Local @Exit, @CloseSent

    call CreateDebuggeeProcess
    If eax = &FALSE
        jmp L9>>
EndIf
(...)
    .While D@Exit = &FALSE

(...)

        mov D$ContinueStatus &DBG_CONTINUE

        .If D$DEBUG_EVENT+DEBUG_EVENT.dwDebugEventCodeDis = &EXIT_PROCESS_DEBUG_EVENT
            mov D@Exit &TRUE
(...)
        .Else_If D$DEBUG_EVENT+DEBUG_EVENT.dwDebugEventCodeDis = &LOAD_DLL_DEBUG_EVENT
            call Debugger_OnLoadDll D$DEBUG_EVENT+DEBUG_EVENT_LOAD_DLL_DEBUG_INFO.lpBaseOfDllDis
        .Else_If D$DEBUG_EVENT+DEBUG_EVENT.dwDebugEventCodeDis = &UNLOAD_DLL_DEBUG_EVENT
            call Debugger_OnUnloadDll D$DEBUG_EVENT+DEBUG_EVENT_UNLOAD_DLL_DEBUG_INFO.lpBaseOfDllDis
(...)
        .Else_If D$DEBUG_EVENT+DEBUG_EVENT.dwDebugEventCodeDis = &CREATE_PROCESS_DEBUG_EVENT
            call Debugger_OnCreateProcess ---- Following this to increase heap
(...)
        .Else
            mov D$ContinueStatus &DBG_EXCEPTION_NOT_HANDLED
        .End_If

        call 'Kernel32.ContinueDebugEvent' D$DEBUG_EVENT+DEBUG_EVENT.dwProcessIdDis, D$DEBUG_EVENT+DEBUG_EVENT.dwThreadIdDis, D$ContinueStatus

    .End_While


3rd step – inside Debugger_OnCreateProcess

Code Select Expand


Debugger_OnCreateProcess:

    ;call 'KERNEL32.GetFileSize' D$CPDI.hFile, 0 | mov D$PE_hdr_ProcessSize eax


    call 'KERNEL32.CloseHandle' D$DEBUG_EVENT+DEBUG_EVENT_CREATE_PROCESS_DEBUG_INFO.hFileDis

    mov D$DebugBaseOfCode 0, D$DebugCodeSize 0

    mov esi D$ThreadIDHandleTable
    move D$esi D$DEBUG_EVENT+DEBUG_EVENT.dwThreadIdDis
    move D$esi+4 D$DEBUG_EVENT+DEBUG_EVENT_CREATE_PROCESS_DEBUG_INFO.hThreadDis
    inc D$NumThreads

    call ScanPEHeader D$DEBUG_EVENT+DEBUG_EVENT_CREATE_PROCESS_DEBUG_INFO.lpBaseOfImageDis ----Following this

    call GetModuleName D$DEBUG_EVENT+DEBUG_EVENT_CREATE_PROCESS_DEBUG_INFO.lpBaseOfImageDis
    FormatString DebugLogString, {'%s ID=%d mapped at 0x%x' 0}, eax, D$DEBUG_EVENT+DEBUG_EVENT.dwProcessIdDis, D$DEBUG_EVENT+DEBUG_EVENT_CREATE_PROCESS_DEBUG_INFO.lpBaseOfImageDis
    call 'User32.PostMessageA' D$DebugDialogHandle, WM_LOG, DebugLogString, ecx
ret


4th step – inside ScanPEHeader

Code Select Expand


Proc ScanPEHeader:
    Arguments @BaseAddress
    Local @Size, @CodeBase, @CodeSize, @ExportBaseRVA, @ExportSize, @Name, @IsExe, @SectionsCount, @PESection, @PEHdr
    Uses esi, edi, ebx


    call GetModuleSize D$PI.hProcess, D@BaseAddress, PE_hdr_ProcessSize
    On eax = 0, ExitP


    mov B@IsExe 0
    mov D@SectionsCount 0
    ; Read size of image, base-address and size of the code section & export table
    ; from PE image header.
    mov D$PEBuffer 0
    mov D$PESectionBuffer 0
    mov D$PESectionSize 0

    VirtualAlloc PEBuffer, D$PE_hdr_ProcessSize

    call ReadProcessMem D@BaseAddress, D$PEBuffer, D$PE_hdr_ProcessSize
    On eax = 0, jmp @Error

    call CheckPE_MZ D$PEBuffer, D$PE_hdr_ProcessSize, PE_hdr_Process
    On eax = 0, jmp @Error
    On W$eax+PeHeader.OptionalHeader.MagicDis <> &IMAGE_NT_OPTIONAL_HDR32_MAGIC, jmp @Error ; check optional header ID
mov esi eax
(...)
        .If B@IsExe = &FALSE
            ; A dll without an export table??
            On D@ExportBaseRVA = 0, jmp @Error
            On D@ExportSize = 0, jmp @Error

            mov eax D@ExportBaseRVA
            add eax D$PEBuffer
            mov esi D$eax+IMAGE_EXPORT_DIRECTORY.nNameDis
            call dbg_IsPointtoVirtualSection esi, D@SectionsCount, D@PESection, D$PEBuffer, D@PEHdr
            If eax = &TRUE
                ;move D@Name {"Packed Dll", 0} | jmp L2>>
                mov esi {"Packed Dll", 0}
            Else
                add esi D$PEBuffer
            End_If

            ;add esi D$PEBuffer
            call StrLenProc esi
            call 'KERNEL32.HeapAlloc' D$ModuleNameHeap, 0, eax --------Here we allocate the heap
            mov D@Name eax, edi D@Name

            While B$esi <> 0 | movsb | End_While
            mov B$edi 0

        ; TODO is it possible that exe files are used as library from the actual application exe?
        ; If really possible this branch would be wrong. (ntoskrnl.exe !)
        .Else
            move D@Name D$DebuggeeExeTitle
        .EndIf

(...)

        VirtualFree D$PEBuffer
        new Module edi ---- This macro does this:
;    IMUL EAX D$NUMMODULES 32
;    ADD EAX D$MODULELIST
;    INC D$NUMMODULES
;    MOV EDI EAX

        move D$edi+ME_Base D@BaseAddress
        move D$edi+ME_Size D@Size
        move D$edi+ME_Name D@Name
        move D$edi+ME_CodeBase D@CodeBase
        move D$edi+ME_CodeSize D@CodeSize
        move D$edi+ME_ExportBase D@ExportBaseRVA
        move D$edi+ME_ExportSize D@ExportSize

        mov eax D@BaseAddress, ecx D@CodeSize, edx D@CodeBase
        .If D$SavingExtension = '.DLL'
            If eax = D$LinkerDllDefault
                mov D$DebugBaseOfCode edx
                mov D$DebugCodeSize ecx
            EndIf
        .Else
            If eax = LINKERDEFAULT
                mov D$DebugBaseOfCode edx
                mov D$DebugCodeSize ecx
            EndIf
        .EndIf

        mov eax 1
        ExitP

@Error:
        VirtualFree D$PEBuffer
       



5th step – inside Debugger_OnLoadDll

Code Select Expand


Proc Debugger_OnLoadDll:
    Arguments @BaseAddress

        call 'Kernel32.CloseHandle' D$DEBUG_EVENT+DEBUG_EVENT_LOAD_DLL_DEBUG_INFO.hFileDis;LoadDll.hFile

        call ScanPEHeader D@BaseAddress --Scan Again whenever a Module is loaded during debugging

        call GetModuleName D@BaseAddress
        FormatString DebugLogString, {'%s mapped at 0x%x' 0}, eax, D@BaseAddress
        call 'User32.PostMessageA' D$DebugDialogHandle, WM_LOG, DebugLogString, ecx

EndP


6th step – inside Debugger_OnUnloadDll -- THE ERROR IS HERE

Code Select Expand


Proc Debugger_OnUnloadDll:
    Arguments @BaseAddress
    Uses esi, edi

        call GetModuleName D@BaseAddress
        FormatString DebugLogString, {'%s unmapped' 0}, eax
        call 'User32.PostMessageA' D$DebugDialogHandle, WM_LOG, DebugLogString, ecx

        ; Search module in table
        mov edi D$ModuleList, ecx 0, eax D@BaseAddress
        While ecx < D$NumModules
            On D$edi = eax, jmp L0>
            add edi SizeOf_ModuleEntry
            inc ecx
        EndWhile
        ExitP

L0:
        ; Do cleanup work. Free the memory for the module name.
        call 'KERNEL32.HeapFree' D$ModuleNameHeap, &HEAP_NO_SERIALIZE, D$edi+ME_Name ------------This is crashing internally on kernel module and giving me a exception. (In fact it seems to be a stack problem inside heapfree/heapalloc functions, when it donp´t have enough room to create the heap or perhaps because while the heap is growing the function looses the stack pointer internally


        ; Overwrite the entry to delete with last entry in table.
        delete Module edi
; The delete macro does this:
;    PUSH ESI
;    PUSH EDI
;    PUSH ECX
;    DEC D$NUMMODULES
;    MOV ECX 32
;    IMUL ESI D$NUMMODULES 32
;    ADD ESI D$MODULELIST
;    MOV EDI EDI
;    REP MOVSB
;    POP ECX
;    POP EDI
;    POP ESI

EndP


[qWord]

debugging, multithreading and then: HEAP_NO_SERIALIZE? - maybe a problem.

[guga]

Hmm analysing SelfExpandingBuffer it seems to work as a replacement to heapxx functions.

one doubt....after it increases the memory with invoke VirtualAlloc, pNextPage, dwPageSize , MEM_COMMIT, PAGE_READWRITE, will it fully release the memory on virtualfree ?

[guga]

Hi qword...It makes no diffrerence with this equate. If i use 0 the error is the same:

[qWord]

The probability that the heap functions are faulty is negligible - it is most probably your code.
Your 'solution' hides the real problem.

qWord

[jj2007]

The probability that the heap functions are faulty is negligible - it is most probably your code.

I agree - there are tons of professional code that depend on the reliability of the heap functions.

Guga, your code is complex, and I won't find the time to dig deep into it. One thing I spotted is PostMessage. Just in case the receiver of this "posting" is doing something with the allocated memory: PostMessage is slow. When the message arrives, you have most probably already freed the memory. And bang it goes...

[guga]

Hmm... yep...i restored to the original code..Using heapalloc to create a new heap for the module names.
I fixed it making heapalloc at the beginnng of the ScanPEHeader function

call 'KERNEL32.HeapAlloc' D$ModuleNameHeap, 0, 64 | mov D@Name eax

It worked as expected when i loaded a dll for the first time but....when i load the host of that dll (vdub on this case) the heapfree function is modofied somehow



Now i don´t know if the problem is within my code or in the vdub plugin i made or in the target´s (host) .

Here seems to be the same problem as i´m having now :

http://social.msdn.microsoft.com/Forums/hu-HU/vclanguage/thread/01bae812-0a5f-4b17-9745-b1c8293a25b1


I tested on a dll i know it have no errors. (The example on masm package)....it disassembled the dll, and i fired the debug....It worked as expected...It loaded the host application without errors.

So...probably the error is on my vdub dll or in vdub itself loaded as a host

[qWord]

What kind of help do you suggest to get here? The images doesn't help anyone. The code is incomplete and the syntax is horribly ...
However, one thing that may also produce problems: you are combining flags using '+'.

[dedndave]

i don't think those are flags   :P

[guga]

yeah...those are not flags.

They are only equates to behave as pointers as in classes.

call D$edx+8
call D$edx+16 etc etc

Btw: I fixed that....The problem is somewhere inside my vdub plugin. I´ll take a look at that later

Many thanks to all :)