Overcoming stack overflow with SEH

[ekx]

EDIT: This part is solved. Read further down: http://www.masm32.com/board/index.php?topic=10485.msg76910#msg76910

I can't figure out how to overcome stack overflow exceptions with SEH.
I know that the Exception Handler can't run without stack space, but there seems to be a way.

I tried writing a try/catch block in C++ and it seems to work for them, but the code is a little tricky so I can't really figure what's different.

A stack overflow such as this:

while (1)
    push eax
endw

Will be caught in a Cpp try block, but not in my asm program (to be more specific, the program will just exit without notice). Any ideas?

Example asm code that exits the program:

Code Select Expand

push offset TimerExceptionHandler1
push fs:[0]
mov fs:[0], esp

.while 1
push eax
.endw

TimerExceptionHandler1:

invoke MessageBox, NULL, addr ClassName, NULL, MB_OK

ret

However, if I would use for instance - mov ecx, 0 - mov dword ptr [ecx], 0 - it would MessageBox.

[japheth]

Hi,

it "should" work - and for me it does.

I successfully tried with a virtually identical source:

Code Select Expand



        .386
        .MODEL FLAT
        option casemap:none

MessageBoxA proto stdcall :dword, :dword, :dword, :dword
ExitProcess proto stdcall :dword

.data
szText db "caught",0
        .CODE

assume fs:nothing

main    proc c

xor edx,edx
        push offset myexc
        push fs:[edx]
        mov fs:[edx],esp
        mov     edi,0
        mov     esi,1
        mov     ebp,2
        mov     ecx,6
        mov     eax,7
        mov     ebx,3
        mov edx, "STAC"
@@:       
        push edx
        jmp @B
myexc:
        invoke MessageBoxA, 0, addr szText, 0, 0
        invoke ExitProcess, 0
       
main    endp

mainCRTStartup proc c

        call main
        invoke ExitProcess, eax

mainCRTStartup endp

        END mainCRTStartup


Makefile was:

Code Select Expand


NAME=testexcE

ASM=ml -coff -c -Sg -Fl$* -Fo$*

$(NAME).exe: $*.obj
link $*.obj /LIBPATH:\win32inc\lib kernel32.lib user32.lib /out:$*.exe /subsystem:console

$(NAME).obj: $(NAME).asm
$(ASM) $(NAME).asm


[ekx]

Thanks for the help, I'll have a look. I think there is more to what I'm trying to do, though...

[ekx]

Looking at your code helped me realize something. My main application was actually using some code that I have copied off somewhere.

I was using this as my exception procedure.

TimerExceptionHandler1 PROC C pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

Because there is arguments, the compiler inserts some pushes at the start and it makes it crash because the stack is overflown.

Removing those arguments fixed the problem.

Thanks for the help.

[ekx]

That wasn't the problem in the end.

After awhile I realized that this works for me if it's done in the main thread of the program. If the same thing is done in a thread (even one that's freshly created), the program just closes, which is why I was having so much trouble with this since I was testing across those two mediums.

invoke CreateThread, NULL, 1024*1024, offset TTimer, NULL, NULL, offset TimerThreadId

Will crash in event of a stack overflow, but

invoke CreateThread, NULL, 1024*1024 + 24, offset TTimer, NULL, NULL, offset TimerThreadId

Will recover successfully. It seems that pushing on the extra 24 bytes raises an exception, but the exception procedure can use those 24 bytes for it's arguments??

invoke CreateThread, NULL, NULL, offset TTimer, NULL, NULL, offset TimerThreadId

of course also works, because it's the default setting.

[ekx]

My main problem right now is that a stack overflow gets only successfully raised once. The second time that it overflows, an exception doesn't get raised, and the stack frame used for the exception handler which follows gets corrupted making the program crash. I have tried using VirtualProtect , VirtualFree, etc, to somehow re-enable guarding of the last stack page, but couldn't get it to work, or to raise exceptions on guarded/no-access pages.

The following code shows that the program crashes only after a second stack overflow:

Code Select Expand


push offset TimerExceptionHandler1
push fs:[0]
mov fs:[0], esp
   
mov TimerExceptionSafe, offset Restart
mov TimerExceptionStack, esp
mov TimerExceptionEbp, ebp
   
Restart:
invoke MessageBoxA, 0, offset szStackOverflow, 0, MB_OK

.while 1
push eax
.endw

ret

TimerExceptionHandler1 PROC C pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

mov eax, pContext
mov ecx, TimerExceptionSafe
mov [eax].CONTEXT.regEip, ecx
mov ecx, TimerExceptionStack
mov [eax].CONTEXT.regEsp, ecx
mov ecx, TimerExceptionEbp
mov [eax].CONTEXT.regEbp, ecx

mov eax, ExceptionContinueExecution

ret

TimerExceptionHandler1 endp


This is some of the things I have tried to restore the stack overflow protection:

Code Select Expand



.data?
TimerMBI MEMORY_BASIC_INFORMATION {}
OldProt dd 2 dup (?)

.code

TimerExceptionHandler1 PROC C pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

mov eax, pExcept
mov ecx, [eax+4].EXCEPTION_RECORD.ExceptionInformation ; address of accessed memory

invoke VirtualQuery, ecx, offset TimerMBI, sizeof TimerMBI

mov eax, TimerMBI.AllocationProtect
or eax, PAGE_GUARD

;invoke VirtualFree, TimerMBI.BaseAddress, 4096, MEM_DECOMMIT
invoke VirtualProtect, TimerMBI.BaseAddress, eax, 4096, offset OldProt

mov eax, pContext
mov ecx, TimerExceptionSafe
mov [eax].CONTEXT.regEip, ecx
mov ecx, TimerExceptionStack
mov [eax].CONTEXT.regEsp, ecx
mov ecx, TimerExceptionEbp
mov [eax].CONTEXT.regEbp, ecx

mov eax, ExceptionContinueExecution

TimerExceptionHandler1 endp


Any ideas are welcome.