Exiting a DLL from Visual Basic

[jj2007]

I am working on a DLL that is callable from MS Word Visual Basic. Everything works fine except if it encounters a runtime error, and I leave the DLL with ExitProcess. In that case, Word says bye without even asking if you want to save your precious work.

Any idea how that could be handled? Triggering On Error GoTo would be ideal, of course....

include \masm32\MasmBasic\MasmBasic.inc

; Demo showing how to use assembler from MS Word Visual Basic
; Do not use - it is buggy

.code
LibMain proc instance:DWORD, reason:DWORD, unused:DWORD
   m2m eax, TRUE
   ret
LibMain endp

GetString proc uses esi MyIndex:DWORD, pString:DWORD   ; $export
  mov eax, chr$("A string")
  .if MyIndex<0
   invoke ExitProcess   ; NO GOOD
  .endif
  ret
GetString endp

end LibMain

The VB part:

Code Select Expand

Private Declare Sub GetString Lib "D:\masm32\MasmBasic\Mb2VB\MbDll.dll" (ByVal StrIndex As Long, ByVal lpString As String)
Sub MasmBasic()
    Dim JJ$
    JJ$ = "Hello Jochen"
    SayHello JJ$ + ", how are you?"
End Sub
Sub ShowString()
    Dim JJ$
    JJ$ = Space$(1000)      ' Loads \masm32\include\Windows.inc
    GetString 2, JJ$
    MsgBox JJ$, vbOKCancel, "MasmBasic returned a string:"
End Sub


[oex]

I may not fully understand the nature of the problem however from my ancient vb archived knowledge

If JJ$ <> "" Then
might work or

If JJ$ = vbNullString Then
might work

In the VB section?.... You could use a 'vararg' type return (I forget the actual name) if NULL was to be possible....

This returns control of any exiting to the VB Code as the parent code

[qWord]

what about FreeLibraryAndExitThread (or simply FreeLibrary)?

qWord

[jj2007]

It is actually a bit trickier:
- VB calls a dll written in Masm
- in the dll code, a fatal error happens
- the dll calls ExitProcess
- MS Word says silently bye

I have debugged the normal return. VB checks if edi==esp on return, if not it raises an error saying Bad DLL calling convention. That could be the best option: Instead of ExitProcess, try a regular return but change edi to raise a VB error.

[donkey]

Have you tried just sending a WM_CLOSE to word ? That way you give it a chance to clean up before it exits.

[jj2007]

Have you tried just sending a WM_CLOSE to word ? That way you give it a chance to clean up before it exits.

Sounds promising. But finding the handle to the right instance is not that simple... ::)

[donkey]

Sounds promising. But finding the handle to the right instance is not that simple... ::)

Oh, I thought you were attached with the DLL, in that case you could have used GetCurrentProcessId to find the process ID then Enumerate the windows checking them against the process id using GetWindowThreadProcessId. Or alternatively you can get the main thread handle and use PostThreadMessage, never used that function myself but it might work.

[qWord]

I'm not an VB expert, but AFAIK:

Code Select Expand

ByVal StrIndex As Longis equivalent to

Code Select Expand

StrIndex: ptr SDWORD.
So removing the 'ByVal' should make it work (mabye ^^).

[jj2007]

Thanks, Edgar. Sounds a bit complicated but probably it is the cleanest option.

In the meantime, I found a dirty hack that works, see attachment. Basically, in case of an error it jumps back to VB with edi different from the value VB expects. So VB complains about a 'bad DLL calling convention' but exits gracefully.

@qWord: It does work already. The only problem is an irregular exit, e.g. if a runtime error occurs.

[hutch--]

JJ,

The ExitProcess() is probably the problem in that your DLL probably does not create a new thread. The ExitProcess() is probably being sent to the parent (Word in this instance) and this shuts it down.

[drizz]

Any idea how that could be handled? Triggering On Error GoTo would be ideal, of course....

Code Select Expand

.if MyIndex<0
  invoke RaiseException ...
.endif


On Error GoTo ...
GetString ...



[jj2007]

The ExitProcess() is probably the problem in that your DLL probably does not create a new thread. The ExitProcess() is probably being sent to the parent (Word in this instance) and this shuts it down.

Hutch,

You are right, ExitProcess shuts down Word in a rather rude manner, without the mechanisms that WM_CLOSE would provide.

Code Select Expand

.if MyIndex<0
  invoke RaiseException ...
.endif


Drizz,

The problem is the RaiseException - at the end, the code must return to where it left VB. My current runtime error routines don't care about the stack (you never know at which level of recursion a file not found happens), they just clean up open handles etc and then call ExitProcess. Works fine for an executable, but VB doesn't like it.

So what I do in the end is store ebp on entry, knowing that ebp=esp+4; and when an error is triggered, I recover the correct esp, zero edi to flag an error, and return to VB. It's an undocumented feature of VB but it works perfectly.

Thanks to all for good advice :U

[sinsi]

Surely if your dll has an error it should just return an error value? ExitProcess isn't for dlls.

Calling ExitProcess in a DLL can lead to unexpected application or system errors. Be sure to call ExitProcess from a DLL only if you know which applications or system components will load the DLL and that it is safe to call ExitProcess in this context.

Make GetString a function with a return value, then handle it in the VB code.

[jj2007]

Surely if your dll has an error it should just return an error value? ExitProcess isn't for dlls.

Sinsi,
The problem is the "return". If you try to Open "I", #1, "NoSuchFileHere.txt", MasmBasic shows you a MsgBox telling you that you are a bad coder, then closes open file handles and exits. That is by design - you can use Exist(MyFile$) if you want it even more gracefully.

But this design implies that you have no control over the current state of the stack. The file not found might happen in your n-th subproc, and in theory you could pass the error message back to all levels but in practice you'll just say get outta here. That works fine with executables, ExitProcess, but VB wants that the call to the dll is being returned. So that is what I am doing, and it works fine, just triggers a "Bad DLL calling convention", and when you click Debug, the VB editor even highlights the line where it happened. Perfect :bg

[drizz]

The problem is ...., MasmBasic

Now we are getting somewhere...  :bg
Why don't you implement seh macros for MasmBasic and have "Open" routine raise an error(exception) instead of "MsgBox".

Simplest seh macros could be written like this:

Code Select Expand

SehHandler PROTO C :DWORD,:DWORD,:DWORD,:DWORD
UnhandledException proto :DWORD

sSEH STRUCT
OrgEsp DD ?
OrgEbp DD ?
SaveEip DD ?
sSEH ENDS

.data?
SEH    sSEH <?>; Single threaded only!

.const

OnErrorBegin macro
LOCAL _AddrOnError
mov SEH.SaveEip, OFFSET _AddrOnError
mov SEH.OrgEbp, ebp
push OFFSET SehHandler
xor edx,edx
push dword ptr fs:[edx]
mov dword ptr fs:[edx], esp
mov SEH.OrgEsp, esp
OnErrorEnd macro
_AddrOnError:
xor edx,edx
pop dword ptr fs:[edx]
add esp,4
endm
endm

SehHandler PROC C pExcept:DWORD,pFrame:DWORD,pContext:DWORD,pDispatch:DWORD
mov eax, pContext
push SEH.OrgEsp
push SEH.OrgEbp
push SEH.SaveEip
pop [eax][CONTEXT.Eip_]
pop [eax][CONTEXT.Ebp_]
pop [eax][CONTEXT.Esp_]
xor eax,eax ;ExceptionContinueExecution
RET
SehHandler ENDP


Code Select Expand

OnErrorBegin
push 10
push 0
mov eax,11
bound eax,dword ptr [esp]
add esp,8
OnErrorEnd