Prologue and Epilogue headaches.

[FunkyMeister]

First, let me give the situation that will help people understand:

I wrote some CRC32 code, and functions to find them in a list, with some features.  Designed to be vtable patched into a VB6 app.  Now, with DEP (New machines with XP seem to be the ONLY ones complaining, Vista doesn't), I can't do this.  So, I figured I'd write a DLL (for the first time).  I moved all the code in after creating a DLL using QEditor (I still happen to like it, even though it's quirky, it still works).  After converting the code to use variables, removed my own Prologues and Epilogues in favor of the automatically generated ones, I assembled the code.  I then used the VB app to call the code, as a test.  OllyDebug popped up instantly, which was rather suprising, since it technically shouldn't at all!  Showed me that the ESI was aimed at 0, yet when it was vtable patched by me, the code received the proper data from the stack.

This code:


CRC32       PROC FAR C EXPORT USES ebx ecx edx esi Item:DWORD
Setup:      mov     esi, dword ptr [Item]       ; Unicode Item String [Pointer] (was esp+14h)


Translates to this:

push ebp
mov ebp, esp
push ebx
push ecx
push edx
push esi
move esi, dword ptr [ebp+0c] *


*  The offset is wrong, it should be +08!  I checked, the string address is at EBP+8 on the stack.

So, I went in and hex edited the code, changed the +C to a +8.  And the code actually DID do it's job as before, without any headaches. The only headache it had was at the end of the code, which caused an access violation at FFFFFFFh on the RETF.

The Epilogue for the function is:

pop esi
pop edx
pop ecx
pop ebx
leave
retf


I've read that the leave instruction is a baaad thing to have.  I'd like to resolve the issue with the stack offsets being wrong.  To me it looks like it's pushing ebp on the stack and not taking it off and over compensating for it in the variable offsets.  I'm using MASM V8.2 with this, does anyone know if 8.2 had any problems with this?  I have the latest ready to go in, but if it's not going to help, I'm not going to bother with it.

I really want to get this all sorted out so I can continue to write more libraries, as I have a large amount of code I want to start putting into libraries.  Anyone care to help?

[BogdanOntanu]

Well, I do not think it is MASM's fault...

CRC32       PROC FAR C EXPORT USES ebx ecx edx esi Item:DWORD
Setup:      mov     esi, dword ptr [Item]       ; Unicode Item String [Pointer] (was esp+14h)

Try to take the "FAR" keyword out from your PROC definition ....

In a flat memory model in user mode you do not have to push both the selector and the return address. User mode code should not do this.

And trust me LEAVE is a very good thing to have in an standard epilogue. Once you become more advanced you could make your own stack frames or remove them and use ESP. However for many cases this is not worth the trouble and potential bugs and the standard prologue and epilogue are just fine.

First mistake of beginers: "...the compiler must be doing something wrong..."
Answer: NO, it is 99.999% your fault.

Sometimes the compilers do make mistakes... but not today.

[FunkyMeister]

If I take the FAR out, I can't use the library at all with Visual Basic, Bad DLL Calling Convention errors (49).  Microsoft states that it's caused by mismatched arguments.  EAX will have the output, I know I protected everything in the functions I was using and I'm not storing any data in anywhere, proto'ed the items correctly, so I'm at a loss.  The missing FAR's put the stack order right, except now its unusable.

As for the Leave instruction, I found an issue pertaining to x64 migration, not found any references to whether or not it's been fixed or not.

I wasn't thinking it was MASM's fault, more over if my copy of it may have had problems (v8.2) with what I was trying, since it's a few years old.

[MichaelW]

In addition to incorrectly specifying the FAR attribute, your PROC statement is specifying the C calling convention.

[BogdanOntanu]

As for the Leave instruction, I found an issue pertaining to x64 migration, not found any references to whether or not it's been fixed or not.

In addition there is no problem with LEAVE instruction in 64 bits. As per Intel original CPU manuals in 64 bits the LEAVE instruction is supported and operates in similar ways to 32 bits but with the extended registers RBP and RSP as expected.

[FunkyMeister]

In addition to incorrectly specifying the FAR attribute, your PROC statement is specifying the C calling convention.

Well, I'm coming from a Motorola chipset assembler knowledge (for quite some years), but I've written only code segments for x86 machines.  The x64 + Leave instruction issue I found was for x64 processors (amd64).  I know it's not Intel, but someone out there will be upset when my library fails on their machine.

From what I can tell the user32.dll is written in C (hence the C calling convention in my PROC statements, I assumed the user32.dll was using C calling convention).  Which one should I be using?  As I am not sure which the user32.dll is using.

I am also new to "letting MASM do some of the work", as I still don't use .if, etc.  I'm a bit "old school" so to speak, back when we had to write assembler code directly in raw memory, then dump that to disc.

[MichaelW]

The norm for Windows is STDCALL. So assuming your .model directive is:

.model flat, stdcall

Then something like this should work:

CRC32 PROC EXPORT USES ebx ecx edx esi Item:DWORD

[FunkyMeister]

The norm for Windows is STDCALL. So assuming your .model directive is:

.model flat, stdcall

Then something like this should work:

CRC32 PROC EXPORT USES ebx ecx edx esi Item:DWORD

I did try that, this time around I'm seeing it working, but I had to actually look at the PE of the DLL, the functions are _(name)@(dword length).  Is there any way to get the naming convention back to normal?  I do know that STDCALL is normal for Windows, just don't understand how User32.dll doesn't have the function naming conventions I got when I assembled it with STDCALL, which QEditor setup by default.

[hutch--]

Funky,

That is called a decorated name which is normal for STDCALL functions. The linker uses this data for argument byte counts.

Code Select Expand


MessageBoxA@16    ; 4 times 4 bytes = 16


[FunkyMeister]

That is called a decorated name which is normal for STDCALL functions. The linker uses this data for argument byte counts.

Code Select Expand


MessageBoxA@16    ; 4 times 4 bytes = 16


Is there any way to get clean call names aside from the _(name)@(bytes needed).  I'm not sure if there's any requirements for that to be present, it's not a big issue, but I was hoping for it to look less ugly and a bit more clean.  Code works just the same (after a small mistake fix with respect to using variables).

I'm looking for a reference in the MASM docs to apply information to the DLL (so when someone displays the properties, it says something).  Once I've got this all done, I'll empty it out and use it as a shell dll project for others.

[MichaelW]

I have not used VB6 in a long time and don't even have it installed anymore, but I think you should be able to correct the procedure name problems by just declaring them correctly in the VB app.

[FunkyMeister]

I have not used VB6 in a long time and don't even have it installed anymore, but I think you should be able to correct the procedure name problems by just declaring them correctly in the VB app.

Oh, I have, though I actually solved the problem with the procedure naming, I removed the EXPORT from the .asm and added EXPORTS into the .def file.  And now the references are "clean" looking, now if I can just find references on how to add the library version header to it, then I'd be happy.