Register Destroyed :/

[oex]

In the following how would esi get destroyed?

oexExample PROC USES esi oexArg1:DWORD

mov esi, alloc(4)

ret

oexExample ENDP

mov esi, 1
invoke oexExample, esi

Error ----> esi is destroyed here :/ Shouldnt esi=1 here?

As I understood it esi would be pushed (and popped) twice so that esi should remain the same before and after the call? Er it is late, lots of caffine, esi might not be popped twice? But it should balance and esi should remain 1 after call?

[clive]

It should be viable, have you looked at the code with DUMPBIN/DUMPPE?

ESI should be saved/restored by the PROC, the parameter from INVOKE is thrown away as "C" is allowed to modify the input parameters without changing the variable they came from in the parent. ESI shouldn't get destroyed, something odd is happening if it does, like something in the PROC is modifying/trashing it on the stack.

Code Select Expand

Microsoft (R) Macro Assembler Version 6.15.8803            10/01/10 23:27:32
test48.asm                             Page 1 - 1


                        .386
                        .MODEL FLAT,C
00000000                    .CODE

00000000            oexExample PROC USES esi oexArg1:DWORD

00000004  BE 0000BEEF                mov esi, 0BEEFh

                        ret

0000000C            oexExample ENDP

0000000C            _start:

0000000C  BE 0000CAFE                mov     esi,0CAFEh

                        invoke oexExample, esi

0000001A  81 FE 0000CAFE            cmp     esi,0CAFEh
00000020  C3                    ret

                        END     _start

Code Select Expand

Disassembly

00000000                    _oexExample:
00000000 55                     push    ebp
00000001 8BEC                   mov     ebp,esp
00000003 56                     push    esi
00000004 BEEFBE0000             mov     esi,0BEEFh
00000009 5E                     pop     esi
0000000A C9                     leave
0000000B C3                     ret
0000000C                    __start:
0000000C BEFECA0000             mov     esi,0CAFEh
00000011 56                     push    esi
00000012 E8E9FFFFFF             call    _oexExample
00000017 83C404                 add     esp,4
0000001A 81FEFECA0000           cmp     esi,0CAFEh
00000020 C3                     ret


[oex]

Yeah I dumped it and thought I must be misunderstanding the pre call push balance.... Something must be wrong in the proc....

if I

mov eax, oexArg1
ret

eax=1

however esi != 1 after call return :(.... It's always the noob issues that hit you at 5:45am :lol

The worst bit is I'm calling an external prebuilt zlib dll in the proc :(

[oex]

invoke oexGZIPInflate, eax, ecx, esi, edi

oexGZIPInflate   PROC   USES esi   oexIn:DWORD, oexInLength:DWORD, oexOut:DWORD, oexOutLength:DWORD

   mov esi, alloc(SIZE Z_STREAM_S)

   m2m [esi].Z_STREAM_S.next_in, oexIn
   m2m [esi].Z_STREAM_S.avail_in, oexInLength
   m2m [esi].Z_STREAM_S.next_out, oexOut
   m2m [esi].Z_STREAM_S.avail_out, oexOutLength

   invoke zlibVersion

   invoke inflateInit2_, esi, 47, eax, SIZE Z_STREAM_S

@@:
   invoke inflate, esi, 4
   eax0 @b

   invoke inflateEnd, esi

   mov eax, oexOut   <---- This works but I shouldnt have to do this.... :/
   mov ecx, [esi].Z_STREAM_S.total_out

   ret

oexGZIPInflate   ENDP

Yeah not freed struct and maybe other errors but esi is being destroyed?....

[BogdanOntanu]

It is very unlikely that ESI is not saved and restored correctly by MASM in a PROC that has the "USES ESI" clause.

It is better to search for errors and assumptions you make in other places :D

For example (not necessarily here but possible) if you overwrite the stack area by mistake then of course that any items saved there are going to become dirty / changed / wrong and not going to be restored corectly anymore.

Then, usually  EAX is to be assumed that it is almost always dirty and not to be assumed to have the same original value after a call to an OS or library or usually any function whatsoever...

Then you should NOT assume that library's like ZLIB follow the same calling conventions like Win32 API does. Maybe that ZIlib's inflate or initialization function does make dirty your precious ESI. Check Zlib's ABI conventions (if any are given)

Always assume that YOU have made an error and then check everything in a paranoid manner...or never assume anything.

It is very rare that a compiler or assemblers has bugs... and usually not in such trivial matters ... hence that is possible but the last thing to consider. (unless the assembler is still under development :)) )

[oex]

ty Bogdan, I'm not sure what I've missed still, I dont expect it to be MASM just assumed I'd missed something obvious in that proc.... Having gone through it multiple times I see nothing so I will test and retest constantly, reposition the code at start of app or empty app etc etc all later when I wake up :lol.... Bound to be something obvious.... The only thing I hadnt considered was that ZLib might dirty esi, I rarely use external dlls so I will consider this for future reference....

It is probably my error, it is a very large ASM app and none of the code is confirmed final build yet.... I have been through your list of suggestions and I havent missed anything I have been doing this all night....

App continues to run without (any noticed) error after this so if it is my error it's probably a nasty hidden one.... Luckily the function call is pretty near the start of the application where I test most of my code initially so there shouldnt be much more than a few k likes to check ::)....

I havent really thought through past dump output but would I find overruns etc by running checks on all app structures for expected data etc, would there be any other easy internal self analysis checks I could setup for my internal debugger? I *think* I could check if data is on stack that shouldnt be etc? I have generally avoided this kind of debugging in the past to try and avoid spagetti code but maybe there are some suggestions for checks that might be usefull?

[japheth]

Hello oex,

you should also consider a calling convention mismatch - that is, you might use a PROTO in your source which is different from the PROC. Such errors may mess up the stack, and the POP <saved_register> will restore the wrong value. Since the proc's epilogue usually contains LEAVE or MOV ESP, EBP, which "cures" the messed value of ESP, you won't get a hard crash, just a wrong register value.

[jj2007]

Why don't you simply run it with Olly, insert an int 3 before the invoke, and see what's happening?

[oex]

Hi JJ,

I already have used int 3 with VC6 however I think the problem could be deeper, caused in a previous proceedure call and undetected until some random and unexpected memory address turns out not to be what it should be.... int 3/debugger is a one dimensional way to check code (at least in the understanding I have of how to check it with the debugger).... ie you can check your current position, set breakpoints etc however I have quite a large app now (just over 1000 functions) and want to do a little more intelligent debugging and structural understanding....

According to VC6 debugger I cant find the issue.... There is *a lot* of code to step through though so I may just not have found it yet....

Hi Japheth,

I think I understand what you are saying, a wrong proto will throw up a compiler error so I'll have to reread your post a few times and have a think about what you're saying.... Not slept last night and looking after a sick dog today so my mind is in limbo atm....

[MichaelW]

oex,

In case you don't already know, the functions in zlib1.dll use the C calling convention.

[oex]

hmm kk ty Michael, I did read this however this may be where my understanding is lacking....

Awesome Yep ty Michael I think I have found my error I was using STDCALL convention.... ty for your input I will test it shortly :bg

Is there a way/hint (maybe with dependancy viewer or something to tell STDCALL from C calling conventions in a dll or do I have to find the appropriate documentation?

[Antariy]

Is there a way/hint (maybe with dependancy viewer or something to tell STDCALL from C calling conventions in a dll or do I have to find the appropriate documentation?

I doubt that my answer would liked to you, because you already post warning to me in "Encryption procedure" thread. But, anyway.
If you don't have header file or any info about convention of function, but you know what it takes one or more parameters, then make this:
1. disassemble file with function, and find the function itself. Find epilogue of function, i.e. - its end. If return from function is by RET command, without an immediate, then this is __cdecl function (C calling convention). IF function have something like RET 4, or RET 20 etc, then this is STDCALL function.
Or:
2. Debug your app to epilogue of function, and see which kind of RET is used.

Sometimes can be functions with other conventions, but this is relatively exotic.



Alex

[MichaelW]

Is there a way/hint (maybe with dependancy viewer or something to tell STDCALL from C calling conventions in a dll or do I have to find the appropriate documentation?

When I have doubts about which it is, I just assume one or the other and then do a test call. If the ESP value before the parameters are stacked matches the ESP value after the call return and stack adjustment, then my assumption was right.

Code Select Expand


    crc32 PROTO C crc:DWORD,pbuff:DWORD,blen:DWORD
    . . .
    mov eax, esp
    print hex$(eax),13,10
    invoke crc32,0,esi,ebx
    print hex$(eax),13,10
    mov eax, esp
    print hex$(eax),13,10,13,10


BTW, I created an import library for zlib1.dll that I can post.

[oex]

Hi Alex,

I'm a stickler for following the rules and our Encryption post was just nudging them slightly :lol, Bogdan graveyarded one of my post replies the other day so dont take it personally :lol.... My guilt was just evident with that post as I was discussing it with the OP before he posted it :/....

ty for the input, i'll have to read more into C vs STDCALL and understand the differences but this certainly gives me an easy indicator to look for :bg

Hi Michael,

Again ty for the awesome hint, I have created a working (I think) import lib now for zlib1.dll, I was up for 2 days straight and havent tested it yet however will check it out once I have been through forum posts/checked mail etc etc

[Antariy]

Hi Alex,

I'm a stickler for following the rules and our Encryption post was just nudging them slightly :lol, Bogdan graveyarded one of my post replies the other day so dont take it personally :lol.... My guilt was just evident with that post as I was discussing it with the OP before he posted it :/....

ty for the input, i'll have to read more into C vs STDCALL and understand the differences but this certainly gives me an easy indicator to look for :bg

No problems  :bg
If you would read documentation, don't forgot other conventions - they used rarely, but used a lot.

Michael's method is more "followed the laws" and forum rules, but if you call an unknown procedure with some parameters, this can cause something like "FORMAT C:"  :bg
But this is still one from best methods.



Alex