ENTER opcode emu

[theunknownguy]

Hey guys, again me trying to understand the ENTER opcode:

Code Select Expand

ENTER 4, 0


I understand that the 4 immed represent the bytes to "allocate" in the stack before entering a procedure (kind of using the common method of sub esp, XX)
The 0 immed represent the number of times i need to push the EBP (stack frame pointer).

So that would be like:

Code Select Expand

Push Ebp
Mov Ebp, Esp
Sub Esp, 4

Now when incrementing the second operand for example:

Code Select Expand

ENTER 4, 2

it would be the same has this?:

Code Select Expand

Push Ebp
Mov StackFrame, ESP
Sub Ebp, 4
Push [Ebp]
Push StackFrame
Sub Esp, 4

Finally the main question, is this correct or i am missing something?  :U

Thanks

[theunknownguy]

This is finally my emulation, can some one see some error:

       

Code Select Expand


        Push Ebp                                                             ;Preserve EBP
Mov Esi, Stack.pCurrentAddr                                          ;Address of the current stack
Mov Ebp, Registers.EBP.Val                                           ;Load EBP virtual register into real EBP reg             
.If (PrefixWord == 1)
    Mov Word Ptr [Esi], Bp                                           ;Emu of PUSH BP
    Sub Esi, 2                                                       ;Adjust Stack
.Else
    Mov [Esi], Ebp                                                   ;Emu of PUSH EBP
    Sub Esi, 4                                                       ;Adjust Stack
.EndIf
Mov Edi, Esi                                                         ;Save on EDI the current stack address
.If (SecondOperand > 0)                                              ;Check for the second operand ENTER XX, YY (YY one)   
   Mov Ebx, 1                                                        ;use EBX has a counter and set it 1
     .Repeat
        .If (PrefixWord == 1)
           Sub Ebp, 2                                                ;-2 to EBP
           Mov Word Ptr [Esi], Bp                                    ;Emu of Push Bp
   Sub Esi, 2                                                ;Adjust Stack
        .Else
   Sub Ebp, 4                                                ;-4 to EBP
   Mov [Esi], Ebp                                            ;Emu of Push EBP
   Sub Esi, 4                                                ;Adjust Stack
.EndIf
     .Until (Ebx == SecondOperand)                                   ;Repeat until SecondOperand ENTER XX, YY (YY one)
     Mov [Esi], Edi                                                  ;Set the stack frame pointer saved on EDI
     Sub Esi, 4                                                      ;Adjust Stack
.EndIf
.If (PrefixWord == 1)
   Mov Word Ptr Registers.EBP.Val, Di                                ;Set on the virtual EBP reg the word of frame pointer saved on EDI
   Sub Esi, FirstOperand                                             ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.Else
   Mov Registers.EBP.Val, Edi                                        ;Set on Virtual EBP reg the dword of the frame pointer saved on EDI
   Sub Esi, [Eax]                                                    ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.EndIf
Mov Registers.ESP.Val, Esi                                           ;Finally set the ESI value has the current Stack pointer on the ESP virtual reg
        Mov Stack.pCurrentAddr, Esi                                          ;Set the ESI has the current Stack pointer
        Pop Ebp                                                              ;Return EBP

Yes, i do the hardwork of align the commentaries for the code... some one know if there is an option to align the source inside ["code"] tags?

[clive]

You are handling 16-bit quantities as 32-bit, they won't wrap correctly.

You load 16-bit values into 32-bit registers with no regard for the prior content of the upper portion of the register.

The nesting level is modulo 32, as I recall.

It's been a while since I looked at the ENTER instruction, but I'm pretty sure the nesting mechanism is more complex than this. Will go review some better documentation, but it is PASCAL related and walks up the stack frames as I recall.

This is finally my emulation, can some one see some error:

       

Code Select Expand


        Push Ebp                                                             ;Preserve EBP
Mov Esi, Stack.pCurrentAddr                                          ;Address of the current stack
Mov Ebp, Registers.EBP.Val                                           ;Load EBP virtual register into real EBP reg             
.If (PrefixWord == 1)
    Mov Word Ptr [Esi], Bp                                           ;Emu of PUSH BP
    Sub Esi, 2                                                       ;Adjust Stack
.Else
    Mov [Esi], Ebp                                                   ;Emu of PUSH EBP
    Sub Esi, 4                                                       ;Adjust Stack
.EndIf
Mov Edi, Esi                                                         ;Save on EDI the current stack address
.If (SecondOperand > 0)                                              ;Check for the second operand ENTER XX, YY (YY one)   
   Mov Ebx, 1                                                        ;use EBX has a counter and set it 1
     .Repeat
        .If (PrefixWord == 1)
           Sub Ebp, 2                                                ;-2 to EBP
           Mov Word Ptr [Esi], Bp                                    ;Emu of Push Bp
   Sub Esi, 2                                                ;Adjust Stack
        .Else
   Sub Ebp, 4                                                ;-4 to EBP
   Mov [Esi], Ebp                                            ;Emu of Push EBP
   Sub Esi, 4                                                ;Adjust Stack
.EndIf
     .Until (Ebx == SecondOperand)                                   ;Repeat until SecondOperand ENTER XX, YY (YY one)
     Mov [Esi], Edi                                                  ;Set the stack frame pointer saved on EDI
     Sub Esi, 4                                                      ;Adjust Stack
.EndIf
.If (PrefixWord == 1)
   Mov Word Ptr Registers.EBP.Val, Di                                ;Set on the virtual EBP reg the word of frame pointer saved on EDI
   Sub Esi, FirstOperand                                             ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.Else
   Mov Registers.EBP.Val, Edi                                        ;Set on Virtual EBP reg the dword of the frame pointer saved on EDI
   Sub Esi, [Eax]                                                    ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.EndIf
Mov Registers.ESP.Val, Esi                                           ;Finally set the ESI value has the current Stack pointer on the ESP virtual reg
        Mov Stack.pCurrentAddr, Esi                                          ;Set the ESI has the current Stack pointer
        Pop Ebp                                                              ;Return EBP

Yes, i do the hardwork of align the commentaries for the code... some one know if there is an option to align the source inside ["code"] tags?

[clive]

The sematics go something like this, where the depth is modulo 32, the local frame size is sign extended (16-bit to 32-bit), and EFP is an internal holding register for the frame point.

It copies data from a prior frame, and assumes the nesting level doesn't exceed the parents level+1

Code Select Expand

ENTER 8,0

  PUSH EBP
  MOV EFP,ESP
  SUB ESP,8
  MOV EBP,EFP

ENTER 16,1

  PUSH EBP
  MOV EFP,ESP
  PUSH EFP
  SUB ESP,16
  MOV EBP,EFP

ENTER 12,2

  PUSH EBP
  MOV EFP,ESP
  PUSH [EBP-4]
  PUSH EFP
  SUB ESP,12
  MOV EBP,EFP

ENTER 20,3

  PUSH EBP
  MOV EFP,ESP
  PUSH [EBP-4]
  PUSH [EBP-8]
  PUSH EFP
  SUB ESP,20
  MOV EBP,EFP

ENTER 64,6

  PUSH EBP
  MOV EFP,ESP
  PUSH [EBP-4]
  PUSH [EBP-8]
  PUSH [EBP-12]
  PUSH [EBP-16]
  PUSH [EBP-20]
  PUSH EFP
  SUB ESP,64
  MOV EBP,EFP


[theunknownguy]

Ok so the only problem i see compared to your description is the modulo 32 on the level of nesting and i cant seems to understand the 16 bit problem you say, i am testing on debugger:

Code Select Expand

PUSH BP

Stack view:

0013FFC2   7067FFF0

Code Select Expand


Mov Word Ptr Ss: [ESP], Bp

Stack View:

0013FFC2   7067FFF0

The same happens for set a register with its 16 bit part or i am wrong?.

Here is the modulo applied. Also the EFP i didnt make a local for it, i just store it on EDI and use it later.


       

Code Select Expand


        Xor Edx, Edx
        Mov Eax, FirstOperand
        Mov Ecx, 32
        Div Ecx
        Mov FirstOperand, Edx
Mov Ebp, Registers.EBP.Val                                           ;Load EBP virtual register into real EBP reg             
.If (PrefixWord == 1)
    Mov Word Ptr [Esi], Bp                                           ;Emu of PUSH BP
           
    Sub Esi, 2                                                       ;Adjust Stack
.Else
    Mov [Esi], Ebp                                                   ;Emu of PUSH EBP
    Sub Esi, 4                                                       ;Adjust Stack
.EndIf
Mov Edi, Esi                                                         ;Save the EFP
.If (SecondOperand > 0)                                              ;Check for the second operand ENTER XX, YY (YY one)   
   Mov Ebx, 1                                                        ;use EBX has a counter and set it 1
     .Repeat
        .If (PrefixWord == 1)
           Sub Ebp, 2                                                ;-2 to EBP
           Mov Word Ptr [Esi], Bp                                    ;Emu of Push Bp
   Sub Esi, 2                                                ;Adjust Stack
        .Else
   Sub Ebp, 4                                                ;-4 to EBP
   Mov [Esi], Ebp                                            ;Emu of Push EBP
   Sub Esi, 4                                                ;Adjust Stack
.EndIf
     .Until (Ebx == SecondOperand)                                   ;Repeat until SecondOperand ENTER XX, YY (YY one)
     Mov [Esi], Edi                                                  ;Set the stack frame pointer saved on EDI
     Sub Esi, 4                                                      ;Adjust Stack
.EndIf
.If (PrefixWord == 1)
   Mov Word Ptr Registers.EBP.Val, Di                                ;Set on the virtual EBP reg the word of frame pointer saved on EDI
   Sub Esi, FirstOperand                                             ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.Else
   Mov Registers.EBP.Val, Edi                                        ;Set on Virtual EBP reg the dword of the frame pointer saved on EDI
   Sub Esi, [Eax]                                                    ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.EndIf
Mov Registers.ESP.Val, Esi                                           ;Finally set the ESI value has the current Stack pointer on the ESP virtual reg
        Mov Stack.pCurrentAddr, Esi                                          ;Set the ESI has the current Stack pointer
        Pop Ebp                                                              ;Return EBP

Thanks clive  :U

[clive]

Well nearly. Please review again the equivalents posted earlier.

It doesn't handle the Depth = 1 case properly (the copy is Depth-1), pulling from the prior frame, and storing the current (nominally in EBP, but now stored in the frame for the next child)

It doesn't copy from the prior frame, you are writing EBP, rather than [EBP] (ie the content of the parent's frame, not the decrementing value of EBP)

I'm not 100% sure of the prefixed operation (16-bit @ 32) and (32-bit @ 16), but I'm not sufficiently invested to test all the combinations. It will be a mess because the pointers, and the things being copied really should be 32-bit in 32-bit mode, it doesn't make sense for them to be 16-bit.

Code Select Expand

        .If (PrefixWord == 1)
           SUB BP, 2                                        ;-2 to BP
MOV AX,[BP]
           Mov Word Ptr [Esi], AX                    ;Emu of Push Bp
   Sub Esi, 2                                                ;Adjust Stack
        .Else
   Sub Ebp, 4                                                ;-4 to EBP
MOV EAX,[EBP]
   Mov [Esi], EAX                                            ;Emu of Push EBP
   Sub Esi, 4                                                ;Adjust Stack
.EndIf


Code Select Expand


        Mov Ecx, 32
        Div Ecx

or

   AND ECX,(32 - 1)
       

[theunknownguy]

Well nearly. Please review again the equivalents posted earlier.

It doesn't handle the Depth = 1 case properly (the copy is Depth-1), pulling from the prior frame, and storing the current (nominally in EBP, but now stored in the frame for the next child)

It doesn't copy from the prior frame, you are writing EBP, rather than [EBP] (ie the content of the parent's frame, not the decrementing value of EBP)

I'm not 100% sure of the prefixed operation (16-bit @ 32) and (32-bit @ 16), but I'm not sufficiently invested to test all the combinations. It will be a mess because the pointers, and the things being copied really should be 32-bit in 32-bit mode, it doesn't make sense for them to be 16-bit.

Code Select Expand

        .If (PrefixWord == 1)
           SUB BP, 2                                        ;-2 to BP
MOV AX,[BP]
           Mov Word Ptr [Esi], AX                    ;Emu of Push Bp
   Sub Esi, 2                                                ;Adjust Stack
        .Else
   Sub Ebp, 4                                                ;-4 to EBP
MOV EAX,[EBP]
   Mov [Esi], EAX                                            ;Emu of Push EBP
   Sub Esi, 4                                                ;Adjust Stack
.EndIf


Code Select Expand


        Mov Ecx, 32
        Div Ecx

or

   AND ECX,(32 - 1)
   

Oh ye now i get it, thanks for remind me the [EBP] stuff, i made this last day 4 AM so i was kind of sleep  :lol

About the 16 bit, i know it should not be used for 32 bit, but my thing is mostly an scanner, and obfuscated code sometimes use technique for confuse tracers/scanners/ etc. Most of these techniques are done with 16 bit opcodes expecting that the analyzer of code didnt have emulated the 16 bit counterpart.

Now this could be final version, hope i havent make any other mistake:

       

Code Select Expand


        Xor Edx, Edx
        Mov Eax, SecondOperand
        And Eax, 31
        Sub Eax, 1                                                                   ;(SecondOperand MODULE 32) -1
        Mov SecondOperand, Eax
Mov Ebp, Registers.EBP.Val                                           ;Load EBP virtual register into real EBP reg             
.If (PrefixWord == 1)
            Sub Esi, 2                                                       ;Adjust Stack
    Mov Word Ptr [Esi], Bp                                           ;Emu of PUSH BP
.Else
            Sub Esi, 4                                                       ;Adjust Stack
    Mov [Esi], Ebp                                                   ;Emu of PUSH EBP
.EndIf
Mov Edi, Esi                                                         ;Save the EFP
.If (SecondOperand > 0)                                              ;Check for the second operand ENTER XX, YY (YY one)   
   Mov Ebx, 1                                                        ;use EBX has a counter and set it 1
     .Repeat
        .If (PrefixWord == 1)
                   Sub Esi, 2                                                ;Adjust Stack
                   Sub Ebp, 2
           Movsx Ebp, Word Ptr [Ebp]
           Mov Word Ptr [Esi], Bp                                    ;Emu of Push [Bp]
        .Else
                   Sub Esi, 4                                                ;Adjust Stack
                   Sub Ebp, 4
   Mov Ebp, [Ebp]
   Mov [Esi], Ebp                                            ;Emu of Push [EBP]
.EndIf
     .Until (Ebx == SecondOperand)                                   ;Repeat until SecondOperand ENTER XX, YY (YY one)
     Sub Esi, 4                                                      ;Adjust Stack
     Mov [Esi], Edi                                                  ;Set the stack frame pointer saved on EDI
.EndIf
.If (PrefixWord == 1)
   Mov Word Ptr Registers.EBP.Val, Di                                ;Set on the virtual EBP reg the word of frame pointer saved on EDI
   Sub Esi, FirstOperand                                             ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.Else
   Mov Registers.EBP.Val, Edi                                        ;Set on Virtual EBP reg the dword of the frame pointer saved on EDI
   Sub Esi, FirstOperand                                                    ;Adjust Stack with the FirtsOperand ENTER XX, YY (XX one)
.EndIf
Mov Registers.ESP.Val, Esi                                           ;Finally set the ESI value has the current Stack pointer on the ESP virtual reg
        Mov Stack.pCurrentAddr, Esi                                          ;Set the ESI has the current Stack pointer
        Pop Ebp                                                              ;Return EBP

And god please some one tell me if there is a trick for align the commentaries inside the CODE tags...

Thanks clive

PS: When making the PUSH [EBP] emulation you are assuming that EBP points to a valid address, i guess i need to add a security check for the address to be valid at the time of making the PUSH [EBP].

[dedndave]

you can use...

Code Select Expand

[pre][/pre]
around pre-formatted text, if you like
that should allow proper expansion of tabs