Bug with vararg functions on x64.

[Damos]

just thought i'd give y'all the heads up on varargs. just been trying to use printf and found that it trashed part of the stack, and this is why:
the api function attempts to store rcx,rdx,r8 and r9 to the stack even if you only use 2 args. I presume that this is how all vararg functions are dealt with by the m$ team but according to m$ documentation the caller treats varargs like non-varargs.
Anyway I worked around this by adusting the stack myself.
Can't think of any other varargs to check out. any suggestions?

[GregL]

Damos,

In x64 the C Run-Time Library is __fastcall, not __cdecl.  This really messed me up for a while.  :bg

I have used printf in my x64 programs, and if I have 4 or less arguments I haven't needed to adjust the stack since nothing is pushed.  Although I have reserved space on the stack for the largest number of parameters that will be called in the procedure, at the start of the procedure.  Also the stack must be kept 16-byte aligned.

I haven't seen the case where the printf function stores rcx, rdx, r8 and r9 to the stack.  Maybe it's because I reserved the space on the stack at the start of the procedure.  Are you doing this?

I am definitely not an expert on x64 assembly, just trying to learn it and make sense of it all.

Are you using ML64?

[Damos]

If you step into printf with you're debugger you will see:

Code Select Expand

000007FEFF247E28 488BC4 msvcrt.printf: mov rax,rsp
000007FEFF247E2B 48894808 mov [rax+08],rcx
000007FEFF247E2F 48895010 mov [rax+10],rdx
000007FEFF247E33 4C894018 mov [rax+18],r8
000007FEFF247E37 4C894820 mov [rax+20],r9
000007FEFF247E3B 53         push rbx
000007FEFF247E3C 56         push rsi
000007FEFF247E3D 4883EC38 sub rsp,38
000007FEFF247E41 33C0                 xor eax,eax
000007FEFF247E43 4885C9 test rcx,rcx
000007FEFF247E46 0F95C0 setnz al
000007FEFF247E49 85C0         test eax,eax
000007FEFF247E4B 0F84FA580100         jz 000007FEFF25D74B
000007FEFF247E51 488D742458 lea rsi,[rsp+58]
000007FEFF247E56 488D1553AC0800 lea rdx,[000007FEFF2D2AB0] ; []=0000000000000000
000007FEFF247E5D B901000000 mov ecx,00000001
000007FEFF247E62 E8699CFFFF call 000007FEFF241AD0
000007FEFF247E67 90         nop
000007FEFF247E68 488D0D41AC0800 lea rcx,[000007FEFF2D2AB0] ; []=0000000000000000

as I said all the first four arguments are stored onto stack even if you don't use them, makes sense I suppose; how can printf know if it has 1 arg or 4.
it probably didn'n effect you cause as you said:

I have reserved space on the stack for the largest number of parameters that will be called in the procedure, at the start of the procedure.

I didn't, I was using fasm's invoke macro wich alters the stack on a call by call basis.

[sinsi]

Even if a function has no parameters, you need to leave 32 bytes for the function's use (spill?).
With a 16-byte align for the stack, the function relies on the stack at [esp+8] to be aligned.

Interesting that C is using this calling convention. Who forced it? MS/intel/amd? Or a language?

>I am definitely not an expert on x64 assembly, just trying to learn it and make sense of it all.
ditto man, ditto  :(

[GregL]

Damos,

I just stepped through a test program that uses printf, RSP is the same before and after the call to printf.  I stepped into printf and I didn't see anything like what you are seeing.  I am using ML64 and Visual Studio 2010 Beta.

[GregL]

Interesting that C is using this calling convention. Who forced it? MS/intel/amd? Or a language?

I'm not sure, but it was decided that there would only be one calling convention in x64. 

I can't remember where I read about that at the moment.

[Damos]

Greg:
what's it like coding assembly in Visual Studio?
I've got legal VS 8 collecting dust somewhere, but, at the moment using fasm on radasm.
hows the debugger? can you step through your own x64 source code?
does it do quick compiles?
Does the text editor have syntax highlighting and autocomplete and such as in when your coding 'c'?
y'know what I think i'll just install it and find out for myself!
by the way regarding the printf thing my code uses the msvcrt.dll you're code is statically linking probably.
the good thing about using DLL's is that it keeps you're exe size down: size of exe's is a bit of an thing with me.

[GregL]

Damos,

what's it like coding assembly in Visual Studio?

I mainly use Visual Studio for the debugger, which in my opinion is the best debugger around for x64 (and 32-bit).  Visual Studio is a little bit cumbersome for assembly projects, but it works pretty well once you get it set up correctly.  ML64 is integrated into Visual Studio as the assembler for x64.  Yes, you can step through your own x64 source code in the debugger, and view the assembled code at the same time.

To use just the debugger, if you have an executable that was built with debug information, open it with File>Open>Project/Solution and then just step into it.  You should see your source code along with the assembled code.

Here's what it looks like:


;------------------------------------------------------
EXTRN GetStdHandle:PROC
EXTRN WriteFile:PROC
EXTRN printf:PROC
EXTRN _getch:PROC
EXTRN ExitProcess:PROC

INCLUDELIB kernel32.lib
INCLUDELIB user32.lib
INCLUDELIB msvcrt.lib

.DATA

    szMsg     BYTE "Hello x64 World!", 13, 10, 0
       
.CODE

;------------------------------------------------------
    main PROC

        sub rsp, 32
000000013F3A1030  sub         rsp,20h 
       
        lea rcx, szMsg
000000013F3A1034  lea         rcx,[szMsg (13F3A6000h)] 
        call StdOut
000000013F3A103B  call        StdOut (13F3A104Ch) 
       
        call WaitKey
000000013F3A1040  call        WaitKey (13F3A1090h) 
       
        xor ecx, ecx        ; exit code = 0
000000013F3A1045  xor         ecx,ecx 
        call ExitProcess
000000013F3A1047  call        ExitProcess (13F3A111Ah) 

    main ENDP
;------------------------------------------------------
    StdOut PROC
000000013F3A104C  push        rbp 
000000013F3A104D  mov         rbp,rsp 
000000013F3A1050  add         rsp,0FFFFFFFFFFFFFFE8h 
   
        ; int StdOut(char* pText);
       
        LOCAL pText:QWORD
        LOCAL hFile:QWORD
        LOCAL dwLen:DWORD
        LOCAL dwBytesWritten:DWORD
       
        sub rsp, 32
000000013F3A1054  sub         rsp,20h
       
        mov pText, rcx
000000013F3A1058  mov         qword ptr [pText],rcx
        call lstrlen
000000013F3A105C  call        lstrlen (13F3A1108h)
        mov dwLen, eax
000000013F3A1061  mov         dword ptr [dwLen],eax

        mov ecx, -11        ; STD_OUTPUT
000000013F3A1064  mov         ecx,0FFFFFFF5h
        call GetStdHandle
000000013F3A1069  call        GetStdHandle (13F3A110Eh) 
        mov hFile, rax
000000013F3A106E  mov         qword ptr [hFile],rax 

        mov rcx, hFile
000000013F3A1072  mov         rcx,qword ptr [hFile] 
        mov rdx, pText       
000000013F3A1076  mov         rdx,qword ptr [pText] 
        mov r8d, dwLen       
000000013F3A107A  mov         r8d,dword ptr [dwLen] 
        lea r9, dwBytesWritten
000000013F3A107E  lea         r9,[dwBytesWritten] 
        call WriteFile   
000000013F3A1082  call        WriteFile (13F3A1114h) 
       
        mov eax, dwBytesWritten
000000013F3A1087  mov         eax,dword ptr [dwBytesWritten]
       
        add rsp, 32
000000013F3A108A  add         rsp,20h
       
        ret
000000013F3A108E  leave 
000000013F3A108F  ret 
    StdOut ENDP     
;------------------------------------------------------       
    WaitKey PROC
000000013F3A1090  push        rbp 
000000013F3A1091  mov         rbp,rsp 
000000013F3A1094  add         rsp,0FFFFFFFFFFFFFFF8h 
   
        ; int WaitKey(void);
       
        LOCAL dwCrLf:DWORD
        LOCAL dwChar:DWORD
       
        .DATA
       
            szPrompt BYTE 13,10,"Press any key to exit ... ", 0     
           
        .CODE
   
            sub rsp, 32
000000013F3A1098  sub         rsp,20h

            mov dwCrLf, 00000A0Dh
000000013F3A109C  mov         dword ptr [dwCrLf],0A0Dh 
           
            lea rcx, szPrompt       
000000013F3A10A3  lea         rcx,[szPrompt (13F3A6013h)] 
            call printf
000000013F3A10AA  call        printf (13F3A1120h) 
            call _getch
000000013F3A10AF  call        _getch (13F3A1126h) 
            cmp eax, 0
000000013F3A10B4  cmp         eax,0 
            je again
000000013F3A10B7  je          again (13F3A10C2h)
            cmp eax, 0E0h
000000013F3A10B9  cmp         eax,0E0h
            je again
000000013F3A10BE  je          again (13F3A10C2h) 
            jmp @F
000000013F3A10C0  jmp         again+5 (13F3A10C7h) 
         again:     
            call _getch
000000013F3A10C2  call        _getch (13F3A1126h) 
         @@:
            mov dwChar, eax
000000013F3A10C7  mov         dword ptr [dwChar],eax 
            lea rcx, dwCrLf
000000013F3A10CA  lea         rcx,[dwCrLf] 
            call printf       
000000013F3A10CE  call        printf (13F3A1120h)
           
            mov eax, dwChar
000000013F3A10D3  mov         eax,dword ptr [dwChar]
            add rsp, 32
000000013F3A10D6  add         rsp,20h 
             
            ret
000000013F3A10DA  leave 
000000013F3A10DB  ret 
;------------------------------------------------------       

If your executable does not have debug information, you will just see the assembled code, no source code and no symbols.

You may have to tweak some settings to display both the source code and the disassembly.  I am using VS 2010 Beta but VS 2008 is very similar.  VS 2010 Beta has a little better support for x64 projects.

by the way regarding the printf thing my code uses the msvcrt.dll you're code is statically linking probably.

I am dynamically linking to msvcrt.dll too, not statically linking.

[Rockoon]

With regards to the calling convention

I believe that this was Microsofts decision, and has to do specifically with the efficiency of exception handling, which often requires a bit of stack walking.