Unexpected 'Invalid Function Error' from WriteFile

[Phil]

I've taken some code from the masm32.lib definition of StdOut as the basis for the following code. I was surprised to see that WriteFile returns 1 in eax rather than the 0 which I had initially expected. The documentation that I'd found indicated that it returns a zero to indicate that no errors occured but that's not what happened. Could anyone point out why this is happening?

I was originally trying to figure out why my Fibonacci program works fine when I redirectect large output requests (61K+) to a file but fails to display anything when directed to the console or the 'more' command thru a pipe. The documenations claims there is a 64K limit to named pipes but it was failing below that. I was using 'invoke StdOut' so I thought I'd break it down a little and find out what was happening. When it fails, WriteFile returns a '0' in bWritten to indicate that no bytes were written but, it also returns a '0' in eax. That surprised me too.

While I was working on the program a 13 was returned in eax when I mistakenly passed bWritten as a pointer instead of the ADDR of a pointer. The 13 in eax corresponded to ERROR_INVALID_DATA which made sense and helped me figure out what I'd done wrong. What are the normal and error return conditions for WriteFile?

As written, the code just displays 'Unexpected ERROR_INVALID_FUNCTION' and exits. I've included extra diagnostic code in the WriteError routine so anyone can play with it and see the returned values by removing the conditional that checks for the unexpected value in eax. It assumes, correctly I believe, that ebx will be preserved across the WriteFile proceedure call. I think that is the case with almost all Windows system calls.

Code Select Expand


.386
.model flat,stdcall
option casemap:none

include     \masm32\include\windows.inc
include     \masm32\include\kernel32.inc
include     \masm32\include\masm32.inc
includelib  \masm32\lib\kernel32.lib
includelib  \masm32\lib\masm32.lib

include \masm32\macros\macros.asm

.data
Msg db 'Unexpected ERROR_INVALID_FUNCTION',13,10
msglen equ $ - Msg

.data?
hStdOut dd ?
bWritten dd ?

.code

start:  invoke  GetStdHandle,STD_OUTPUT_HANDLE
        mov     hStdOut,eax
        call    WriteString
        invoke  ExitProcess,0

WriteString:
        mov     ebx,msglen
        invoke  WriteFile,hStdOut,ADDR Msg,ebx,ADDR bWritten,NULL
        cmp     ebx,bWritten
        jne      WriteError
        cmp     eax,ERROR_INVALID_FUNCTION
        jneWriteError
        ret

WriteError:
        push    ebx
        push    eax
        print   chr$("Write error: ");
        pop     eax
        print   ustr$(eax)
        print   chr$(13,10);
        print   chr$("Bytes requested: ");
        pop     ebx
        print   ustr$(ebx)
        print   chr$(13,10);
        print   chr$("Bytes written: ");
        mov     eax,bWritten
        print   ustr$(eax);
        print   chr$(13,10);
        ret

        end start


[QvasiModo]

Hi :)

WriteFile returns zero on error and nonzero on success, like most Win32 API calls. The ERROR_* equates are unrelated here, they're used with the GetLastError API, not this one.

BTW you are correct, EBX is preserved across API calls, as well as ESI, EDI, EBP, and the direction flag. :)

Here are a couple links to the MSDN docs:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/writefile.asp

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/debug/base/getlasterror.asp

Hope this answers your questions. Good luck with that Fibonacci program! :U

[Phil]

Thanks so much for the clarification and links. The SDK link was the first one that I'd managed to find when I was searching for info but, I guess my i's were crossed and my t's must have been dotted when I read:

WriteFile Return Values
If the function succeeds, the return value is nonzero.

If the function fails, the return value is zero. To get extended error information, call GetLastError.