Linker Subsystem

[RedXVII]

I was having trouble with linking the basic code below, i was building it and nothing appeared after execution - my trouble was i was accidently useing:

link.exe /SUBSYSTEM:WINDOWS hello.obj

when it should be

link.exe /SUBSYSTEM:CONSOLE hello.obj

My question is - why does this make a difference?
I pulled up the faithful old hex editor and only 2 bytes had acutally changed between the two versions (of which i have no idea what they represent). But only the latter displayed correctly! Just curious really.

Thanks alot  :U

Code Select Expand

.386
.model flat,stdcall
option casemap:none

include \masm32\include\windows.inc
include \masm32\include\kernel32.inc
include \masm32\include\masm32.inc

includelib \masm32\lib\kernel32.lib
includelib \masm32\lib\masm32.lib

.data

hello db "Hello world!!",0

.code

start:

invoke StdOut, addr hello
invoke ExitProcess,0

end start

[tenkey]

I haven't checked with Win2000 and XP.
But under Win9x, executables created with /SUBSYSTEM:WINDOWS will not start with standard input, output, and error files opened.

[hutch--]

Regularly making the same mistake while developing console apps, it appears that you must have a console to point console specific API code at. I regularly get caught with using a keystroke exit for console mode and accidentally building a normal GUI app so it does not close witout task manager.

[RedXVII]

Hmm

Well, i just opend up each of them in olly to see what was happening. I just noticed that the one with /SUBSYSTEM:CONSOLE loads up a command prompt while loading the PE. The other one however doesnt load up a console. Other than that - they are completely the same code. It must be some sort of windows execution thingy that it reads a byte in the PE which tells it to run in a console window or not.

Errr, sooo, yeah  :bg

[PBrennick]

Hello RedXVII,
The best way to explain it is that because we do not create a window in a console app the code has nowhere to do the output.  The console, in this case, is the window and is created for us.

Paul