Installation changes.

[KeepingRealBusy]

What happens on an install of MASM32?

I recently installed MASM32 and up to now have just run a few of the Timings and counts. I then tried to bring up an old piece of code and assemble it to complete some changes I had made and not yet completed. I then tried to execute the code and it died very early in the game, right after my initialization started. The program just died, it did not crash. I finally tracked this down to a call to CreateFileA. I make this call checking for file presence, and the program just quits. I have looked at the .LST file, and all of the pushes and pops look correct. I inserted a DIV by 0, and executed, and got into VS. I skipped by the DIV and tried to execute the CreateFileA. This hung VS, had to use Task Manager to kill it.

I have run into problems with kernel32.lib before and queried MS (several years back) trying to get info on the API 'SetConsoleMaximumWindowSize" which had no MSDN documentation. I used it and wanted to get the documentation. I was informed that the API did nothing. I disagreed, I used it to keep the user from exceeding a console display width once initialized. I could use the API and when I tried to exceed the size I got an error. The .LIB and .DLL WERE working. Then, I downloaded ORCAS Beta2 (VS2008 Beta). Suddenly, I could not assemble my application any more, came up with an undefined. By resorting to the pre-beta .LIB I could assemble and verify correct operation - the .DLL was still there, they only deleted the .LIB entry!

What I am worried about is that the installation of MASM32 has installed a new kernel32.LIB, and that .LIB is based on a .DLL that is different than my kernel32.dll  i.e. it is vectoring into code other than API CreateFileA. I have executed applications that were assembled before I installed MASM32 and they correctly open and close files, only new assembles have this problem.

Can a .LIB and a .DLL get out of sync? What is happening? What all does Install do during install? I know that executing the editor grabs the .ASM extension (I always have used PFE and now I jump right into MASM32 editor). I have corrected that back via file options, but still would like to know what could be happening with CreateFileA.

Dave.

[MichaelW]

I can't recall ever having a problem with CreateFileA that was not due to something in my code. With 7 parameters and multiple options there is a lot opportunity to make mistakes. Regarding SetConsoleMaximumWindowSize, I see nothing unusual in Microsoft leaving an undocumented function out of an import library. You can always use run-time dynamic linking, and for an undocumented function that could be removed from the DLL at some point, run-time dynamic linking makes more sense because it allows you to make a graceful exit if the function is not present.

[Mark Jones]

Dave, I don't believe the MASM installer alters any configuration settings, not even the registry. In fact, once I asked Hutch why there was no option of adding "\masm32\bin\" to the path (and even gave him a working code snippet which toggled this) and it never made it into production. He vehemently dislikes code which "does things behind your back."

I can say however, that some recent WindowsUpdates had affected things on my system. Any debugged program was causing a page fault in RPCRT4. After updating video and audio drivers, the problem went away...

[Vortex]

Hi KeepingRealBusy,

The import library kernel32.lib coming with the latest version of the Masm32 package version 10 does not export SetConsoleMaximumWindowSize. You can verify this by viewing the export list of kernel32.lib with the help of the tool dumpbin :

Code Select Expand

\masm32\bin\dumpbin /EXPORTS \masm32\lib\kernel32.lib

Code Select Expand


.
.
.
_SetConsoleCursorPosition@8
_SetConsoleMode@8
_SetConsoleOutputCP@4
.
.
.


An import library is a file with a special format to inform the linker about the owner ( DLLs )  of a group of symbols, the exported functions. It's easy to modify and create import libraries.

First, get the module definition file from kernel32.lib :

Code Select Expand

\masm32\tools\l2def\l2def \masm32\lib\kernel32.lib

Edit kernel32.def to add the extra symbol :

Code Select Expand


.
.
.
"_lstrcpynA@12"
"_lstrcpynW@12"
"_lstrlenA@4"
"_lstrlenW@4"
"_SetConsoleMaximumWindowSize@8"


Don't forget to insert the leading underscore _  , the decoration symbols @8 ( 8 = 2 parameters * 4 bytes ( 4 bytes = 1 DWORD ) per parameter ) and finally the double quotes.

After modifying kernel32.def, build the new version of kernel32.lib :

Code Select Expand

\masm32\bin\polib /OUT:kernel32.lib /DEF:kernel32.def /MACHINE:IX86
[attachment deleted by admin]

[KeepingRealBusy]

Vortex,

Thank you! Thank you!

This will solve a problem or two.

Now If I only could find out what is happening with CreateFileA.

Dave.

[jj2007]

Now If I only could find out what is happening with CreateFileA.

Does this work for you?

Code Select Expand

include \masm32\include\masm32rt.inc

Open_for_input PROTO:DWORD

.code
AppName db "Masm32:", 0

start: invoke Open_for_input,  chr$("\masm32\macros\macros.asm")
MsgBox 0, str$(eax), "Ret value:", MB_OK
exit


Open_for_input proc OpenF$:DWORD
  invoke CreateFileA, OpenF$, GENERIC_READ, FILE_SHARE_READ,
  NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0
  .if eax==INVALID_HANDLE_VALUE
invoke MessageBox, NULL,  OpenF$, chr$("File not found:"), MB_OK
invoke ExitProcess, 0
  .endif
  ret
Open_for_input endp

end start


[KeepingRealBusy]

JJ,

I have many examples of open file calls. They all work, all except this one that I just assembled after installing MASM32.

Here are some snipits from both the program and from the .LST file with  ASM commentary:

Code Select Expand

;   The file name definition:

    szInfoFile       BYTE "Siveinfo.dat",0

;   The open call:

    pushfd
    pushad
    INVOKE CreateFileA,\
           OFFSET szInfoFile,\
           GENERIC_READ,\
           FILE_SHARE_NONE,\
           NULL,\
           OPEN_EXISTING,\
           FILE_ATTRIBUTE_NORMAL,\
           0
    mov    dInfoHandle,eax
    popad
    popfd

    cmp    dInfoHandle,INVALID_HANDLE_VALUE
    jz     Exit

;   The .LST file name storage:

  0000005A 53 69 76 65 69     szInfoFile       BYTE "Siveinfo.dat",0
   6E 66 6F 2E 64
   61 74 00

;   The .LST call:

00000D68  9C     pushfd
00000D69  60     pushad
    INVOKE CreateFileA,\
           OFFSET szInfoFile,\
           GENERIC_READ,\
           FILE_SHARE_NONE,\
           NULL,\
           OPEN_EXISTING,\
           FILE_ATTRIBUTE_NORMAL,\
           0
00000D6A  6A 00    *     push   +000000000h
00000D6C  68 00000080    *     push   +000000080h
00000D71  6A 03    *     push   +000000003h
00000D73  6A 00    *     push   +000000000h
00000D75  6A 00    *     push   +000000000h
00000D77  68 80000000    *     push   +080000000h
00000D7C  68 0000005A R   *     push   dword  ptr OFFSET FLAT: szInfoFile
00000D81  E8 00000000 E   *     call   CreateFileA

;   The program never gets to here. When just executing, the program ends up
;   exiting. When trapping via a DIV by 0 just before the call, a VS execution
;   via F10 or setting a breakpoint at the following instruction and using F5
;   both cause VS to hang, eating 50% of a CPU, need Task Manager End Task to
;   get rid of it.

00000D86  A3 00000506 R     mov    dInfoHandle,eax
00000D8B  61     popad
00000D8C  9D     popfd

00000D8D  83 3D 00000506 R     cmp    dInfoHandle,INVALID_HANDLE_VALUE
   FF
00000D94  0F 84 000000AB     jz     Exit


All the values look correct, only, it just doesn't work, it dies in the DLL.

Dave.

[jj2007]

Hmmm. And what happens if you run it with Olly? Can you compare its behaviour to what the latest "good" exe did?

    pushad
    int 3
    INVOKE CreateFileA

[KeepingRealBusy]

I've never used Olly. Don't you need to start it first? How does the system know to start Olly when it sees an int 3?

What happens normally when I run a good .EXE (I don't have a working .EXE of this program, that was what I was cleaning up) is that it correctly opens, reads/writes the file, and closes the file. This is for .EXE files that I had created before installing MASM32. I have tried several programs and they work, BUT they were assembled before installing MASM32. I'm afraid to try to assemble any of my working .EXEs. I guess I will do a save of the working .EXE, and re-assemble and see what happens to another formerly working program.

I am mystified.

Dave

[Mark Jones]

I've never used Olly. Don't you need to start it first? How does the system know to start Olly when it sees an int 3?

Olly can be set as the system debugger, and when the int 3 is executed, "run with debugger" will be an option on the crash report that opens up. Or start Olly and tell it to open or attach to the executable.

Olly, or another assembly-level debugger, is absolutely essential IMO.

I am mystified.

Sounds like you need a break. Grab some coffee, watch some TV, then come back to it.

[jj2007]

I've never used Olly. Don't you need to start it first? How does the system know to start Olly when it sees an int 3?

Olly is a must, and not too difficult. Drag your *.exe over ollydbg.exe; the debug window should open. Then press F9: the program executes until it encounters the int 3 command. Press F7 repeatedly to see your code execute. If it takes too long, press F9 again to see if it crashes somewhere...

[Vortex]

Hi

Another option is to disassemble your executable to see what's happening with CreateFile. Using an interactive debugger like Ollydbg is the best idea to solve the problem.

Code Select Expand

\masm32\bin\dumppe -quiet -disasm Yourfile.exe

[KeepingRealBusy]

Is there any documentation about how to use OllyDbg? I would really like to get more information on how to use it effectively. I finally figured out how to get a trace and write it to a file, then I found out what had happened. It had nothing to do with installing MASM32. It was only coincidental with the installation. The module I was working on allocated a heap, the allocated a huge buffer from the heap (all of it). Then it tried to open a file checking for presence using CreateFileA. CreateFileA tries to convert the ASCII file name into Unicode then called CreateFileW (I didn't know that?). This required a buffer so NTDLL tries to get space from my heap, of course there is no more space, so this fails, but NTDLL still tries to call CreateFileW. This ends up in an exception for access violation (mov ecx,[eax] ;where eax is 0) but this is in the kernel so it just exits the app.

I bypassed the heap code and just opened the file, and all worked until I tried to use the non-created buffer. I will now convert this to use VirtualAlloc to get the space.

If this error occurs in OllyDbg, you get fault on fault, but you can terminate OllyDbg with the X button or File Exit. If this occurs in VS debug (Windbg?) then it hangs VS and you need to use Task Manager to kill it.

There are enough problems to go around.

1. My program needs to be fixed.

2. NTDLL should handle the failure of the convert ASCII to Unicode a little more gracefully.

3. VS debugger should not hang on such a condition, requiring Task Manager to kill the process.

Again, any good documentation for how to use OllyDbg?

Dave.