GoAsm and parallel build

[nsquare]

Hi, Jeremy

When I run two compilation tasks in parallel, one of them gets blocked by
the other and often fails with

Code Select Expand


Could not open file:-
#include "windows.h"


I guess the included files are opened in exclusive mode, which prevents
subsequent open requests. In fact opening them in FILE_SHARE_READ
share mode is good enough and even better, since parallel compilation
is desired to take advantage of the HyperThreading/Multi-core CPUs.

[nsquare]

Code Select Expand


00401D3E 833906                        cmp     dword ptr [ecx],6
00401D41 7401                             jz      loc_00401D44
00401D43                        loc_00401D43:
00401D43 40                                 inc     eax
00401D44                        loc_00401D44:
00401D44 50                                 push    eax
00401D45 6800000080                 push    80000000h
00401D4A 52                                 push    edx
00401D4B F6050550410080         test    byte ptr [415005h],80h
00401D52 7408                              jz      loc_00401D5C
00401D54 FF15E4A14100            call    dword ptr [41A1E4h]


The cmp/jz instructions at 00401D3E might skip the following 'inc eax', which
results in the file-share-mode argument be set to 0. By nopping out the 5 bytes,
GoAsm.exe can be run in parallel and the total compilation time reduced to
1/2.

[ecube]

wow nice find, I ran into this issue earlier, just didn't realize it had to due with file access.

[jorgon]

Well as you've discovered, GoAsm opens some files permitting file sharing and others not.
The ones for which GoAsm does not permit file sharing are the header files (that is to say any include file without "a" or "A" as the first letter of the extension).
The reason for this is that GoAsm uses the failure to open an include file because it is already open, to avoid reading the same file twice.  Also it is used as part of its error checking when processing include files.
The usual reason for such an error is that the include file has been included more than once.  This doesn't stop processing, but the fact of the duplication is reported as a warning.

I've added a new switch to the GoAsm command line which will stop this happening for those who use parallel processing.
The new switch is /sh which stands for "share header files".

I have assumed here that you have actively to decide to use parallel processing and therefore will be sufficiently aware of the problems which might arise in this, to use the /sh switch.  If I am wrong about this, I could make GoAsm permit file sharing as a matter of course, but I would have to find some other way of avoiding GoAsm reading the same file twice and of finding the duplication error.

So the new GoAsm is available from here (Version 0.56.8).

[ecube]

Great! thanks for the quick fix jorgon :) assuming i'm understanding the problem fully, you could just add the include file names opened to an array in memory, and each new entry look to see if its already been added or not, and go from there. Donkey has neat simple array functions written in GoASM, in his Strings library, that let you do this, and alphabetically sort entries for quick searching etc...

[nsquare]

Thank you Jeremy.

Since donkey's headers are self-protected with '#ifndef ... #endif', duplicate reading
won't cause parsing problems. Furthermore, for my own header files, I use
'#ifndef _HEADER_H_\n#include header.h\n#endif' to avoid duplicate inclusion. So the
problem you worry about can be easily dealt with by existing facilities and a little
good coding style. On the other hand, the improvement on overall performance is
too attractive to resist.

[nsquare]

Hi Jeremy, a small suggestion.

When /sh is specified and opening an include file fails, please treat it as an error
and don't make the .obj file.