Building Multi-Module Projects using GoASM.

[Chip]

Donkey gave a good writeup on the reason and method
for creating and linking DLL's.  Unfortunately, his
write up last year focused on using RADASM, which I
do not use.

I would like to get clean compiles, and then convert
them to dll's after initial testing.  The program will
have many modules, some quite large.  I do not wish to
export/import individual procedures and data items.
Rather, I wish to just call the whole program and use
"Shared" data for all modules.  (Some of the modules
have very large arrays.)

However, I am unsure "what" to declare as imports or
exports.  I do not wish to be specific, since it's
the whole program I am using, not selected subsets
of it.

I have included below the link listings for both the
calling program and the called Dll.  I have also
included  code snippets from my software.

If anyone could be of assistance, I would appreciate it!

Chip





Output from Link - "Analyse" (Called by "Control" )
       
        GoLink.Exe Version 0.26.4 - Copyright Jeremy Gordon 2002/6-JG@JGnet.co.uk
       
        C:\Program Files\Assembly\My Programs\Analyse\Analyse.obj
            Made on Wednesday, September 27, 2006 at 02:51 size: 720,362 bytes
       
        Warning!
        There are no exports in this Dll
       
        Output file:
        C:\Program Files\Assembly\My Programs\Analyse\Analyse.dll
        Format: win32 size: 715,264 bytes
            Number of exports: none
            MSLU_LOADER function: not added
            Image base: 10000000h




Output from Link - "Control"  (Calls "Analyse" in DLL)       
       
        GoLink.Exe Version 0.26.4 - Copyright Jeremy Gordon 2002/6-JG@JGnet.co.uk
       
        C:\Program Files\Assembly\My Programs\Control\Control.obj
            Made on Wednesday, September 27, 2006 at 04:43 size: 787 bytes
        C:\Program Files\Assembly\My Programs\Control\analyse.dll
            Made on Wednesday, September 27, 2006 at 02:51 size: 715,264 bytes
            Number of required imports found in this file: none
       
        Error!
        The following symbols were not defined in the object file or files:-
        Register_Save_Array
        Analyse
        Output file not made
       
       

Snippets of my code
       
        *****************************
        DATA Section "Control" Shared
        ;#INCLUDE Analyse.dll
       
        Code Section "Control" Shared
        Start: Control:
       
        Call Analyse
        Call something else
        ret
        *****************************
       
       
        *****************************
        DATA Section "Analyse" Shared
       
        Code Section "Analyse" Shared
        Start:
       
        Analyse:
        .Do This
            cmp thisandthat
        .Do That
            add whatever
        Ret
        *****************************

[jorgon]

I do not wish to
export/import individual procedures and data items.
Rather, I wish to just call the whole program and use
"Shared" data for all modules.  (Some of the modules
have very large arrays.)

Could you explain why you want to use DLLs rather than one large EXE?

I think it would be much easier to use one large EXE, which could be made up of several object modules which would be joined up together at link-time.  If you do this you would not need to worry about declaring the correct exports or using shared data methods, it would all be automatic.

[Chip]

Jeremy,

What you suggested, is what I wished to do. 

(Perhaps I did not make myself clear.  It's not
the concept of using a Dll that I need, but just
the ability to break my program down into
manageable "chunks". 

Chip

[jorgon]

You can have several asm files.
Pass each one to GoAsm (one by one) and then you will have several obj files.
Pass all the obj files to GoLink at the same time (using a command file is easiest), and then you will have one large exe.

As for sharing code and data between the files, if you use this method of working you can just regard the asm files as one big asm file.  So for example any asm file can call functions in any other asm file.  Data declared in one asm file can be addressed in any of the others in the same way.  It's the linker's job to join them all up.

[Chip]

Jeremy,

That's about what i was doing (see above).
Could it be that having one as an execute
file messed it up?

Chip


Wait, Wait - Don't Tell Me!  (PBS Radio program)

One was already a Dll, I guess both should
have been obj files.

Chip

[Chip]

Jeremy,

Following your suggestion, the link went fine, but Gobug
is dying.  Could this have anything to do with the .dbg
files?

It keeps geting an exception C0000005h at the instruction
which calls the "Analyse" sub-routine.

It is trying to jmp to 403000h.  This appears to be the correct
address from the dump below.

Chip



                   GoBug - copyright Jeremy Gordon 1996-2005
               Date: Wednesday, September 27, 2006 Time: 06:28:15
                          Dump of debuggee information
                          whilst debugging Control.exe


    ************ The debuggee itself ************
    Path:- Control.exe
    Symbols:- COFF symbols in DBG file
    Subsystem type:- console
    Filesize (on disk):- 699K
    Handle to process:- 71Ch
    Handle to main thread:- 724h
    ProcessId:- 8A0h
    ThreadId:- 123Ch
     
    ************ Dynamic-link libraries *********
    C:\SYSTEM\system32\ntdll.dll at 7C900000h
    Symbols:- no symbols
    Subsystem type:- console
    Filesize (on disk):- 691K
    C:\SYSTEM\system32\KERNEL32.dll at 7C800000h
    Symbols:- no symbols
    Subsystem type:- console
    Filesize (on disk):- 960K
     
    ************ Inside the PE file ************
    Expected image base from PE file:- 400000h
    Actual image base as loaded:- 400000h
    Address of debuggee program start:- 401000h
    First known eip in debug loop:- 7C901231h
    Code start address:- 401000h
    Data start address:- 403000h
    Number of sections in PE file:- 4
     
    ************ The sections ************
    Section name:- Control
    Address of this section:- 401000h
    Size of raw data:- 200h
    Size of section (including padding):- 80h
    Characteristics flag:- 70000020h
    **************************************************
    Section name:- Analyse
    Address of this section:- 402000h
    Size of raw data:- 400h
    Size of section (including padding):- 2F0h
    Characteristics flag:- 70000020h
    **************************************************
    Section name:- Analyse
    Address of this section:- 403000h
    Size of raw data:- AE200h
    Size of section (including padding):- AE160h
    Characteristics flag:- D0000040h
    **************************************************
    Section name:- .rdata
    Address of this section:- 4B2000h
    Size of raw data:- 200h
    Size of section (including padding):- 34h
    Characteristics flag:- 40000040h
    **************************************************
    ****** Process specific memory areas ******
    Debuggee Process Environment Block:- 7FFD5000
    Debuggee Command Line:- 00020498
    Debuggee process flags:- 00000001
    Debuggee default heap:- 00140000
    Debuggee module header:- 00241EC4
    Debuggee context block :- FFDFF13C
    Environment strings:- 00010000
    Shared data in debuggee memory:- 7FFE0000
    Process Parameters:- 00020000
    Shared data in system memory:- FFDF0000
    Page directory and tables:- C0000000
    Processor Control Region:- FFDFF000
     
    ********* Thread specific memory areas *********
    hThread=724(main)
    Thread Environment Block:- 7FFDF000
    FiberData/Version:- 00001E00
    Thread Local Storage:- 00000000
    Thread Environment Pointer:- 00000000
    Thread Information Block:- 7FFDF000
    Bottom of stack:- 00130000
    Limit of stack top:- 00020000
    Current actual stack top:- 00120000
    Contents of user pointer in TIB:- 00000000
    UniqueProcess ID:- 000008A0
    UniqueThread ID:- 0000123C
    **************************************************
   

[Chip]

Jeremy,

Thanks for your fast responses!

The sun is coming up,
    the birds are chirping,
       from my mountain home a 1/2 mile high
           I can see silicon valley is under the clouds.

All in all, a good time to go to bed. 
(I have been working all night, and
the llink problem was just the icing
on my proverbial cake.)

Chip
Los Gatos

[Chip]

I set up the compile and links as suggested
by Jeremy.  The organization of my object
modules and consolidated execute is just
what I wanted.  (Multiple compiles, one link.)

However, If I use a CALL to routines in the
other sections, it incurs an exception on
the instruction "CALL"
       

                           GoBug - copyright Jeremy Gordon 1996-2005
                       Date: Thursday, September 28, 2006 Time: 03:41:28
                                        Dump of codepane
                                  whilst debugging Control.exe
       
       
            ~~~Setup.Clear_Registers:~~~>
            40102F: XOR EAX,EAX
            401031: XOR EBX,EBX
            401033: XOR ECX,ECX
            401035: XOR EDX,EDX
            401037: XOR EDI,EDI
            401039: XOR ESI,ESI
            40103B: XOR ESP,ESP
           
            ~~~Process_Control:~~~>
            40103D: CALL Analyse_Input
           
            ~~~Process_Control.End_of_Control:~~~>
            401042: INT3
           
            ~~~Process_Control.Restore_All_Registers:~~~>
            401043: MOV EAX,[Register_Save_Array]
            401048: MOV EBX,[40803C]
            40104E: MOV ECX,[408040]
            401054: MOV EDX,[408044]
            40105A: MOV EDI,[408048]
            401060: MOV ESI,[40804C]
            401066: MOV EBP,[408050]
            40106C: MOV ESP,[408054]
            401072: RET
       
       
                           GoBug - copyright Jeremy Gordon 1996-2005
                       Date: Thursday, September 28, 2006 Time: 03:41:21
                                      Dump of main logpane
                                  whilst debugging Control.exe
       
       
        Running to starting address
        Starting address reached
        Exception ...C0000005h at: 40103Dh Thread=718h
        ~ Instruction: CALL Analyse_Input    Function: Process_Control
        ~ EDI=00000000   ESI=00000000      EBX=00000000      EDX=00000000
        ~ ECX=00000000   EAX=00000000      EBP=0012FFF0      ESP=00000000
       
       



However, If I replace the "CALL" with a "JMP,
it jumps into the procedure desired with no
problem.  (NOTE: That the Call did not have
an address, while the JMP does.)


                           GoBug - copyright Jeremy Gordon 1996-2005
                       Date: Thursday, September 28, 2006 Time: 03:31:14
                                        Dump of codepane
                                  whilst debugging Control.exe
       
       
            ~~~Setup.Clear_Registers:~~~>
            40102F: XOR EAX,EAX
            401031: XOR EBX,EBX
            401033: XOR ECX,ECX
            401035: XOR EDX,EDX
            401037: XOR EDI,EDI
            401039: XOR ESI,ESI
            40103B: XOR ESP,ESP
           
            ~~~Process_Control:~~~>
            40103D: JMP [402000]
           
            ~~~Process_Control.End_of_Control:~~~>
            401043: INT3
           
            ~~~Process_Control.Restore_All_Registers:~~~>
            401044: MOV EAX,[Register_Save_Array]
            401049: MOV EBX,[40803C]
            40104F: MOV ECX,[408040]
            401055: MOV EDX,[408044]
            40105B: MOV EDI,[408048]
            401061: MOV ESI,[40804C]
            401067: MOV EBP,[408050]
            40106D: MOV ESP,[408054]
            401073: RET
       

I have tried every version of call, jump, renaming
procedures, etc.  Both the compile and link listings
indicate no problem.  Using GoDebug, the JMP and CALL
addresses appear correct when examining code addresses.

I have not a clue as to how to troubleshoot this.


Chip
Los Gatos, Ca

[Chip]

I am assuming that the exception error mentioned
above must have some solution.  I decided to
temporarily bypass it in order to link further
object modules into the exec that controls all
of the sub modules.  (I have four written, and
about four more sub modules to write.)

However, I ran into a new problem.  I am not sure
if it is my lack of understanding, or some kind
of limitation.

Two of the sub-modules being linked use the same
data. Both get clean compiles.  But, if I try and
link them, I get duplicate symbols, and it refuses
to create an EXE. 

Yet, if I pull the duplicated data out of one of
them, I can not get a clean compile. Therefore,
it will not make an OBJ file for linking.  (See Below.)

What am I doing wrong?

Chip


        GoLink.Exe Version 0.26.4 - Copyright Jeremy Gordon 2002/6-JG@JGnet.co.uk
       
        C:\Program Files\Assembly\Programs\Objects\Control.obj
            Made on Thursday, September 28, 2006 at 07:22 size: 861 bytes
        C:\Program Files\Assembly\Programs\Objects\Analyse.obj
            Made on Thursday, September 28, 2006 at 07:26 size: 720,290 bytes
        C:\Program Files\Assembly\Programs\Objects\Compress.obj
            Made on Thursday, September 28, 2006 at 07:23 size: 17,117 bytes
       
        Error!
        The following symbols were defined more than once:-
        Start (Control.obj, Analyse.obj, Compress.obj)
        Compression_Control_Block (Analyse.obj, Compress.obj)
        Compression_Sum_Higher (Analyse.obj, Compress.obj)
        Compression_Sum_High (Analyse.obj, Compress.obj)
        Compression_Sum_Low (Analyse.obj, Compress.obj)
        Compression_Power (Analyse.obj, Compress.obj)
        Output file not made


Addenda to above:       
I changed all of my programs to eliminate duplicate names,
at least for now. 

If this has to be the permanent solution,  it would defeat
the purpose of linking all of them together.  It seems that
I could not reference shared data in other modules without
causing the individual compiles to fail.

Finally,  any ideas on the duplicate Start: problem below?


        GoLink.Exe Version 0.26.4 - Copyright Jeremy Gordon 2002/6-JG@JGnet.co.uk
       
        C:\Program Files\Assembly\Programs\Objects\Control.obj
            Made on Thursday, September 28, 2006 at 07:22 size: 861 bytes
        C:\Program Files\Assembly\Programs\Objects\Analyse.obj
            Made on Thursday, September 28, 2006 at 07:26 size: 720,290 bytes
        C:\Program Files\Assembly\Programs\Objects\Compress.obj
            Made on Thursday, September 28, 2006 at 07:45 size: 17,117 bytes
        C:\Program Files\Assembly\Programs\Objects\Expand.obj
            Made on Thursday, September 28, 2006 at 07:47 size: 16,935 bytes
        C:\Program Files\Assembly\Programs\Objects\Read.obj
            Made on Thursday, September 28, 2006 at 07:50 size: 527,858 bytes
        C:\Program Files\Assembly\Programs\Objects\Write.obj
            Made on Thursday, September 28, 2006 at 07:50 size: 3,590 bytes
        C:\SYSTEM\system32\Kernel32.dll
            Made on Wednesday, August 04, 2004 at 05:00 size: 983,552 bytes
            Number of required imports found in this file: 6
       
        Error!
        The following symbol was defined more than once:-
        Start (Control.obj, Analyse.obj, Compress.obj, Expand.obj, Read.obj, Write.obj)
        Output file not made
       

[wjr]

Now that you are building one EXE from multiple ASM->OBJ files, GoLink expects to see only one Start symbol for the entry point address of your Control.exe. The Start within Analyse, Compress, etc. may not be needed if this was simple entry point code for a DLL, which is no longer the case. If Analyse, Compress, etc. require some initialisation, rename those parts of those Start procedures to something like AnalyseInit, CompressInit, etc. and call these from within the Start procedure of Control.

Your section declarations most likely no longer need the shared attribute and I would suggest simply using just CODE, DATA, and CONST if needed, throughout your project.

As for sharing various module data, after removing the duplicate symbol definitions (not by renaming), what GoAsm error message do you get for a module that references data in another module?

WJR

[jorgon]

If I use a CALL to routines in the
other sections, it incurs an exception on
the instruction "CALL"
XOR ESP,ESP
CALL Analyse_Input

However, If I replace the "CALL" with a "JMP,
it jumps into the procedure desired with no
problem.
XOR ESP,ESP
JMP [402000]

I have not a clue as to how to troubleshoot this.

The ESP register is the "extended stack pointer" register. 
It contains the current position on the stack.
The stack is an area of memory used by the PUSH, POP CALL and RET instructions.
These instructions read from or write to the stack, at the position given by ESP.
So having made ESP zero using the XOR instruction, your CALL instruction causes the processor to try to write to an address of zero. 
Hence the exception.
You can't use ESP as a general register.

[Chip]

Thank you both for your help!

WJR - I did as you said, and undid my renames.  At first I got a duplicate
symbol error.  I then ensured that the data was only defined once in all of the modules. 
That worked.  But, why I did not get a compile error is beyond me.  A sub-routine,
at compile time, had no way of knowing that it would be subsequently corrected
by the LinkAll.  Anytime in the past that I even mispelled a label or symbol, it gave
me compile errors, much less removing the references all together.

Jeremy - You were right about the ESP register being zeroed out just prior to the
call.  Most of my modules are completely self-contained, and do not call other modules
mid-stream.  Because I am doing almost exclusive bit processing on large files, I have
managed to use registers only to avoid unnecessary memory accesses. 

To protect myself from register corruption, my sub-routines are supposed to save and restore
all registers at beginning and end.  Unfortunately, I also included these routines in the
control module, which was a mistake.  (Thank goodness, only there - I use esp all through
my other processing - which I just successfully tested.) 

I assume that once I have to interface with calls to windows, then
the option of using esp will probably be lost.  But these routines are only executed once,
and I can tolerate some performance degredation there.  It is only in the intensive bit
calculations that I need every register I can get my hands on.

Once again, thank you both.  I am under a lot of pressure to finish quickly,
and a timely reply means a lot to me.  (I am also not getting much sleep,
which is why I included that wacky part about sunrise a few messages ago.)

Chip
Los Gatos, CA

[jorgon]

To protect myself from register corruption, my sub-routines are supposed to save and restore
all registers at beginning and end.

Instead of saving these values in data areas as in your previous sample code, just save them on the stack and restore them afterwards as follows:-

Code Select Expand


MyLabel:
PUSH EAX,EBX,ECX,EDI,EDX,ESI
              ;code goes here which uses the registers which are saved
POP ESI,EDX,EDI,ECX,EBX,EAX
RET

With GoAsm you can also use the USES ... ENDU statement to do this.

Also in your previous sample code seemingly you zeroed registers before using them.  Consider whether it really is necessary to do this.

[Chip]

Jeremy,

I was afraid to touch the stack if i am using ESP and EBP.  I agree that it would be
faster than using data areas.  But, the ability to have two extra register means that
once I get into very large loops, it is all register processing.  It is these loops, that i
have focused my optimising attention on since they will consist of 99.99% of the
processing.  (So far, I have gotten them down to 10 register add instructions, or about
20 cycles - if I include those that will run parallel.)

You are right about zeroing all of the registers being unnecessary.  During current
development, it is easier to just zero them all rather than analyse each module and
routine within modules.  However, once I am done,  going back through and removing
redundant intructions will be a priority.  I have also created some very large single arrays,
which has one module exceeding 700k.  Hopefully, I can go back and make most of them
into two dimensional arrays.  I plan to use "unions" to redefine arrays that that are not
subject to concurrent processing.

However, my biggest push right now is to concentrate on the Mathematical aspects of
my solution.  Once that is behind me, the rest will be more or less routine cleanup, error
checking, optimising, etc.  (I had not really wished to program the full production version,
but only the prototype that demonstrated feasability.  Yet even a prototype has to be
logically sound and robust enough to survive the usual system problems.)

Of course, I thought that all of the above would be done by now.  However, a spreadsheet
prototype developed two years ago had a serious, but hidden flaw.  The only solution
was to go back and redevelop the math theory before proceeding futher.  (And, we all know
that system solutions take at least 3X longer than even the best laid plans - Smile!) 
Tack on a bit of a learning curve, and it just gets worse.

Of course, my time and funds are running out, the theory is sound but unproven, and
life just gets more focused the faster I run.

Sorry, got a bit more chatty than I intended.

Chip
Los Gatos, CA

[jorgon]

You can use EBP since this does not use, and is not affected by, any PUSH, POP, CALL or RET instructions.
So the only register whose value you need to keep in data and to restore after your processing is ESP. 
Provided you ensure that ESP correctly points to the stack whenever you use PUSH, POP, CALL or RET you won't have any problems.
Of course you should not call any Windows APIs (or make any calls) while ESP is out of place.

For example:

Code Select Expand

MyCodeLabel:
PUSH EAX,EBP,EBX,ECX,EDI,EDX,ESI
MOV [SavedESP],ESP
              ;code goes here which uses the registers which are
              ;saved and also uses ESP, but makes no calls etc.
MOV ESP,[SavedESP]
POP ESI,EDX,EDI,ECX,EBX,EBP,EAX
RET