Assembler hangs for big buffers in .data? section

[xmetal]

This allocation makes use of DUP operator.
You have to know your tools. In MASM DUP is a macro. In fact you can write something like this:

Code Select Expand


buffer db 100 dup ( 5 dup (1,2,3,4) )


This means that the DUP operation will not be performed in one single steep.
And it also means that the premise of jj2007 that the amount of data is "known" is FALSE.

It is most likely that MASM will generate 40000h of db ? in a buffer and assemble that buffer. Other open source assembler perform exactly like this (but they name the keyword "TIMES").

This behaviour seems counter-intuitive because the declaration is in the .data? section. No matter how complex the declaration is, we only need to parse it and generate a constant as the result. This parsing is possible without the allocation of any buffer at all. The buffer would be required for the .data section since the data would need to be written to the object file.

Evidently, MASM uses a "general" parsing procedure for the .data and .data? sections. I generated the full intermediate listing for programs in which the buffer was declared in the .data? section and found that the "?" terminal is evaluated to 0, as if it was declared in the .data section.

[jj2007]

You have to know your tools. In MASM DUP is a macro. In fact you can write something like this:

Code Select Expand


buffer db 100 dup ( 5 dup (1,2,3,4) )


This means that the DUP operation will not be performed in one single steep.
And it also means that the premise of jj2007 that the amount of data is "known" is FALSE.

No, it just means that a tired nerd was at work here. If he had been less tired, he would have written some extra lines to circumvent the macro and thus avoided a speed problem. Not that this is the only case of this kind: As Hutch rightly noted in another post, some BASIC dialects including the one I use (Gfa) suffer from the same problem:
For n=1 to 10000
a$=a$+" some bytes"
Next n
That takes ages indeed. It took me some time to identify the problem 'I don't claim to be Sherlock Holmes), but since then I simply allocate a fat string at the beginning and use Mid$(a$,position)=" some bytes" etc.

It was not very hard...was it? Elementary my dear Watson....

Here is one for Bogdan Holmes:

Code Select Expand


You are the bus driver.  At your first stop, you pick up 29 people.  On the second stop,
18 of those 29 people get off, and at the same time 10 new passengers arrive.
At the next stop, 3 of those 10 passengers get off, and 13 new passengers come on.
On the fourth stop 4 of the remaining 10 passengers get off, 6 of those new
13 passengers get off as well, then 17 new passengers get on.

What is the colour of the bus driver's eyes?


Elementary, really elementary!

[jdoe]

Bogdan,

In the wake of speculations, I don't know how the uninitialize data gets assembled within the PE, but why you say that the amount of data is not "known". Why it could be hard to compute the size before any allocation occur. Your guess about the bug make sense but it seems like it could be avoided easily. No ? And each new MASM versions inherits this bug since long time. It does not seem strange ?

[jdoe]

As Hutch rightly noted in another post, some BASIC dialects including the one I use (Gfa) suffer from the same problem:
For n=1 to 10000
a$=a$+" some bytes"
Next n
That takes ages indeed. It took me some time to identify the problem 'I don't claim to be Sherlock Holmes), but since then I simply allocate a fat string at the beginning and use Mid$(a$,position)=" some bytes" etc.

This is a problem at runtime and it's not a bug. BASIC is just slow with strings. Am I wrong ?
A well written StringBuilder class could be faster for that, or some "Low Level PowerBASIC Code".

[jj2007]

BASIC is just slow with strings. Am I wrong ?
A well written StringBuilder class could be faster for that, or some "Low Level PowerBASIC Code".

GFA Basic is an old 16-bit dialect, and in general not significantly slower than C++. But is has, like other Basic dialects, this strange habit of slowing down drastically for multiple string concatenations - exactly the same problem that Bogdan described for MASM (although I don't see any excuse in the latter case).

[BogdanOntanu]

'I don't claim to be Sherlock Holmes

I do not claim to be Sherlok Holmes. I am sorry that you understood it that way.

I do not even claim to exist or to have any kind of inteligence. It was just a matter of speaking in order to make a logical and boring presentation more attractive.

It was a mistake to explain this bug...  but I had to do this experiment in order to understand by my own experience why exactly it was wrong. I thank you for giving me this possibility of understanding.

[drizz]

Code Select Expand

.data?
buffy label byte
org $+100000h-1
db ?
sizeof_buffy equ $-buffy
i thought of this workaround, but never used it.

[jj2007]

It was a mistake to explain this bug...

No, Bogdan, explaining the bug was not a mistake - sharing experience and knowledge is what this forum is meant for. Thanks for sharing your knowledge.

[MichaelW]

Code Select Expand

.data?
buffy label byte
org $+100000h-1
db ?
sizeof_buffy equ $-buffy
i thought of this workaround, but never used it.

It works, solves the speed problem and produces the same size exe.

Code Select Expand


; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    include \masm32\include\masm32rt.inc

    bssbuf MACRO ln
      LOCAL lbl
      .data?
        lbl label byte
        org $+ln-1
        db ?
      .code
      EXITM <OFFSET lbl>
    ENDM

; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    .data
    ;.data?
      ;buf db 1000000 dup(?)
    .code
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
start:
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    mov edi, bssbuf(1000000)
    xor ebx, ebx
    .WHILE ebx < 1000000
      mov BYTE PTR[edi+ebx], 255
      movzx eax, BYTE PTR[edi+ebx]
      .IF eax != 255
        print "oops",13,10
      .ENDIF
      inc ebx
    .ENDW
    inkey "Press any key to exit..."
    exit
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
end start


Although assembling it took ~ 1 minute, and the first time I tried the system issued a low virtual memory warning, and on every try the linker issued this warning:

Code Select Expand


Microsoft (R) Incremental Linker Version 5.12.8078
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

bssbuf.exe : warning LNK4084: total image size 1000013824 exceeds max (268435456)
; image may not run

Everything seems to work OK even with a billion-byte buffer.

[hutch--]

it is my guess that masm being well over 20 years old has a bit of antique code in it that is not up to modern speed when it comes to allocating data in the uninitialised data section. If it was a normally used method it would have been fixed years ago but I suggest that Microsoft who mainly keep masm for their own internal usage cannot see a use for a high speed .DATA? section builder.

There is no instance when dynamically allocated memory is not a better method of making memory available so there is little reason to change this well known bug in masm. The initialised data section is not much faster on large data and in the instance where I need a large file embedded into an exe as with an installation I wrote the toys called FDA and FDA2 to do this.

A few years ago I built a test file to test memory page thrashing that had 1.5 gig of autogenerated code. MASM baulks at files much over 100 megabytes so I built each file separately and linked it all together as a 350 meg exe file. Apart fom massive slowdowns due to page thrashing the file built in a few minutes as each file was not all that large in comparison to the total code.

Basically an assembler is a tool, not a consumer toy so part of using an assembler is knowing its limitations and workig within a safe range to get the results you need.

[NightWare]

Everything seems to work OK even with a billion-byte buffer.

omg,
:eek ok, before someone try my algo with extremly large buffer, you need to know this algo work perfectly until you don't exceed 7FFFFFFFh. (speed has always a price... readability/limitations/alignment/multi alignment/etc...) and personnaly i think, in THIS case, the price to pay is acceptable... (it treat enough mem for most cases...). i certainly should say that before, but it was so evident for me...  :P