Allocating a large array

[CooPs]

Hi, I'm using masm32 to compile my code. I want to allocate an array of 10.000.000 dwords, (like 32 mb of memory). What would the best way be to do this?

[asmfan]

Use
LargeArray  DD 10.000.000  DUP(0)
Then wait a couple of hours for completion;)

[CooPs]

And that was exaaactly what I was trying to avoid by allocating at runtime.. :P

Btw, I need this for my prime calculator (me and my friend are competing how fast we can calculate 10 milion prime numbers) and I'm kind of done with the algoritm.. just need the space. He spanked my VB algoritm with C++ and now I'm gonna spank him with Assembler.

[asmfan]

Assembler doesn't allocate memory dynamically, but OS does...
Use GlobalAlloc, LocalAlloc, HeapAlloc, VirtualAlloc (!) and other allocs;)
Refer to MSDN...

[CooPs]

Yeah, now I'm trying to use ViritualAlloc to allocate memory but it doesn't work..

Code Select Expand

call VirtualAlloc, ebx, 10000000*4, MEM_COMMIT, PAGE_READWRITE

I get a compile error complaining about the ","
Maybe you are supposed to send it parameters some other way?
Like push or something?

[mnemonic]

If you want to "call" a funtion that way you will have to use "invoke" instead of "call".

Please try to get familiar with the plain basics. :eek

You may have a look at the file "MASM32.HLP".

[CooPs]

Aha, so that's the differance between invoke and call?

I had no idea.

[mnemonic]

Yes.

You may use

Code Select Expand

invoke VirtualAlloc, ebx, 10000000*4, MEM_COMMIT, PAGE_READWRITE
or

Code Select Expand

push PAGE_READWRITE
push MEM_COMMIT
push 10000000*4
push ebx
call VirtualAlloc

[redskull]

Does the algorithm require storing all the numbers at once? It might be better (and faster) to just calculate the next number in the series, dump it out to the screen, forget about it, and move onto the next.  After all, with that much memory, Windows is bound to swap a majority of it to the hard disk, slowing things down even more.

[stanhebben]

I'm not sure, but LocalAlloc could be faster than VirtualAlloc, becouse VirtualAlloc has to clear the memory, while LocalAlloc doesn't.

Thought I don't know if LocalAlloc can reserve such large spaces...

You can benchmark it if you want to be sure, CooPs.

[CooPs]

Ah well it only has to allocate the memory once so I don't worry about that speed. And I got it to work too.  :U

Does the algorithm require storing all the numbers at once? It might be better (and faster) to just calculate the next number in the series, dump it out to the screen, forget about it, and move onto the next.  After all, with that much memory, Windows is bound to swap a majority of it to the hard disk, slowing things down even more.

Yes, it needs to store all the numbers becouse it can use the lower primes to calculate if higher ones are primes.

[dsouza123]

After all, with that much memory, Windows is bound to swap a majority of it to the hard disk, slowing things down even more.

Wouldn't it depend on the OS and how much RAM the PC has ?

I am using XP Pro SP2 with 512 MB and it still has 300+MB of available free physical memory.
An allocation of 40 Million bytes should be able to stay in RAM.

Are there any API functions or parameters that keep the allocated memory in RAM
rather than paged out to disk, which cause page faults and has very slow access times ?
Keeping it in RAM should help performance .

How can it be determined if a particular allocation is still in RAM (physical memory) ?

I tried Process Explorer from sysinternals.com and selected the Opera browser, right clicked Properties, selected Performance Tab
it has a Memory section which shows

Private Bytes:  a K
Peak Private Bytes: b K
Working Set: c K
Peak Working Set: d K
Page Faults:  e
Page Faults Delta: f

Which if any of these values will tell if the allocated memory is still in RAM ?

Is it better to allocate an amount of memory that is a multiple of the page size ?

[MichaelW]

At least when allocating from the heap, the available physical memory value returned by GlobalMemoryStatus does not work as I expected it to. I think you can get some idea of the physical memory load, and some idea of when to expect a sharp decrease in memory access speeds, from the change in the available virtual memory (or at least some idea of when Windows will start complaining, or for Windows 9x maybe crash). I did not have any significant problems running this test app, but I did not attempt to push the available virtual memory much below about 50% of the initial value.

Code Select Expand


; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    include \masm32\include\masm32rt.inc
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    .data   
        ms MEMORYSTATUS <>
    .code
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
start:
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
    invoke GlobalMemoryStatus, ADDR ms
    mov ebx, ms.dwTotalPhys
    shr ebx, 20
    print "~TotalPhys: "
    print ustr$(ebx)
    print "MB "

    mov ebx, ms.dwAvailPhys
    shr ebx, 20
    print " ~AvailPhys: "
    print ustr$(ebx)
    print "MB "

    mov ebx, ms.dwAvailVirtual
    shr ebx, 20
    print " ~AvailVirtual "
    print ustr$(ebx)
    print " MB",13,10,13,10

    mov edi, (1 shl 20) * 4
  @@: 
    print "Allocate:"

    mov eax, edi
    shr eax, 20
    print ustr$(eax)
    print "MB "

    mov esi, halloc(edi)

    invoke GlobalMemoryStatus, ADDR ms
    mov ebx, ms.dwAvailPhys
    shr ebx, 20
    print " ~AvailPhys: "
    print ustr$(ebx)
    print "MB "
    mov ebx, ms.dwAvailVirtual
    shr ebx, 20
    print " ~AvailVirtual:"
    print ustr$(ebx)
    print "MB "

    .IF esi
        call GetTickCount
        push eax

        xor ecx, ecx
        .WHILE ecx < edi
            movzx eax, BYTE PTR[esi+ecx]
            inc ecx
        .ENDW

        hfree esi

        call GetTickCount
        push eax
        print " access:"
        pop eax
        pop edx
        sub eax, edx
        print ustr$(eax)
        print "ms",13,10

   .ENDIF
    shl edi, 1
    print chr$(13,10)
    inkey "Press key to 2x allocation...or close console window to exit"
    print chr$(13,10)
    jmp @B
    exit
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
end start

MEMORYSTATUS STRUCT
  dwLength          DWORD      ?
  dwMemoryLoad      DWORD      ?
  dwTotalPhys       DWORD      ?
  dwAvailPhys       DWORD      ?
  dwTotalPageFile   DWORD      ?
  dwAvailPageFile   DWORD      ?
  dwTotalVirtual    DWORD      ?
  dwAvailVirtual    DWORD      ?
MEMORYSTATUS ENDS


[hutch--]

CooPs,

If you are using anything like a modern machine, 40 meg (not 32) of memory is no big deal and as you allocate it in one block rather than repeatedly reallocating small blocks and in this context, the old GlobalAlloc() works fine as long as you use the fixed memory flag.

The trick will be to get the addressing mode right. Something like this.

Code Select Expand



LOCAL hMem :DWORD

invoke GlobalAlloc etc ......
mov hMem, eax
............
mov edx, hMem
mov ecx, 1234  ; index for array
mov eax, [edx+ecx*4]       ; load array content into register

; or

mov [edx+ecx*4], eax    ; move value in register into array member.

; when you have finished with the memory

invoke GlobalFree etc ....


[dsouza123]

In connection with VirtualAlloc or VirtualAllocEx
the functions VirtualLock or VirtualLockEx lock allocated virtual address space into memory
which prevents it from paging.

It only does maximum 30 pages by default, to get more SetProcessWorkingSetSize has to be called first.
It has both dwMinimumWorkingSetSize and dwMaximumWorkingSetSize among its parameters.

Using this on a system without enough free physical memory
can cause the performance of other programs running to be greatly reduced.

The other programs would end up being run from pages swapped to and from the disk.

Other connected functions are:
GetProcessWorkingSetSize which gets the current values of the parameters mentioned before.
GetSystemInfo which gets the page size which VirtualAlloc uses.