Memory Allocation

[RuiLoureiro]

How can i alloc more than 0FFFFFFFFh bytes of memory
Is there a way ?
Thanks

[dedndave]

create a temporary file ?   :P

[Twister]

I think you can allocate that much. Windows handles large amounts of memory in a program by sending some of it to the system swap.

[MichaelW]

Rui,

Are you running 32-bit Windows or 64-bit Windows? For 32-bit Windows your process is normally limited to a 2GB address space, so even if you could allocate >4GB you could not access it. I think AWE may provide a way to allocate and use >4GB, but the system would need to have sufficient physical memory available.

MSDN: About Memory Management

MSDN: Large Memory Support

[RuiLoureiro]

dedndave & GTX,
               Thanks
MichaelW,
          Iam running 32-bit Windows on a P4 in a XP OS
          total physical memory = 512 MB; physical memory available = 148 MB
          forget ! My process wants to alloc 2G when i have only 148M !
          forget, thanks

[Farabi]

dedndave & GTX,
               Thanks
MichaelW,
          Iam running 32-bit Windows on a P4 in a XP OS
          total physical memory = 512 MB; physical memory available = 148 MB
          forget ! My process wants to alloc 2G when i have only 148M !
          forget, thanks

If Im not mistaken, you still can allocate 2 GB of memory even your RAM is only 512 MB. But I never tried it, also, it might slowing down the whole system if you do that.

[RuiLoureiro]

If Im not mistaken, you still can allocate 2 GB of memory even your RAM is only 512 MB. But I never tried it, also, it might slowing down the whole system if you do that.

Thanks, Farabi but i am not follow this way/method

[oex]

'Virtual Memory' I think he means.... On XP Control Panel -> System -> Advanced -> Performance

[MichaelW]

On my system with 512MB of physical memory I can allocate 1GB, using the code here, but because the system is dipping into the swap file access to the memory is slow and everything else slows down somewhat. I think I have some other code somewhere that managed to allocate 2GB on the same system.

[Twister]

It does become noticeably slow if you allocate twice or triple the memory than you have. You could take this route if your application does use memory for reading often. It's just if you have to read and write so often it will slow things down, and you will get bad performance.

The temporary file method: I don't think it would be any faster; I do think it would be slower than the above method. The operating system already does this automatic by "page swapping". It would become a real bore on your application since the operating system does this already. So there is not a real need to do this unless your application is recording heavy amounts of data.


There are many approaches you can take. It just depends on what all you are planning to do. :wink

[Rockoon]

By manually doing the paging with a temp file you can yield better results than the OS, because of several factors

(a) its paging mechanism is generic and cannot predict your usage pattern
(b) you can cause a paging fight between multiple processes running on the system

You can eliminate (b) by keeping your own allocation small enough that those other processes dont do any paging at all.

This of course implies that you actually have a good usage pattern to leverage, rather than random accessing, and that your usage pattern *IS* hard to predict (if its easy to predict, just let the OS predict it)

[Twister]

Ah, well I use random access much more so I am more tied to the methods I said in the post above yours, Rackoon.  :P

[MichaelW]

I failed to consider the effects of random access. The code I used times sequential access, and even for that the access time increased by a factor of 8.4 going from 256MB to 1024MB.

[RuiLoureiro]

On my system with 512MB of physical memory I can allocate 1GB, using the code here, but because the system is dipping into the swap file access to the memory is slow and everything else slows down somewhat. I think I have some other code somewhere that managed to allocate 2GB on the same system.

I want to access memory quickly. So it doesnt seem the way to solve the problem
Thanks all               

[MichaelW]

Hi Rui,

"Quickly" is relative. In my tests the access time was the time to access each byte of the allocation, a byte at a time. After I posted I realized that the factor 8.4 included a 4x increase in the size of the allocation, so the "virtual-memory" penalty was just over 2x. Here are the timing results running under Windows 2000 on a 500MHz P3 with 512MB of physical memory (and 32 tasks showing in Task Manager):

Code Select Expand


Allocate:8MB  ~AvailPhys: 355MB  ~AvailVirtual:2028MB  access:80ms
Allocate:16MB  ~AvailPhys: 355MB  ~AvailVirtual:2020MB  access:180ms
Allocate:32MB  ~AvailPhys: 355MB  ~AvailVirtual:2004MB  access:350ms
Allocate:64MB  ~AvailPhys: 355MB  ~AvailVirtual:1972MB  access:721ms
Allocate:128MB  ~AvailPhys: 355MB  ~AvailVirtual:1908MB  access:1482ms
Allocate:256MB  ~AvailPhys: 355MB  ~AvailVirtual:1780MB  access:3154ms
Allocate:512MB  ~AvailPhys: 355MB  ~AvailVirtual:1524MB  access:9293ms
Allocate:1024MB  ~AvailPhys: 389MB  ~AvailVirtual:1012MB  access:25316ms


And the results accessing the memory 4 bytes at a time:

Code Select Expand


Allocate:4MB  ~AvailPhys: 418MB  ~AvailVirtual:2032MB  access:20ms
Allocate:8MB  ~AvailPhys: 418MB  ~AvailVirtual:2028MB  access:30ms
Allocate:16MB  ~AvailPhys: 418MB  ~AvailVirtual:2020MB  access:71ms
Allocate:32MB  ~AvailPhys: 418MB  ~AvailVirtual:2004MB  access:150ms
Allocate:64MB  ~AvailPhys: 418MB  ~AvailVirtual:1972MB  access:310ms
Allocate:128MB  ~AvailPhys: 418MB  ~AvailVirtual:1908MB  access:630ms
Allocate:256MB  ~AvailPhys: 418MB  ~AvailVirtual:1780MB  access:1252ms
Allocate:512MB  ~AvailPhys: 418MB  ~AvailVirtual:1524MB  access:4937ms
Allocate:1024MB  ~AvailPhys: 430MB  ~AvailVirtual:1012MB  access:18937ms


As you can see, the increase in access time is approximately linear up to the point that the allocation exceeds the available physical memory. I don't have enough data points to support this, but it looks like an approximately 2x increase in slope beyond that point. So my point is that depending on the speed of your system, the available physical memory, the type of access, your needs, etc, virtual memory may be fast enough.