32-bit memory allocation on 64-bit OS

[Astro]

Hi,

Assume the OS wants to assign me memory above the first 2^32 addresses (so pushing the allocation into 64-bit territory). Does the memory wrap, or does the OS fool the 32-bit ap into thinking it is within the first 4 Gb of memory?

Best regards,
Robin.

[Astro]

No-one?

I can't find anything about this - only that a 64-bit OS increases the amount of total memory available, but nothing on how a 32-bit app handles memory over 4 Gb.

Best regards,
Robin.

[theunknownguy]

No-one?

I can't find anything about this - only that a 64-bit OS increases the amount of total memory available, but nothing on how a 32-bit app handles memory over 4 Gb.

Best regards,
Robin.

Cant find much information too but ill guess OS should trick the application. Probably the most close i found on MSDN was this:

http://msdn.microsoft.com/en-us/library/aa366720%28VS.85%29.aspx

Code Select Expand

three orders of magnitude larger than the native page size

But pretty much cant find more.

[sinsi]

The OS can take memory from any physical address and give it to you with a logical address via paging - the paging tables are used by the CPU to determine the address.
Your 4K page at logical address 00400000h could be anywhere physically.

[joemc]

No-one?

I can't find anything about this - only that a 64-bit OS increases the amount of total memory available, but nothing on how a 32-bit app handles memory over 4 Gb.

Best regards,
Robin.

What makes sense to me is a 32bit app can only see 4gb of memory.... but two 32bit apps can obviously use 8gb.  That is assuming a flat memory model, i never learned near and far pointers and i don't want too :).  How would a 32bit pointer work if this is not the case?  I don't think the OS can trick your application by guessing what you want to point at. The OS can see more than 4GB. Your application cannot.   Unless i am just wrong :)

[Astro]

Hi,

@joemc: This is the same logical wall I kept hitting, too.

@sinsi: I wondered about this. Where is this actually written though, or is it just an assumed extension to the way virtual memory works?

Does the OS lie to the 32-bit app about where it (the OS itself) and other resources reside in memory to make them accessible?

This: http://msdn.microsoft.com/en-us/library/aa384219%28VS.85%29.aspx and related links is about as much as I can find.

Best regards,
Robin.

[sinsi]

Look in the intel manuals under paging. The OS even 'lies' to itself - when it allocates memory it just takes some free memory from somewhere and maps it into your virtual address space.
That's supposed to be the idea behind dll's - the unchanging part (usually code) is mapped into address spaces, it only exists in one block of memory but more than one process can see it.

[Astro]

That's true.

I'll have a look over the weekend. Thanks!

Best regards,
Robin.

[hutch--]

Robin, I don't think fro memory that a 32 bit app cn allocate more that 2 gig in one allocation, its a left over from pre-64 bit OS design. Bottom 2 gig is for apps, top 2 gig is for the OS, device drivers, video adaptors etc ....

There was a variation of win2000 server that could use apps that were linked in a particular way to address the 2 to 3 gig address space but it came at the cost of limiting how much memory was available to the OS.

[Astro]

Hi,

I was aware of that limitation when running a 32-bit OS, but didn't think it applied to a 32-bit app running on 64-bit OS?

Very interesting. It would appear that memory allocations can silently fail and the 32-bit app would not necessarily crash immediately.

Best regards,
Robin.

[sinsi]

Even on a 64-bit OS you can only allocate upto 2GB (or ~3GB with the switch in boot.ini and a flag in the PE header set by /largeaddressaware link switch) from your 32-bit exe, but the memory can come from anywhere, even above 4GB (I think the CPU limit at the moment is 48 or 52 bits, not the whole 64 bits).
A 32-bit program is limited by the address space and windows usually reserves the top 2GB for system dll's and such.

One thing, don't confuse virtual swap space (pagefile) with memory. The page tables have a 'present' bit which means if the CPU accesses memory with that bit clear it sends an exception to windows, windows will allocate a page of ram then load your 'memory' back into memory from the pagefile transparently. Every memory access is checked by the CPU via the page tables and the segment descriptor.

Paging is a real pain-in-the-arse, it was originally used because nobody had 4GB of ram. Nowadays it is mostly used for protection, since you can e.g. make a page non-writeable and this is a CPU protection, not software.

Memory allocation should never 'silently fail' though.

[Astro]

Hi,

From: http://msdn.microsoft.com/en-us/library/aa366778%28VS.85%29.aspx

32-bit app running on 64-bit OS:

* 2 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE cleared (default)
* 4 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE set

Which is confusing, because when you read how WOW64 works, it states: http://msdn.microsoft.com/en-us/library/aa384219%28VS.85%29.aspx

However, 32-bit applications may not be able to create as many threads under WOW64 as they can when running natively on x86-based systems because WOW64 allocates an additional 64-bit stack (usually 512 KB) for each thread. In addition, some amount of address space is reserved for WOW64 itself and the data structures it uses. The amount reserved depends on the processor; more is reserved on the Intel Itanium than on the x64 processor.

Which suggests the memory available is perhaps more than under a native 32-bit OS due to a smaller "OS" footprint, but that the number of threads is resticted due to the memory required for each thread created. This suggests that the traditional 32-bit memory map is preserved under WOW64, although in a slightly altered state (i.e. it isn't a real 32-bit OS). Does this mean the same rules apply under WOW64 as running natively (including memory limitations such as the upper memory area being reserved for the kernel), or can a 32-bit process under WOW64 go crazy and use a full 4096 Mb (minus WOW64 overhead and stack) without any problems?????

Maybe another way to phrase this question is: how much memory is allocated to what in a WOW64 environment, and what does a WOW64 memory map look like?

One thing, don't confuse virtual swap space (pagefile) with memory.

Definitely not! I'm very conscious that virtual memory has NOTHING to do with the page file that is on the hard disk!  :U

Best regards,
Robin.

[joemc]

32-bit app running on 64-bit OS:

* 2 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE cleared (default)
* 4 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE set

Makes sense to me. 2GB if it is running just like on a 32 bit operating system. 4GB if it knows it is a 32bit on a 64 bit, because you can use the full 4 bytes for memory addresses since all the memory to control the process is going to be in 8 byte addresses.

[hutch--]

Robin,

When you create a thread you can control how much stack it uses. By default you get a meg which limits how many threads but if you set it much lower you can start far more threads. It comes with a catch though, the more threads you have running relative to your processor core count, the higher the percentage of time is spent task switching which in turn slows down the entire machine. If you need a large thread count look at methods of thread suspension so that the task switching load doe not get too high.

[Astro]

Hi Hutch,

Thanks for the tip! I didn't realize an excessive thread count affected the whole machine; I thought it only affected the process that spawned those threads.

The reason I asked is because I'm trying to figure out which part of MS FSX is dying first, whether it is running out of process memory or something else (video memory). There is a large amount of texture corruption before the sim finally loses it, but it never actually seems to run out of memory. It is one hog of an application that is for sure. :(

Best regards,
Robin.