I have a program that "defragments" my computers RAM resulting in quicker operation ( I guess it supposedly more RAM for use upon request ) . How does something like this work? What would be involved in coding something like this in assembly? Thanks for any tips..
Quote from: cman on October 07, 2009, 05:28:30 PM
How does something like this work?
It's like HD defragment, but for RAM.
RAM defragmentation? I call B.S. There's no way to control where in physical memory anything is, nor is there any benefit to having things "closer". The best you could do is free up wasted RAM and possibly ensure that nothing was paged out, but then that's not really defragging, is it?
-r
Hi cman,
Turn of the unnecessary services in your computer and remove the applications adding icons to your system tray if possible. Simple solutions will increase the perfomance of your RAM.
Quote from: redskull on October 07, 2009, 07:00:49 PM
RAM defragmentation? I call B.S.
It would not be the first such scam, for example SoftRAM 95:
http://en.wikipedia.org/wiki/SoftRAM
I've been using something called "CyberLat Ram Cleaner" for a long time now and it seems to work pretty well ( one makes a request to "free" a certain amount of ram and the system speed seems to increase after the request ). I've often wondered how it works and thought about how to implement my own version. What API functions are involved in an application like this? Are algorithms like those used in Operating System memory allocation used ( First-Fit , Best-Fit storage selection strategies ? ). Thanks for any information......
There is no such thing as "free RAM"; there is either RAM in use by programs, or RAM in use by the operating system. Windows loads old stuff in memory under the assumption that you'll use it again soon, only ditching it when there is an actual need (after all, what's the point of having oodles of RAM if it's actually empty?). I would suppose the way this program "frees up" memory is to allocate a ridiculously huge chunk for itself, which would cause other memory to be paged out all at once. Then the program frees it all, leaving a huge empty hole in memory that Windows will, eventually, fill back up; programs would potentially run faster while it does, because everything has been "prepaged out" already. I suspect you could program your own version by just doing HeapAlloc(1000000000), and then a corresponding HeapFree.
-r
Memory fragmentation is a well known problem which seriously impacts on performance, major gaps in the mapping of free memory to a memory block mean page thrashing with a performance hit every page change.
Here is a quick and dirty way to defrag memory, find out how much physical memory you have in your system, allocate the whole lot, deallocate it and let the OS decide what ends up where after the memory is freed. Used to have to do it regularly in Win9x otherwise when the free memory indicator went too low it would crash.
I use FreeRAM and it works like magic. When I use VirtualBox, my RAM is just dropping every second, until it hits near 20s and then FreeRAM cleans it up to 480s and for some time, all applications run slower until they re-allocate their stuff, at which point I have around 380 MB of RAM. So the real profit is around 360 MB. Not bad, eh? :wink
Btw my total amount of RAM is 1 GB, much of it being used by...stuff.
Right now I have 327 MB free - not running VirtualBox.
Just a second, lol, I'll start it.
Ok now I have 17 MB, let's run FreeRAM... It's cleaning atm and everything else is veryyy slow, that's normal - duh.
Ok finished, now I have 463 MB.
Now, Firefox froze... Now I have 455 MB.
Now I gave focus to VirtualBox and closed 2 windows which were open, now I have 424 MB.
Sometimes, VirtualBox won't start, when I'm low on RAM. That's also when FreeRAM comes into play.
-
All in all, it's definitely a GOOD choice to have. And DEFINITELY no bs. :wink
Now it was 432 MB and went up, 1 MB each few seconds, up to 443 MB - lol.
Thanks for the information! I think redskull is correct in that the Operating System fills the RAM according to the "principle of temporal locality" , sticking everything in memory with the assumption that it will be accessed at a latter time ( soon ). Calling HeapAllocate() then probably removes any reference of the old memory from whatever data structure the OS uses to keep track of such things and then the call to HeapFree() allows the OS to start filling the memory again with new items according to the same principle. That sounds kind of like what the software does! If so I guess I could write my own version with just a few lines of code. :bg
The solution to all problems is :
1) 4gb RAM or more and use win7 x64
2) 2gb RAM or more and use winXP 32bit
Reread Hutch's post, it says it all (and redskull is correct - as far as defragging goes, just not possible because of locks). As several of you guys have pointed out, the OS keeps a latent hold on some portions of memory in case of reuse. Forcing an allocation of all available physical memory and then releasing it forces windows to release its hold on all memory capable of being released thereby giving you a boost in available memory. I suspect that is all these type of programs are doing. When a program allocates memory that memory is locked from the system so it is impossible to relocate these areas in a defragmention sort of way so as to create contiguous areas. As Hutch suggests, write a program that works as he says (or if you ask R-E-A-L nice, he may see if he still has the said program - along with the millions of others he may have.
:U
Paul
You might try using SetProcessWorkingSetSize on all minimised windows.
I'm dubious of any gains, however I'm nearing the end of my knowledge, so i won't presume. The whole idea behind this (memory-leaky programs and majorly insufficient RAM aside), is that you can trick the MM into "giving back" all the just-in-case memory, which is then free for applications to use. But it's already free for applications to use, albeit on a demand basis. It seems as though all you are doing is paging stuff out all at once, instead of bit-by-bit, which I doubt would be a real improvement (though perhaps a apparent one). Basically, you spend several seconds waiting all at once at the beginning, instead of a few ms here, a few ms there, etc (obviously rounded times). I have a *very* hard time believing that on a modern NT system, with an appropriate amount of RAM for the demand placed on it, without running buggy programs that leak wicked bad, that any program can do better than what's already there. Maybe that's just my cynical nature...
-r
Quote from: redskull on October 08, 2009, 11:04:34 PMyou can trick the MM into "giving back" all the just-in-case memory, which is then free for applications to use. But it's already free for applications to use
I share your doubts. What happens, though, to mem that is reserved but not yet committed? Would the MM see a need to write that mem to the pagefile? Is fragmentation an issue?
They are a scam.
RAM access time is likely in nanoseconds, whereas a harddrive is in milliseconds. Random being the keyword that it can retrieve from anywhere with no penalty, as 'fragmented' as the data becomes, it does not matter. Access time remains constant.
Older versions of windows will probably move idle / minimized programs out to save memory, Vista and W7 do the smarter thing and make use of it though. Free ram is unused ram. RAM is blazingly fast compared to your HD and the OS will free space when something else requests it.
slovach,
What you have said here is nonsense. Allocate a gigabyte of memory and reference sequential addresses more than the processor cache size apart and watch how slow it is. Its called page thrashing and it is based on how the hardware works, not gizmos in the operating system.
Here is a rough toy that was similar to what I used to use in win9x. For what its worth it does not seem to matter on the XP I am working on which is an NT based system.
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
include \masm32\include\masm32rt.inc
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
comment * -----------------------------------------------------
Build this template with
"CONSOLE ASSEMBLE AND LINK"
----------------------------------------------------- *
.code
start:
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
call main
inkey
exit
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
main proc
LOCAL pfree :DWORD
LOCAL mst :MEMORYSTATUS
LOCAL msiz :DWORD
LOCAL hArr[128] :DWORD
LOCAL hcnt :DWORD
push ebx
push esi
mov hcnt, 0
lea esi, hArr
@@:
mov [esi], alloc(1024*1024*64) ; allocate 64 meg block
test eax, eax ; exit loop when it fails
jz @F
mov ebx, [esi]
print "Block allocated at "
print ustr$(ebx)," memory address",13,10 ; display the address
add esi, 4
add hcnt, 1
jmp @B
@@:
invoke SleepEx,1000,0 ; wait a second
@@:
free [esi] ; loop down the handle array
sub esi, 4 ; freeing each memory block
sub hcnt, 1
jnz @B
pop esi
pop ebx
ret
main endp
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
end start
I tried just that and my system yawned at me in return.
My understanding is:
Obviously if you end up off the multiples, you'll suffer unaligned horrors. In theory, say you were to access 0-7 then 64-71, the access would be fast and aligned.
You have missed the obvious, page load time, if you have ever had to write code that addresses location spread over a large address range you will know what PAGE THRASHING is about.
> I tried just that and my system yawned at me in return.
So does mine with 4 gig on 32 bit XP but someone may find a use for it.
And if you have more memory available than the working set requires?
I can only see this becomming less and less problematic. The last time I bought memory, I got a set of 4gb set of Corsair for $15 after rebate. At the ridiculous rate it's currently going, I'm pretty sure I'll end up with a previously seen as ludicrous quantity the next time I upgrade. I could always just make it into a RAM disk.
You can't defrag RAM - windows allocates virtual memory from its pool of physical ram. Your memory at virtual address 00400000h could be any physical address, and the
page after it (virtually) might not be physically the next page.
This fascinates me, big servers suffer the problem all the time and more recent server software has had to develop strategies to solve the problem. While each app has memory mapped to 400000h the operating system handles ALL of the physical memory and one of the problems of OS level memory management is block of memory that are deallocated inbetween other blocks. Think of an allocation scheme where you have a total of a gigabyte of memory allocate in 64k blocks, then delete every alternate block and you have a gigabyte of address space with half of it as 64k holes. This is MEMORY FRAGMENTATION. Then as extra programs allocate memory the OS has to start using those holes so you do not get contiguous reads but stepped reads from one fragment to another.
Web servers do this type of memory allocation regularly, a connection is made, memory is allocated for it, another connection is closed so the memory is freed. Multiply this by a year running and you will understand why servers must have a memory strategy that restricts memory fragmentation.
From what I've read, every memory access is checked by the processor, so it doesn't matter if the physical pages are contiguous or not. The problem comes from
the steps the processor goes through to check, the first step is the TLB. If the page is cached here, there isn't a penalty but if it isn't the next steps are expensive
in terms of the time taken.
Quote from: hutch-- on October 09, 2009, 05:18:41 AM
Think of an allocation scheme where you have a total of a gigabyte of memory allocate in 64k blocks, then delete every alternate block and you have a gigabyte of address space with half of it as 64k holes.
Why can't each of the 64K holes be reused and remapped as another half-gig of continuous space for another process (at least, under x86)? Every address is translated via a PTE, every time, no exceptions, so why wouldn't Windows just remap a continuous *virtual* address space to the discontinuous physical pages? The page thrashing meltdown comes from reading and writing pages back and forth *from disk*, not from jumping around between pages in RAM. That's a function of the NUMBER of pages you need in memory, not the location in RAM where they're put.
Also, @jj, i would assume reserved memory has no effect on anything. IIRC memory that's not committed has no pages (or PTE's) setup, so there would be nothing to page out at all.
:bg
> Why can't each of the 64K holes be reused and remapped ....
It can be and it is but the price is memory fragmentation, that is why server software that runs continuously with endless allocations and deallocations have special purpose memory managers to control the reuse of memory.
What makes you able to assume that the originally allocated 1GB of memory would be contingious physical pages in the first place? The only way you control that is to actually be the one who makes the PTE's (and/or VDA's), which I doubt Windows would allow willingly. Considering you can't avoid having to translate an address whether the pages are sequential or not, I still just don't see where any speed gain would be.
-R
The memory that is reserved IS technically available as just said so to be specific, all that is probably being done is to force windows to do a clean up and release of reserved memory. It should be noted, reserved memory is only done when a running program (or process) reserves and then releases memory while it is still running. Once the program exits, windows releases it. This means you can probably just close all programs and get a gain. The onlly potential gain you will not see would be from active processes not involved with programs started by the user that manipulate the memory pool. Older versions of windows were not very good at doing this type of clean up, though, so I still favor Hutch's solution. Also, I do not much trust even the most recent versions of windows.
Paul
Oh, and thanks for the little program Hutch. I plan to make use of it.
Paul
I'll second that, thanks for the code Hutch! :U Is there an api for reporting the amount of free RAM of your system? I'd like to see the effects of running the above code. Thanks for any information.
cman,
About your question, I am attaching a program written by John Lyons that shows memory usage. I have not the time to look at it right now but I am sure it will give you the answer you need.
Paul
Thanks for the code PBrennick ! :U
To address a couple of issues about the actual memory range available the output of the toy I posted does this.
Block allocated at 4259872 memory address
Block allocated at 71434272 memory address
Block allocated at 138608672 memory address
Block allocated at 205783072 memory address
Block allocated at 272957472 memory address
Block allocated at 340131872 memory address
Block allocated at 407306272 memory address
Block allocated at 474480672 memory address
Block allocated at 541655072 memory address
Block allocated at 608829472 memory address
Block allocated at 676003872 memory address
Block allocated at 743178272 memory address
Block allocated at 810352672 memory address
Block allocated at 877527072 memory address
Block allocated at 944701472 memory address
Block allocated at 1011875872 memory address
Block allocated at 1079050272 memory address
Block allocated at 1146224672 memory address
Block allocated at 1213399072 memory address
Block allocated at 1280573472 memory address
Block allocated at 1347747872 memory address
Block allocated at 1414922272 memory address
Block allocated at 1482096672 memory address
Block allocated at 1549271072 memory address
Block allocated at 1616445472 memory address
Block allocated at 1683619872 memory address
Block allocated at 1750794272 memory address
Block allocated at 1817968672 memory address
Block allocated at 1885143072 memory address
Block allocated at 2009530400 memory address
Press any key to continue ...
The addresses make the point, memory is allocated in most instances at the lowest address available and the next allocation is the next lowest address available.
For application that are messy in how they allocate and free memory, Microsoft introduced the Low Fragmentation Heap memory strategy to address applications/programmers that don't accurately control how memory is used. From memory the OS also provides a strategy that allocates memory at the highest address available, apparently to try and avoid a fragmented mess at lower addresses. I can't find the article in a hurry but recent design of Microsoft servers addressed the memory fragmentation problem by controlling their own memory pool and tracking when a connection was closed so the SAME MEMORY ADDRESS was reused to avoid ever more fragmentation. The technique was something like a circular buffer and each new connection was placed in the memory hole left by the last connection that had closed.
There is a mountain of reference material on Google that addresses the problems of memory fragmentation and it is an old problem that has been around for years, the older techniques were called memory compaction.
Quote from: hutch-- on October 09, 2009, 11:53:50 PM
the SAME MEMORY ADDRESS was reused to avoid ever more fragmentation. The technique was something like a circular buffer and each new connection was placed in the memory hole left by the last connection that had closed.
That's exactly the technique I use in MasmBasic for the Print macro and a number of other purposes. Much faster than HeapAlloc'ating every time a new string, and fragmentation-proof, but you are limited by the size of the buffer and need to maintain a "rotating" pointer.
JJ, I am curious. Why do you feel there is a limit on the size of the buffer? For example, my editor is using a 4mb buffer.
Paul
Quote from: PBrennick on October 10, 2009, 03:54:41 PM
JJ, I am curious. Why do you feel there is a limit on the size of the buffer? For example, my editor is using a 4mb buffer.
Paul,
You are right, the limit is more theoretical. Currently I use 640000 bytes allocated in the .data? section with this well-known technique:
MbBufferStart LABEL DWORD
ORG $+MbBufSize-1+16 ; MbBufSize plus one para of 'airbag'
db ?
That is more than generous for printing, but it has of course a
theoretical limit. If a user ever decides to print more than 640k to the console (or to a pipe), a runtime error message will pop up. Anyway, I don't see a compelling argument to allocate more for this "common buffer". Ordinary strings and arrays created with Dim use HeapAlloc and have no limit other than available memory.
Regards,
Jochen
I agree that you are using a more than sufficient size. My buffer is larger only because it is a general use buffer that serves multiple purposes. Printing in the normal sense is done by use of a que (spool) set up by the system so you are actually creating a bottleneck that generates an error on overflow. You are actually restricting what is actually available instead of making use of available space. In other words, your reserved space is smaller than the que space. Are you aware of this? You could probably increase the capability of your Print function by spooling your own buffer from within your editor if the need is detected by continually dumping to the same buffer when you receive a signal that you can do so. RichEdit does this automatically for you which is why I only need 4mb whereas I can print up to 4gb. Don't think that will ever happen, though, because both of us are only dealing with text files. You need to be aware, though, that printing in RTF style carries a much larger penalty than just the text.
Paul
Quote from: PBrennick on October 10, 2009, 08:24:26 PM
Printing in the normal sense is done by use of a que (spool) set up by the system
Paul,
There is a misunderstanding: I mean printing as used here in the forum, i.e. send a string to the console:
QuotePrint Str$("\nConverting %i hex equates in Windows.inc to decimals", MyCounter), Str$(" took %i ms\n", 0-MyTimer)
The same algo is used for assigning a string:
Quote
Let L$(n)=Left$(L$(n), pos-1)+Str$("EQU %i", esi)+Mid$(L$(n), posAfter)
In both cases, the problem is that the argument macros usually return eax before the Print or Let macros start acting, i.e. inside the
Let macro you get Let eax=eax+eax+eax - very funny ::)
Now in the case of the Print macro, I let the argument macros write to a circular buffer, where I can pick them inside the Print macro by counting backwards. The technique is tricky but fast.
I understand. Programming can be fun or funny. ::)
Paul
Quote from: PBrennick on October 10, 2009, 10:29:17 PM
I understand. Programming can be fun or funny. ::)
Paul
LoL that's so true, you all have very interesting points here, I'd like to see more posts about this, this is getting really interesting! ^^
~
On another note, I see that people say that OS > guest apps, that's not always true though, you can always hack up ring0. :P
Hmmmm,
> On another note, I see that people say that OS > guest apps, that's not always true though, you can always hack up ring0.
Tread carefully here, we would hate to see you disappear forever. :green
Quote from: hutch-- on October 12, 2009, 11:21:02 PM
Hmmmm,
> On another note, I see that people say that OS > guest apps, that's not always true though, you can always hack up ring0.
Tread carefully here, we would hate to see you disappear forever. :green
Lol I don't even know how to do it. :toothy But I've been reading lots about it from a VM security article, it's amazing what assembly can do...