Does any one knows a good reference book of creating debuggers, as well as the format of debugging symbols in windows executables and object files.
Intel used to have systems programming manuals that dealt with some of this.
Microsoft has documentation for the PE/COFF files, SYM symbol files, and the CodeView format. The PDB format is poorly documented, and there are multiple format variations, and internal CodeView symbol representation. The general route to accessing symbols is to use several Microsoft DLL's provided for that purpose, or to write your own. You'd have to dig through the SDK and MSDN releases over the years for some of the details.
I'm sure Intel and AMD has current documentation for their internal operation, but you'd have to get familiar with paging, task context, control and debug registers, etc. to stand much of a chance. Then you'd need to look at the OS from a kernel programming perspective.
The only references i have seen are too old to be of any use; in protected mode, creating a debugger is a very O/S specific thing. Unlike DOS, where the debugger is essentially all-knowing, in windows it's really just another user mode application. Unless, that is, you are looking to write a kernel-mode debugger, which is damn near impossible; since the kernel is stopped, you have to write handle ALL your own input, using your own drivers, from scratch; it's basically like writing your own rudimentary operating system
Also, for the record, you probably won't get much help with the specifics, as most of the debugging related functions (ReadProcessMemory, etc) are of dubious forum appropriateness. Basically, you open the memory of the process you are debugging, save the byte at the location you want to stop at, overwrite it with an INT 3, and when the exception occurs, put the original byte back.
http://msdn.microsoft.com/en-us/library/ms679288(v=VS.85).aspx
-r
this is the only book i can find
http://www.amazon.com/How-Debuggers-Work-Algorithms-Architecture/dp/0471149667
i actually have this book in djvu format. but i haven't got around to reading it yet
Thanks everyone.