Hi All,
Privately, Hutch has been discussing using a database like SQL to maintain the windows.inc header file(s) for the MASM32 project. Using a database to maintain the Windows equates has been something I've been thinking about for a year or so now (ever since Beth Stone starting making noises about ConvInc over at alt.lang.asm a while back).
Using a database application would make it easier to maintain the data *and* it would also make it very easy to produce different source files based on different assemblers (e.g., such a database could produce FASM, NASM, MASM, HLA, etc., output very simply).
I've been toying around with the idea of creating such an application in Delphi (which makes database design quite simple). Anyone out there share an interest in doing something like this? I suspect that the application itself would be fairly easy to write, the real work will be populating the database (and verifying it). While a bit of automated conversion could be done (not unlike how I generated the first HLA w.hhf header file), there would still need to be a fair amount of work. Any interest?
Cheers,
Randy Hyde
I will be interested to see what ideas turn up here. I have learnt so far that an Access interface is probably not the way to go and it will need to be a database format that is easily accessible programatically as the output needs to be controlled to get the include file right.
The original idea I floated around was for a database but I know it can be done as a collection of text files that are written together into a single file for output. It is probably worth avoiding a manual entry system as additions tend to occur in large numbers and the data is obtained through conversions and subsequent testing. The other factor that is important is that the include file must remain properly backwards compatible so that it does not break a mountain of existing code.
QuoteI suspect that the application itself would be fairly easy to write, the real work will be populating the database (and verifying it). While a bit of automated conversion could be done (not unlike how I generated the first HLA w.hhf header file), there would still need to be a fair amount of work.
Perhaps I'm not understand what you are proposing, but it seems to me that the potential for error would be much lower if most of the data were generated through automatic conversion. And I think the primary source for the conversion should be the MASM32 windows.inc, warts and all. And while I do understand the benefits of using a HLL for something like this, considering the target audience, I think the whole thing should be done in ASM, preferably MASM.
What I have done so far is manually split the windows.inc file up into recognisable components and have written a toy that combines the components back into windows.inc. This has so far allowed me to put together some disconnected values that belong to a single category which helps to make the file a lot easier to work on.
It is a left over of how the file was created years ago. I did the oiginal win32 version of it and Iczelion added the WinNT stuff a bit later and the file got out of conrol in terms of size and layout. It is in reasonable shape at the moment because it has had a lot of work done on it but I must get it into editable form without it being such a pig to work on as I would like to be able to finalise it and start on a later include file that has the XP and later stuff in it.
It would be far easier to create a new one but it generates problems of backwards compatibility that are very hard to solve so I have not gone down that path because of the risk of breaking a mountain of code that many different people have written.
Why don't you like access database? I use them with ODBC in my asm programs and are very easy to use.
To avoid backward compatiility problems, anyone is free to choose between old windows.inc and new windows.inc.
The old will be unmantained. If someone wants to migrate to new windows.inc can rewrite some parts of his code, otherwise he will use the old windows.inc
[suggestion]
I don't like the current structure of windows.inc
Why don't use a structure similar to MS SDK include files? Just translate the files and use the same file hierarchy, with the same #define.
We must change the extension of the .inc file.
We have .inc files that contain proto directives and other inc files that contain stuct and equate declarations.
Maybe we can change the extension of the former into ash or put protos,structures and equates into the same file
[/suggestion]
Quote from: MichaelW on January 21, 2005, 05:59:39 AM
QuoteI suspect that the application itself would be fairly easy to write, the real work will be populating the database (and verifying it). While a bit of automated conversion could be done (not unlike how I generated the first HLA w.hhf header file), there would still need to be a fair amount of work.
Perhaps I'm not understand what you are proposing, but it seems to me that the potential for error would be much lower if most of the data were generated through automatic conversion. And I think the primary source for the conversion should be the MASM32 windows.inc, warts and all. And while I do understand the benefits of using a HLL for something like this, considering the target audience, I think the whole thing should be done in ASM, preferably MASM.
Automatic conversion sounds wonderful, but the devil is in the details.
For example, the current HLA header files *were* originally created using an "automated conversion program." Alas, no conversion is perfect, and the result had several errors introduced by the conversion process. Furthermore, as HLA is semantically richer than MASM, a straight conversion of MASM header files (particularly things like API prototypes) isn't what you really want. Finally, conversion of things like macros is a real problem.
Even if one could solve the problem for MASM <-> HLA, there's still FASM, NASM, GoAsm, etc., to worry about. There was some talk on A.L.A. a year ago or so about creating a "canonical header file format" that could be easily converted to other assemblers. As is typical for A.L.A., the talk yielded very little other than talk, but the idea is still a good one -- provide a generic database of declarations that can be easily translated to all popular assemblers. If the information were put into some sort of database program, emission of header files for a particular assembler would be pretty easy. The hard part is deciding how to arrange the tables in the database and how to represent data (for example, the traditional "everything is a dword" approach is *not* good; it's easy to convert things like "uns32" or "int32" into "dword" or "sdword", but going the other direction is impossible).
Cheers,
Randy Hyde
This would not in any way solve the backward compatibility problem, but...
Assuming that most of the data in the current assembler header files was derived from C/C++ header files, why not store the data in that format? You could populate the database directly from the original header files, minimizing the need to verify the correctness of the data, and the conversion would be fairly simple, minimizing the number of errors introduced by conversion process.
And considering the current low price of disk space and the speed of recent processors, and assuming the data would be split up into reasonable sized files, I can see no good reason to store the data as anything other than text.
And not to offend anyone here, but such a database could be useful for HLL programmers as well.
Quote from: MichaelW on January 26, 2005, 01:41:36 AM
This would not in any way solve the backward compatibility problem, but...
Assuming that most of the data in the current assembler header files was derived from C/C++ header files, why not store the data in that format? You could populate the database directly from the original header files, minimizing the need to verify the correctness of the data, and the conversion would be fairly simple, minimizing the number of errors introduced by conversion process.
And considering the current low price of disk space and the speed of recent processors, and assuming the data would be split up into reasonable sized files, I can see no good reason to store the data as anything other than text.
And not to offend anyone here, but such a database could be useful for HLL programmers as well.
check out those header files some time.
They contain lots of macros that will not translate (easily or reasonably) into assembly language. Sure, we could edit them so that they only contain statements that are easy to translate into assembly, but if we do that, we may as well create a more reasonable canonical form that is *very* easy to translate between assemblers. A database application seems like a reasonable approach to me.
Cheers,
Randy HYde
hi guys, I'm back again since a long time working on other projects and still trying to get started in learning this.
Being avid in using MS Access a lot, I would like to say, please dont use MySQL for a database for such a seemingly simple thing.
After all, it may be adaptable to all platforms, but it is hogging up at least 11 Mb RAM just for being installed on your system.
Something not even the slow little ms access has done. Besides, once you get the hang on ADO to Access you can in
fact run 255 simultanous connects over the internet to a access db rather than being limited to 10 users over a whole server.
Anyway, just wanted to toss my experience in to that topic, cause it's something close to my daily routines.
I agree with "greenant" and would like to "refloat" what I have already said:
http://www.masmforum.com/simple/index.php?topic=98.0
There is a similar thing done with the VC 6.0 SDK by japheth at:
http://www.japheth.de/
Forgive me if my first post on these forums is off-topic, but if i undestand this correctly...
Extracting info to build these include files from .h .cpp etc seems very awkward and I praise you for what you have achieved so far.
You mentioned a database which sparked an old program I used to play with before I ditched VB6 many moons ago.
VB6 comes with a program called APILOAD, that contains 3+ text files with most win api functions/consts in it.
The program can convert these to access db for speed too.
The original text files (eg. WIN32API.TXT) as very very very easy to parse. I believe more upto date versions are released with office developer and later VS. (I'm not sure about this but MSDN seems to mention them abit).
The reason I mentioned it, given they may not be as definitive as you need, importing them is trivial.
Having a button to generate code for a given assembler and bang it on the clipboard is easy. So a button for each assembler can be put on, or a combo to select the desired assembler.
I would post one of the text files here but it may not be legal and I wouldn't want to get banned on my first post. :lol
moldy,
Welcome on board.
Sorry but , win32api.txt , isn't it a copyrighted material of Microsoft?
Vortex,
if you take the win32api.txt explode it apart
and then compile it into a different format
where are you getting into copyright shit?
You are making something new.
Zcoder....
AFAIK it would still be copyrighted as a derivative work, but in this case modifying and distributing it should be no problem:
' ------------------------------------------------------------------------
'
' WIN32API.TXT -- Win32 API Declarations for Visual Basic
'
' Copyright (C) 1994 Microsoft Corporation
'
'
' This file contains only the Const, Type,
' and Declare statements for Win32 APIs.
'
' You have a royalty-free right to use, modify, reproduce and distribute
' this file (and/or any modified version) in any way you find useful,
' provided that you agree that Microsoft has no warranty, obligation or
' liability for its contents. Refer to the Microsoft Windows Programmer's
' Reference for further information.
'
' ------------------------------------------------------------------------
There is an updated version here:
http://vb.mvps.org/tools/
Quote from: zcoder on March 15, 2006, 07:50:09 PM
Vortex,
if you take the win32api.txt explode it apart
and then compile it into a different format
where are you getting into copyright shit?
You are making something new.
Zcoder....
zcoder, let's keep our language more friendly. All what I do is to ask a simple question : is it copyrighted or not? How do you conclude that I am doing something new?
Vortex,
Take the win32api.txt and try to assemble
in asm with it, it won't work it's in a diffrent
format then what assemblers need. so if you
change the format of what is in the file you
have made something new.
Infact, Microsoft wants you or I or anyone to
do so. Example if I made up a new language
Microsoft would love it if I made their file
compatible to my new Language and
assembler or compiler.
An old saying is, the more support for
a OS the more users will use it.
whis is why Unix or Red hat is not
what everyone is using as the #1
OS, not mush software out their
that runs in it.
If you think about it you should be
able to see why MS does not care
what form you put their win32api.txt
file into as long as you realize that they
retain rights on the contents.
Zcoder....
Randall Hyde,
Oh, your idea is right and good.
I thought about this idea for about 6 months now. Infact I wrote a simple php script that lets me add( in groups)
the equates, structs, and API's in raw form from the VC++ h files.
Then I wrote a small script to parse some of these out to file in MASM form and it looks promising. So I have been adding to the
database and playing with adding more parsing to the PHP script in hopes of being able to make a windows.inc.
In PHP I can even do string testing easy to tell when an Item should appear in the file. Example is you have to make sure a
Structure, is not placeed in the file before an item in that structure is defined. PHP is good for this.
So I agree that this idea is good for this type of operation. Data base's are good for sorting, and everything like that.
Lets just say I have done this in ASM and C++ trying to sort, convert ect and I get more error's here and there.
but what I have done so far in PHP the output file had been error free and DUP free, so I have been working
harder on this but doing this alone building the database up is slow at best.
Right now I have been making a script to take a H file explode it and place it into the database each struct, equate ect.
I am also storing them all in groups as to what file they had came from as I want INC files in this same structre.
I want all API's, Structures, and Equates in a Found H file to be in a INC file, so I don't have to have a LARGE Windows.inc file.
I am suprised to see that someone had came up with the same idea or at least the idea to use a database to help sort the junk out.
Zcoder....
QuoteI am also storing them all in groups as to what file they had came from as I want INC files in this same structre.
I want all API's, Structures, and Equates in a Found H file to be in a INC file, so I don't have to have a LARGE Windows.inc file.
Yes..yes, I agree and I definitely want to be a part of this one !!! :green
QuoteI am suprised to see that someone had came up with the same idea or at least the idea to use a database to help sort the junk out.
Not a "database" idea...but atleast the grouping thing !!!
(My previous post (and the responses :P to it) http://www.masmforum.com/simple/index.php?topic=98 )
Most of these suggestions come to grief when it comes to the format of the PLATFORMSDK header files which are a genuine bundle to parse and each successive version gets more complex. Much of the content will not convert to either MASM or POASM format and there are many other complex problems to solve to get much of it to run with an assembler.
Microsoft have long ago given up on H2INC so i suggest there is no simple way to get the data and this is after spending 8 years on the current version. Contrast this with getting the API prototype data from the libraries as its needed by the linker and apart from using a non proprietry method to scan for names that does not get broken each time Microsoft change the library format internally, it has produced super reliable include files for years where each include matches the library it is to be used with.
The current windows.inc is about 1.2 meg which is hardly a problem on modern machines. Include files of that size worked fine on my mid 90s P166 so its not as if its a machine capacity problem.
With the introduction of POASM, it became another problem in that POASM loads a complete symbol table and stops with an error with a duplicate that MASM would tolerate if it did not find the duplicate so I wrote a hash table based duplicate checking app just to solve this problem which either displays the duplicates to the console or automatically removes them depending on the option selected.
What most people don't understand is the massive tme it takes to get this stuff right and every time I have shared this file with someone else, I have had to spend days to weeks to fix it so the version I distribute with the masm32 and poasm projects will not be a collective version as I don't need the extra work.
If the only gain is to be able to ape the problems in using Microsoft C headers where there is no published correlation between the include file you use and the library you use, I just don't want it as the existing method is proven reliable.
zcoder,
All what I wanted to know was the copyright status of that txt file. Everything is fine for me, now, I know that you can use that file for your own purposes. It's OK for me.
To show you what I meant.
From your windows.inc file
CHOOSEFONTA STRUCT
lStructSize DWORD ?
hwndOwner DWORD ?
hDC DWORD ?
lpLogFont DWORD ?
iPointSize DWORD ?
Flags DWORD ?
rgbColors DWORD ?
lCustData DWORD ?
lpfnHook DWORD ?
lpTemplateName DWORD ?
hInstance DWORD ?
lpszStyle DWORD ?
nFontType WORD ?
Alignment WORD ?
nSizeMin DWORD ?
nSizeMax DWORD ?
CHOOSEFONTA ENDS
CHOOSEFONT equ <CHOOSEFONTA>
Now from the vb6 files
Type CHOOSEFONT
lStructSize As Long
hwndOwner As Long ' caller's window handle
hdc As Long ' printer DC/IC or NULL
lpLogFont As Long
iPointSize As Long ' 10 * size in points of selected font
flags As Long ' enum. type flags
rgbColors As Long ' returned text color
lCustData As Long ' data passed to hook fn.
lpfnHook As Long ' ptr. to hook function
lpTemplateName As String ' custom template name
hInstance As Long ' instance handle of.EXE that
' contains cust. dlg. template
lpszStyle As String ' return the style field here
' must be LF_FACESIZE or bigger
nFontType As Integer ' same value reported to the EnumFonts
' call back with the extra FONTTYPE_
' bits added
MISSING_ALIGNMENT As Integer
nSizeMin As Long ' minimum pt size allowed &
nSizeMax As Long ' max pt size allowed if
' CF_LIMITSIZE is used
End Type
All of the definitions in the apiload directory are in this format. Is it not straightforward to extract this into a database (as the original app did) and just copy paste the relevant structures/constants etc into your programs as you design them? (again as the original app did)
As i mentioned in my first post. The vb files are trivial to parse into an ASM format. (as you would expect from such a basic language...whoops bad pun)
Comments and relevant defined values for structures could easily be copy/pasted at the same time.
This would breakdown the incompatabilities between various assemblers and possibly remove the burdon of constantly updating certain include files.
Maybe you could freeze the original includes and recommend using the new include/copy/paste/db to avoid breaking older code.
Again, I hope I am on the same train of thought of you peeps. :U
moldy,
The VB file is only a small part of the complete header set from the platformsdk. Basic is very easy to parse where the C++ include files are just about impossible without detailed knowledge of the guts of CL.EXE. The problem is keeping up with the additions every time a newer Windows version is released.