Extracting data from this forum

[Robert Collins]

I am building a knowledge data-base that I will use for my own personal needs. The items in the data base are then read into a Web Page that has a tree view of the items on the left and the selected item on the right side (as seen in the included picture below).

I have the tools needed to transfer the data from the data base to the Web Page once the data is in the data base but what I don't have is a decent way to get the data. As you can see, I have started with this project using the topics from this forum (I plan on expanding this to include topics from other forums that I am a member of).

For me to capture this information I do it all manually (ie, I parse out what I need from each HTML source page) and piece all the information together and then put it in the data base. My question is, is there some way I can obtain this information with a much simplier way?

Hutch, if this is something you prefer that I do not do please let me know and I will not continue with this project.

[roticv]

Ask for the database from hutch and making sure he removes the password table.

[hutch--]

Robert,

The server for the forum is different to the server that carries the MYSQL database and even I cannot directly get at it. The security around the database for obvious reasons is so tight that what you are after is not possible. I can get the database backup but then there are other matters that I have to take notice of. As per the rules of the forum, anyone who posts code that they own is treated as the owner of that code and while by posting it in the forum, they make it available to other members for their own use, I do not attempt to control any of that code as I don't own it so I am not in a position to pass the database to anyone for any reason.

I don't see any problem with you collecting data from the forum as this is why members post code but there are too many reasons to allow the database to be passed around as it involves ownership issues as well as security issues.

[Robert Collins]

Hutch....

I understand completely.

The main idea of what I am doing is to be able to see all the posts (or as many as I can obtain) in a nutshell, so to speak. The tree view is for that purpose. This saves me alot of time as I can simply scroll down the tree view list for topics that I want to read into further.

There is another Web Site, VBCode.com, that has over a 1000 entries of usefule topics and sample codes. The big problem that I had with this site is that only about 5 - 10 items are visible per page. To see more I have to click on the next page button at the bottom of each page and this takes a great deal of time because each page has to load. To avoid this problem I made some 'quick-n-dirty' programs to connect to VBCode.com and through the program it loads each and every page from that site. The program then parses out all of the topic links per page and builds a database of these links (with topic titles). The database is then input to another program (similar to the one shown in my first posting here) which builds a tree view of all the entries. Now, I can see all the topics from a list and click on the ones that I am interested in at that time. The program that parses out the links takes about 2 hours to run but in the meantime I can do other things. It saves me much more time than that once finished.

Once I have the programs finished to do the dirty work I can then launch them like every month so as to keep my database updated with all the new entries.

I want to do the same with this site (and others) for the same reason. I have already completed a program that retrieves all the links for the topic postings from the 'Orphanage' forum.

I was just thinking that maybe there could be a simplier way but it's not a big deal and writing 'q-n-d's to do the work is OK.