I have recently been playing with loading a tree view control with complete directory listings for each drive and while its reasonably straight forward code to use FindFirstFile() - FindNextFile() and it is showing the correct results but it takes a lot longer to do the first pass on a large drive than a number of other applications that use a tree view control for the same task. I have set the task in a recursive algo that climbs up and down the directory structure loading the nodes of the tree view control on the fly.
As I don't claim to have done all that much work with tree view controls to list directory structures, I wonder if anyone is familiar with the speed problem I have with the control when loading this much data into it.
These are the two procedures I am using to load the directory tree into the tree view control.
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
load_tview proc hTree:DWORD,drv:DWORD
; arguments
; 1 tree view handle
; 2 address drive string "c:\" etc ...
LOCAL tvi :TV_INSERTSTRUCT
LOCAL itx :TV_ITEMEX
LOCAL hRoot :DWORD
LOCAL var :DWORD
LOCAL hThread :DWORD
fn LoadCursor,NULL,IDC_WAIT
fn SetCursor,eax
fn SetCurrentDirectory,drv
fn SendMessage,hTree,WM_SETREDRAW,FALSE,0
fn SendMessage,hTree,TVM_DELETEITEM,0,TVI_ROOT
mov tvi.hParent, TVI_ROOT
mov tvi.hInsertAfter, TVI_FIRST
mov tvi.itemex.imask, TVIF_TEXT
m2m tvi.itemex.pszText, drv
mov hRoot, rv(SendMessage,hTree,TVM_INSERTITEM,0,ADDR tvi)
fn recurse_tree,hRoot
invoke SendMessage,hTree,TVM_EXPAND,TVE_EXPAND,hRoot
fn SendMessage,hTree,WM_SETREDRAW,TRUE,0
ret
load_tview endp
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
recurse_tree proc hParent:DWORD
LOCAL wfd :WIN32_FIND_DATA
LOCAL hFile :DWORD
LOCAL tvi :TV_INSERTSTRUCT
LOCAL itx :TV_ITEMEX
LOCAL hItem :DWORD
LOCAL nfrv :DWORD
LOCAL pnam :DWORD
mov hFile, rv(FindFirstFile,"*",ADDR wfd)
cmp hFile, 0
je overit
DIRN = FILE_ATTRIBUTE_DIRECTORY
DIRR = FILE_ATTRIBUTE_DIRECTORY or FILE_ATTRIBUTE_READONLY
DIRH = FILE_ATTRIBUTE_DIRECTORY or FILE_ATTRIBUTE_HIDDEN
DHRO = FILE_ATTRIBUTE_DIRECTORY or FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_READONLY
.if wfd.dwFileAttributes == DIRN || wfd.dwFileAttributes == DIRR || \
wfd.dwFileAttributes == DIRH || wfd.dwFileAttributes == DHRO
lea eax, wfd.cFileName
mov pnam, eax
switch$ pnam
case$ "."
jmp nextfile
case$ ".."
jmp nextfile
endsw$
m2m tvi.hParent, hParent
mov tvi.hInsertAfter, TVI_LAST
mov tvi.itemex.imask, TVIF_TEXT
m2m tvi.itemex.pszText, pnam
mov hItem, rv(SendMessage,hTview,TVM_INSERTITEM,0,ADDR tvi)
chdir ADDR wfd.cFileName
invoke recurse_tree,hItem
.endif
nextfile:
mov nfrv, rv(FindNextFile,hFile,ADDR wfd)
cmp nfrv, 0
je overit
.if wfd.dwFileAttributes == DIRN || wfd.dwFileAttributes == DIRR || \
wfd.dwFileAttributes == DIRH || wfd.dwFileAttributes == DHRO
lea eax, wfd.cFileName
mov pnam, eax
switch$ pnam
case$ "."
jmp nextfile
case$ ".."
jmp nextfile
endsw$
m2m tvi.hParent, hParent
mov tvi.hInsertAfter, TVI_LAST
mov tvi.itemex.imask, TVIF_TEXT
m2m tvi.itemex.pszText, pnam
mov hItem, rv(SendMessage,hTview,TVM_INSERTITEM,0,ADDR tvi)
chdir ADDR wfd.cFileName
invoke recurse_tree,hItem
.endif
jmp nextfile
overit:
chdir ".."
invoke FindClose,hFile
ret
recurse_tree endp
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
Is it creation of an index that takes the time?.... This is not the treeview but the filesystem functions....
If you're using a treeview simply to display the contents of the drive then recursing it will be very slow indeed. There could be 100's of thousands of files to find and if you access anything about them, even just their name, they could be indexed. The way Windows Explorer does it is to use "Just In Time" this is also the technique I used in WinExplorer. The idea is to read the contents of the root folder and display only that. When someone expands a folder you read that folder, adding its contents to the treeview, and so on. This will make it appear that you have read the entire drive contents in the blink of an eye and no one will be the wiser. This is the only practical way I have found to do a Treeview Directory, on a system like mine at home which has 1 Terabyte of drive space and probably around 600,000 files and folders, it would be impractical to recurse through the whole drive. Windows search which is the fastest recurser I know takes nearly 3 minutes.
Yes, I guess that makes sense, the info is a bit thin on the ground with a Google search but I did find one Microsoft article that described the "Just in time" approach.
Unrelated I have a small application from the old forum called WinExplorer dated about 2004, no menu, a set of buttons along the top and a combo box control to set the drives. It works OK but has no name on it. Written in MASM.
You will most likely come across the problem again if you add search functionality that searches the drive.... Just a heads up, again as Edgar suggested a more narrow approach helps ie defining lower level root folders....
Still if there is a fix for this somehow it would be nice.... Atm I can only think to maintain my own index and monitor the hard drive for any use where the delay between broad searches is too large....
With your own index you would be able to recurse extremely fast but it is also a lot more work.... I have not attempted it with ASM only in VB years ago....
I believe file system monitoring also slows your system however so there is some tradeoff here also....
Quote from: hutch-- on March 24, 2011, 03:02:28 AM
Unrelated I have a small application from the old forum called WinExplorer dated about 2004, no menu, a set of buttons along the top and a combo box control to set the drives. It works OK but has no name on it. Written in MASM.
I think that that might be EWayne's, haven't heard from him in a while though.
BTW WinExplorer (mine) is available on my website if you want to take a look. The code for Just In Time is pretty winding but you can take a look at how it performs and decide whether its right for your application. If you just want to try the app, I've attached the executable.
Gratsie, works fine here.
It works for me too...but when I pressed the last icon (Most recently used) it doesn't works..Win7 64bit here.
I wrote that something like 6 years ago, it is what it is. 64 bit Windows also has problems with the show imports/exports function, the structure was wrong I think or something like that, when it was written there weren't any examples I had access to to test it and I never bothered to take it up again. Just a quick peek at the source shows that it predates the header project and I used the MASM32 definitions for structures (ie imask instead of mask), that makes it pretty old. In the end it was never much more than a platform for me to test shell and gui ideas.
EDIT: A quick peek at the version on my website shows that the MRU button isn't even there, must have been looking into it then decided against it. There's no handler for it that I can find in the source. I uploaded my working copy to this thread so it probably has a lot of potential errors.
having played with a tree view control in my spare time over the last couple of days, I am surprised at how they actually work. The WM_NOTIFY interface is reasonably straight forward but I am fascinated by the effect that within a number of the notification message handlers you get instant GP faults if you attempt to either send or post a number of tree view control messages. Recovering the path back to root was easy enough to do from the TVN_SELCHANGED notification but try it anywhere else and BANG.
This code is an instant GP fault.
case TVN_ITEMEXPANDED
push esi
mov esi, rv(SendMessage,hTview,TVM_GETNEXTITEM,TVGN_CARET,0)
invoke PostMessage,hTview,TVM_SELECTITEM,TVGN_CARET,0
invoke PostMessage,hTview,TVM_SELECTITEM,TVGN_CARET,esi
pop esi
I have one working example written in MASM and I have just had a look at Edgar's GoAsm code and they both use a technique that I did not expect, basically a hit test approach on the visible portion of the tree view window. It looks like I will have to redesign the test pieces I have to do this as the messaging has unusual overlaps that cause GP faults.
Set the lParam value of each inserted node to the handle of its parent (NULL for root items.) Tracing a path to the root is then a sequence of TVM_GETITEM calls, which I would hope shouldn't cause any issues since it requires no modification of the treeview state.
Also, I'd respond to TVN_ITEMEXPANDING for inserting your new child nodes (before the branch actually expands.)
Plus a small hint about WM_SETREDRAW when inserting any potentially large number of items :wink
Gratsie,
I have been getting unusual results, did the WM_SETREDRAW with no problems and can load the tree in one pass from a recursive directory algo but have had problems trying to do the "just in time" version. The complete version works fine but is slow when it crawls an entire directory tree. Where I have tried partial listings I have had the problem that when I click on the unexpanded branch and try and recurse the directory tree from that location I get an instant GP fault.
I have the TVN_ITEMEXPANDING and TVN_ITEMEXPANDED notifications working correctly and both indicate expanding and collapsing the tree but as soon as I try and modify the tree from those notifications it goes BANG. Tried posting a message to WM_COMMAND but same result.
It's your lucky day - have a working example :bg
very nice Tedd :U
:bg
A gentleman and a scholar and if that offends you at least a scholar. :U
Long time ago I "invented" that DispatchMessage API is faster than SendMessage API,
so I like this prog and re-wrote it with DispatchMessage API only... :wink
Lingo,
Thanks for the example. I have run it alongside Tedd's but it has a distinct lag on the first click on a drive. I tested it back and forth against Tedd's but his is always faster on the drive click where there are many subdirectories on the first level of expansion.
"it has a distinct lag on the first click"
Every job has its technological time. :P
If you want to "hid out" it particularly just comment the "push offset msg1 + call edi" and
"push offset msg2 + call edi" in the "MWP_WM_NOTIFY:" part.
I updated it but with FindFirst & FindNext I can't achieve more speed.... :wink
Lingo,
Thansk for the example but it has the same problem, Tedd's is a lot faster on the first click, I have been testing it on deeply nested directories and Tedd's is uniformly fast where both of yours have a distinct lag when you click on a deeply nested directory. I have not had a good look at it but are you recursing extra depths in each pass with the FindFirstFile() code ?
"of yours have a distinct lag when you click on a deeply nested directory"
I tested it before to post the new update and they (my last update and Tedd's) have equal times for
"distinct lag" on the first click after restarting the PC.
I tested with C:\Windows\winsxs folder of Win7 64bit OS and the time is 15 sec for both.
Next I tested with C:\Windows\winsxs folder of Vista 64bit and the time is 29 sec for both.
For me the bottleneck is in FindFirst & FindNext APIs.
When I have time I will try with NtQueryDirectoryFile from Ntdll.dll.. :P
Is this even necessary, Lingo? It was just a quick example I put together on a Sunday afternoon. I realise you have a need to make everything faster, but wouldn't you waste your time much more efficiently elsewhere? :lol
For a freshly booted system, the initial traversing of the directory-tree is going to dominate the time, no matter what method you use - so they will both take roughly the same time in that instance.
"It was just a quick example I put together on a Sunday afternoon"
I like it. :wink :wink
"I realise you have a need to make everything faster"
It is an answer of your 1st question: "Is this even necessary, Lingo?" :P
"waste your time much more efficiently elsewhere?"
I did it but now I'm married and must obey... :lol
"For a freshly booted system, the initial traversing of the directory-tree is going to dominate the time,
no matter what method you use - so they will both take roughly the same time in that instance."
I agree.. :P
FindFirstFile is slow - i think we all know that
i managed to find one alternative, although, there may be others
QuoteThe best way to solve your problem seems to be by using the Windows Change Journal.
Problem: If it is not enabled for a volume or the volume is a non-NTFS you need a fallback (or enable
the Change Journal if it is NTFS). You need administrator rights as well to access the Change Journal.
You get the files by using the FSCTL_ENUM_USN_DATA and DeviceIOControll with LowUsn=0. This
directly accesses the MFT and writes all filenames into the supplied buffer. Because it sequentially
acesses the MFT it is faster than the FindFirstFile API.
http://stackoverflow.com/questions/4203573/finding-a-set-of-file-names-quickly-on-ntfs-volumes-ideally-via-its-mft
it seems to me that there may be a way to read the contents of a folder as a file
although, i am not NTFS-savvy enough to know how :P
on a related note, i have learned that the ms picture/fax viewer can be extremely slow opening a folder
if that folder contains any dead shortcut files (LNK shortcut files that point to non-existung stuff)
this leads me to believe that it tries to locate the targets
not sure if it also applies to URL files :lol
anyways - thought i would mention it as food for thought
i think if i really wanted to research the subject in depth, i would start with an e-mail to Mark Russinovich :bg
It is old stuff from Jeffrey Richter rather than by Russinovich link (http://technet.microsoft.com/en-us/library/bb742450.aspx)...
I am in debt to Tedd for having decyphered the JIT technique for loading a tree view control. On my dev box and the i7 I have alongside it to watch TV, Tedd's technique is clearly the fastest and this is for first time scanning the drive where the JIT technique only scans directories required for display at the selected tree level. I wrote earlier code that did a full drive crawl but on first pass of a drive it was seeing scan time of 30+ seconds with high disk activity for the entire scan time.