How do I speed up this treeview insert?

[Gunner]

I use a Treeview in many of my apps and really like em...  But am having issues with speed on an app I am working on.. 
I open a csv file and parse each line into an array.
Then I parse each array member for field 3, and field 4, and the whole field of strings will be in the tvitem lparam for later parsing...
I can load a 2 meg file (about 6500 lines) in about 2 seconds by doing JUST the above...  The slowdown happens when I add the items to the tree..

I take field 3, if tree is empty, then add as root and field 4 as child..  If tree is not empty,
Compare all root text and if it is in tree then add field 4 as child....

This is no problem with 100 or so treeview items.. but with the 2 meg file it takes about 10 or so seconds..

How can I speed this up?  I just don't know what to search for on the board  :(

Code Select Expand

        mov     ebx, RecordInfo
       
        push    0
        push    0
        push    TVM_GETCOUNT
        push    hTVMainUser
        call    SendMessage
        test    eax, eax

        jnz     AddEm

AddFirstOne:
        mov     tvis.hParent, TVI_ROOT
        mov     tvis.hInsertAfter, TVI_SORT
        mov     tvis.item.imask, TVIF_TEXT or TVIF_STATE or TVIF_IMAGE or TVIF_SELECTEDIMAGE
        mov     tvis.item.iImage, 0
        mov     tvis.item.iSelectedImage, 0
        mov     tvis.item.state, TVIS_BOLD or TVIS_EXPANDED
        mov     tvis.item.stateMask, TVIS_BOLD or TVIS_EXPANDED
        mov     eax, (LogFields PTR [ebx]).szDate
        mov     tvis.item.pszText, eax     
        invoke  SendMessage, hTVMainUser, TVM_INSERTITEM, 0, addr tvis

        mov     tvis.item.imask, TVIF_TEXT or TVIF_PARAM or TVIF_IMAGE or TVIF_SELECTEDIMAGE
        mov     tvis.item.iImage, 1
        mov     tvis.item.iSelectedImage, 1
        mov     tvis.hParent, eax
        mov     eax, (LogFields PTR [ebx]).szIP
        mov     tvis.item.pszText,  eax
        mov     tvis.item.lParam, ebx
        invoke  SendMessage, hTVMainUser, TVM_INSERTITEM, 0, addr tvis
        ret

AddEm:
        ; Get root node handle
        push    0
        push    TVGN_ROOT
        push    TVM_GETNEXTITEM
        push    hTVMainUser
        call    SendMessage
        test    eax, eax
        jnz     @F
        ret
           
@@:
        mov     edi, eax
        invoke  memfill, addr RootText, sizeof RootText, NULL
        mov     tvi._mask, TVIF_TEXT
        mov     tvi.hItem, edi
        mov     tvi.cchTextMax, 50
        lea     ecx, RootText
        mov     tvi.pszText, ecx
        lea     eax, tvi
        push    eax
        push    0
        push    TVM_GETITEM
        push    hTVMainUser
        call    SendMessage     
        lea     eax, RootText
       
        invoke  Cmpi, eax, (LogFields PTR [ebx]).szDate
        .if eax == 0
            mov     tvis.item.imask, TVIF_TEXT or TVIF_PARAM or TVIF_IMAGE or TVIF_SELECTEDIMAGE
            mov     tvis.item.iImage, 1
            mov     tvis.item.iSelectedImage, 1
            mov     tvis.hParent, edi
            mov     eax, [ebx].LogFields.szIP
            mov     tvis.item.pszText, eax
            mov     tvis.item.lParam, ebx
            invoke  SendMessage, hTVMainUser, TVM_INSERTITEM, 0, addr tvis
            ret
        .endif

CheckNextRoot:
        push    edi
        push    TVGN_NEXT
        push    TVM_GETNEXTITEM
        push    hTVMainUser
        call    SendMessage
        mov     esi, eax
        test    eax, eax
        jz      AddFirstOne     
   
        invoke  memfill, addr RootText, sizeof RootText, NULL
        mov     tvi._mask, TVIF_TEXT
        mov     tvi.hItem, esi
        mov     tvi.cchTextMax, 50
        lea     ecx, RootText
        mov     tvi.pszText, ecx
        lea     eax, tvi
        push    eax
        push    0
        push    TVM_GETITEM
        push    hTVMainUser
        call    SendMessage     
        lea     eax, RootText
       
        invoke  Cmpi, eax, (LogFields PTR [ebx]).szDate
        .if eax == 0
            mov     tvis.item.imask, TVIF_TEXT or TVIF_PARAM or TVIF_IMAGE or TVIF_SELECTEDIMAGE
            mov     tvis.item.iImage, 1
            mov     tvis.item.iSelectedImage, 1
            mov     tvis.hParent, esi
            mov     eax, (LogFields PTR [ebx]).szIP
            mov     tvis.item.pszText, eax
            mov     tvis.item.lParam, ebx
            invoke  SendMessage, hTVMainUser, TVM_INSERTITEM, 0, addr tvis
            ret
        .endif
        mov     edi, esi
        jmp     CheckNextRoot

[oex]

Is treeview visible? If it is hide it.... Sorry if stupid answer :lol

[dedndave]

well - it could probably be sped up a little
but, i doubt that sending the messages is the big bottleneck
it is how the messages are handled that is time-consuming
you are showing us the wrong end of the stick

[Gunner]

Give me a sec

[sinsi]

WM_SETREDRAW maybe?

[dedndave]

but, if you have to pass the address of a structure - and you want to go a little faster...

Code Select Expand

       push     structure.member6
       push     structureMember5
       push     structureMember4
       push     structureMember3
       push     structureMember2
       push     structureMember1
       INVOKE  SomeAPI,parm1,parm2,esp
       add     esp,(4 * NumberOfDwordMembers)   ;24 total bytes, in this example
note that the structure pointer is passed as the last parm in this example, as it is in most of what i see in your code
if it is not the last parm (first parm pushed), you have to play it a little differently

it also makes for smaller code, as opposed to all those MOV's

[Gunner]

the code that calls this is a menu handler

Code Select Expand

LoadTree proc uses esi edi ebx
LOCAL   lpMem:DWORD
LOCAL   lpLen:DWORD
LOCAL   pArray:DWORD
LOCAL   lcnt:DWORD
LOCAL   pArray2:DWORD
LOCAL   lcnt2:DWORD
LOCAL   TempCount[8]:BYTE


    invoke  read_disk_file, addr lpszLogFile, addr lpMem, addr lpLen
    test    eax, eax
    jnz     OpenIt
    invoke  MessageBox, Main.hMain, offset szErrMsg, offset szErrTitle, MB_ICONERROR
    jmp     Done
   
OpenIt:
    invoke  ShowWindow, hTVMainUser, SW_HIDE
    invoke  ShowWindow, hProgress, SW_SHOW
    invoke  InvalidateRect, hProgress, 0, TRUE
    invoke  ltok, lpMem, addr pArray
    mov     lcnt, eax
    invoke  memfill, addr lpszTotalRecords, sizeof lpszTotalRecords, NULL
    invoke  dwtoa, lcnt, addr lpszTotalRecords
    invoke  memfill, addr lpszRecords, sizeof lpszRecords, NULL
    invoke  szMultiCat, 3, addr lpszRecords, offset szOf, addr lpszTotalRecords, offset szRecords
    mov esi, pArray
    mov edi, lcnt
    xor ebx, ebx
    invoke  GetWindowDC, hProgress
    mov     hDC, eax
    invoke  SelectObject, hDC, hFontBold
    mov     hObj,eax
  lbl0:
    add ebx, 1
   
    invoke  memfill, addr lpszLoading, sizeof lpszLoading, NULL
   invoke   memfill, addr TempCount, sizeof TempCount, NULL
   invoke   dwtoa, ebx, addr TempCount
    invoke  szMultiCat, 3, addr lpszLoading, offset szLoading, addr TempCount, addr lpszRecords
    invoke  szLen, addr lpszLoading
    invoke TextOut,hDC,1,1,addr lpszLoading, eax



;    invoke ParseLine, [esi]
;;;;;;;;;;;;;;;;;;;;;;;;;
        pushad
        mov     esi, [esi]
        invoke  HeapAlloc, hMainHeap, HEAP_ZERO_MEMORY, sizeof LogFields
        mov     ebx, eax
        invoke  szLen, esi
        add     eax, 1
        invoke  HeapAlloc, hMainHeap, HEAP_ZERO_MEMORY, eax
        mov     (LogFields PTR [ebx]).szFullString, eax
        invoke  szCopy, esi, (LogFields PTR [ebx]).szFullString

    SkipRecordField: ; Fixed field     
        invoke  InString, 1, esi, offset szComma
        add     esi, eax
   
    GetDateField:
        invoke  HeapAlloc, hMainHeap, HEAP_ZERO_MEMORY, 12
        mov     (LogFields PTR [ebx]).szDate, eax
        mov     edi, (LogFields PTR [ebx]).szDate
    @@:
        mov     ax, word ptr [esi]
        add     esi, 2
        cmp     al, ","
        je      SkipTimeField
        mov     word ptr [edi], ax
        add     edi, 2
        mov     byte ptr [edi], "-"
        inc     edi
        jmp     @B
       
    SkipTimeField: ; Fixed field
    ;PrintText "Getting Time"
        dec     edi
        mov     byte ptr [edi], 00H
       
        invoke  InString, 1, esi, offset szComma
        add     esi, eax
       
        invoke  InString, 1, esi, offset szComma
        mov     edi, eax
        invoke  HeapAlloc, hMainHeap, HEAP_ZERO_MEMORY, eax
        mov     (LogFields PTR [ebx]).szIP, eax
        dec     edi
        invoke  szLeft, esi, (LogFields PTR [ebx]).szIP, edi
        invoke  szRemove, (LogFields PTR [ebx]).szIP, (LogFields PTR [ebx]).szIP, offset szQuote
        invoke  AddItem, ebx
        popad
;;;;;;;;;;;;;;;;;;;;;

    add esi, 4

    cmp ebx, edi
   
    jl lbl0
     ;
Done:
    invoke  GlobalFree, pArray
    invoke  GlobalFree, lpMem
   
    invoke  ShowWindow, hProgress, SW_HIDE 
    invoke  ShowWindow, hTVMainUser, SW_SHOW
       
    invoke  DeleteObject,hObj
    invoke  ReleaseDC, hProgress, hDC
    ret
LoadTree endp

AddItem is the code in the first post..  But the bottleneck seems to be in AddItem, because if I don't call it, I can load and parse in a jiffy  :bg

[donkey]

When adding items to the treeview its best not to have it redrawn constantly, one of the most time consuming aspects of the control. Use LockWindowUpdate to lock out the window until you are done with populating it.

http://msdn.microsoft.com/en-us/library/dd145034%28v=vs.85%29.aspx

[dedndave]

sure am glad we have Edgar around   :U

[Gunner]

When adding items to the treeview its best not to have it redrawn constantly, one of the most time consuming aspects of the control. Use LockWindowUpdate to lock out the window until you are done with populating it.

http://msdn.microsoft.com/en-us/library/dd145034%28v=vs.85%29.aspx

Yes, that is one of the first things I tried...  Doesn't seem to make a difference, since I hide the tree during loading.. I dunno.

[sinsi]

LockWindowUpdate was my first thought, but MSDN says

LockWindowUpdate is not intended for general-purpose suppression of window redraw. Use the WM_SETREDRAW message to disable redrawing of a particular window.

[Gunner]

WM_SETREDRAW maybe?

Maybe?  YES!  That seems to do the trick!  Using that message, got the tree loading down from about 10 or so seconds to about 6 seconds!  For a 2 meg file, I think that is livable... but can it get better?

There is TVN_GETDISPINFO/TVN_SETDISPINFO would that help?  Don't know how to use it though...

[oex]

It is a very long time since I used.... I used to have (very) large treeviews on one application I wrote and although there were methods to speed it up which you have probably used all of here and more another method I used was to selectively add as required but as I say this was 5-10 years back maybe and in VB6 so if there were issues with this also I dont remember :lol

[dedndave]

all those memfill calls shouldn't be needed, Rob

Code Select Expand

invoke  memfill, addr lpszTotalRecords, sizeof lpszTotalRecords, NULL
just use

Code Select Expand

mov byte ptr lpszTotalRecords,0
or, if AL happens to be 0

Code Select Expand

mov lpszTotalRecords,al


this one isn't needed at all

Code Select Expand

invoke   memfill, addr TempCount, sizeof TempCount, NULL

[jj2007]

Hi Gunner,
The loading part could be done faster - I load (with MB Recall) a 13MB csv file in about 32 ms. But the bottleneck is not there.
Check if there is an equivalent to the LB_INITSTORAGE message - that would yield a factor 30-40 improvement. However, MSDN has a rather pessimistic view:

Performance Tips

This section provides practical recommendations for using the TreeView.
Tree Size

Rendering enormous hierarchical data sets in a single TreeView data island is not recommended. As a general rule, try to keep the number of nodes in a tree under 1000.

Each time an update is made to the UI of a TreeView control, for example, whenever a node is expanded or collapsed, additional data is appended to the view state, which maintains a history of state changes in the Web page. This means that each time a view is updated with a large data island, the amount of data appended to the view state history is increased. In ASP.NET, there is a limit in the size of the view state, so the combination of very large tree hierarchies combined with multiple updates may cause performance problems, and in certain cases portions of the history could be lost.