How to increase reading speed

[minor28]

I am playing with Scintilla editor and a rather hughe textfile like winuser.h. I read the text line by line to the editor. Elapsed time is just over (~250ms) 8 sec. Writing to the editor does not affect the time significantly.

The same thing with C# using StringBuilder and StreamReader line by line takes just over (~310ms) 1 sec. But with c# I also do some work with each line before writing.

Here is my code.

I would appreciate if someone could propose measures that would reduce time to on par with C#. It would be sad if c# would be so much faster.

Code Select Expand


invoke ReadTextFromFile,offset szWinuser,0,FALSE
mov pFileMap,eax
invoke LockWindowUpdate,hEditor
invoke ParseHeaderFile,pFileMap
invoke LockWindowUpdate,0
invoke UnmapViewOfFile,pFileMap


Code Select Expand


ParseHeaderFile proc pFileMap:dword
LOCAL pCodeLine:dword
LOCAL i:dword
LOCAL pText:dword

push edi
push edx

invoke HeapAlloc,hHeap,HEAP_ZERO_MEMORY,0
mov pCodeLine,eax

invoke GetLineCount,pFileMap
push pFileMap
pop pText
mov i,0
.while i<eax
push eax

invoke ReadLine,i,pText,0,pCodeLine
mov pCodeLine,eax

invoke lstrlen,eax
invoke EdMessage,SCI_APPENDTEXT,eax,pCodeLine
invoke EdMessage,SCI_APPENDTEXT,2,offset szNL

pop eax
inc i
.endw

invoke HeapFree,hHeap,HEAP_NO_SERIALIZE,pCodeLine

pop edx
pop edi

ret

ParseHeaderFile endp


Code Select Expand


ReadLine proc lineNo:dword,pText:dword,Case:dword,pCodeLine:dword
LOCAL count:dword

push edi
push esi
push edx

invoke lstrlen,pText
mov count,eax

.if lineNo==0
invoke GetLineLength,pText,count
inc eax
push eax
invoke HeapReAlloc,hHeap,HEAP_ZERO_MEMORY,pCodeLine,eax
mov pCodeLine,eax

pop ecx
dec ecx
mov esi,pText
mov edi,pCodeLine
rep movsb
mov byte ptr [edi],0

mov eax,pCodeLine
jmp @end
.endif

invoke FindLine,lineNo,pText,count
push eax
push eax
invoke lstrlen,eax
pop ecx
.if eax>0
invoke GetLineLength,ecx,eax
.endif
inc eax
push eax
invoke HeapReAlloc,hHeap,HEAP_ZERO_MEMORY,pCodeLine,eax
mov pCodeLine,eax

pop ecx
dec ecx
pop esi
mov edi,pCodeLine
rep movsb
mov byte ptr [edi],0

mov eax,pCodeLine

@end:
.if Case==1
invoke ToLowerCase,eax
.elseif Case==2
invoke ToUpperCase,eax
.endif

pop edx
pop esi
pop edi

ret ;pointer to line start

ReadLine endp


Code Select Expand


GetLineLength proc pText:dword,count:dword
push edi

mov ecx,count
mov edi,pText
;\r\n = 0Dh 0Ah
mov eax,0Ah ;\n

repne scasb

.if byte ptr [edi - 1]==0Ah
inc ecx
.endif
.if byte ptr [edi - 2]==0Dh
inc ecx
.endif

mov eax,count
sub eax,ecx

pop edi

ret

GetLineLength endp


Code Select Expand


FindLine proc lineNo:dword,pText:dword,count:dword
LOCAL i:dword

push edi

mov i,0
mov ecx,count
mov edi,pText
;\r\n = 0Dh 0Ah
mov eax,0Ah ;\n

@@:
repne scasb
inc i
mov edx,i
.if edx<lineNo && ecx>0
jmp @B
.endif
mov eax,edi
pop edi

ret

FindLine endp


[jj2007]

Elapsed time is just over (~250ms) 8 sec

Is it 8 secs, or 250ms?

Check first what your bottleneck is, e.g. by

Code Select Expand

  ; invoke ReadLine,i,pText,0,pCodeLine
  mov pCodeLine, chr$("one line of average length")

or by commenting out the two SCI_APPENDTEXT messages.

[minor28]

Elapsed time is about 8 sec and 250 msec. Decreace to 7 sec if the two messages are commented out.

Writing the same line 13000 times takes less than 4 sec and commenting out the two messages the elapsed time is about 1 sec.

I just notised that in my c# code I collect all the lines in a StringBuilder and writes "Scintilla1.Text = strBldr.ToString();". I guess this is faster writing.

[jj2007]

I just notised that in my c# code I collect all the lines in a StringBuilder and writes "Scintilla1.Text = strBldr.ToString();".

Try reading the whole file into a buffer, and send SCI_APPEND only once for that buffer.

[minor28]

Less than 1 sec. Reading from filemapping.

[minor28]

This C# snippet write winuser.h to scintilla editor in 0.08 sec.

Code Select Expand


LockWindowUpdate(Scintilla1.Handle);
StringBuilder strBldr = new StringBuilder();
using (StreamReader stream = new StreamReader(fileName))
{
while (!stream.EndOfStream)
{
string line = stream.ReadLine();
strBldr.AppendLine(line);
}
}
Scintilla1.Text = strBldr.ToString();
LockWindowUpdate(IntPtr.Zero);


Processing each line like remove comments, add lines to a treeview, remove certain treeview nodes, translate to masm syntax and back to plain text will take 1.29 sec.

I thought that masm was much faster than C# but I can not get my masm to write faster than 8 seconds. Small variations in micro seconds is the only thing I achieved. Would be interesting to know how StreamReader and StringBuilder are coded.

[hutch--]

Having had a quick look at the source you posted, you are comparing a pile of high level API code to a dedicated algorithm type in C#. If you want to use the advantage of assembler, try coding the base capacity in assembler. Parsing text in assembler is genuinely fast but you have to know what you are doing writing it.

Have a look at a masm32 library module "ltok proc pTxt:DWORD,pArray:DWORD" to perform the initial tokenising into an array.

[dedndave]

istream probably uses OLE (BSTR'ds)
have a look at Hutch's arr* macros and the associated functions

\masm32\help\hlhelp.chm
Macro Catagories - Dynamic String Arrays

[BogdanOntanu]

....
I thought that masm was much faster than C# but I can not get my masm to write faster than 8 seconds. Small variations in micro seconds is the only thing I achieved. Would be interesting to know how StreamReader and StringBuilder are coded.

Of course it is BUT only IF you use the good algorithms and concepts. You are using ASM completly wrong here (bad algorithm, wasting time) and I guess that the C# guys are not so naive.

Learn to optimize algorithmically and conceptually first and use ASM only after you have achieved that.

ASM is NOT the magic bullet that will speed up anything by simply converting it to ASM while still using plain bad concepts.

I see a lot of beginners dreaming that if they write bad code in ASM then it will become faster and better magically.

Unfortunately bad code is bad code in any programming language you use and yes you can write painfully slow things in ASM like you can do it in C# and any other language.

[jj2007]

This C# snippet write winuser.h to scintilla editor in 0.08 sec.

Processing each line like remove comments, add lines to a treeview, remove certain treeview nodes, translate to masm syntax and back to plain text will take 1.29 sec.

I thought that masm was much faster than C# but I can not get my masm to write faster than 8 seconds.

I have no idea what "huge" means, the winuser.h files on the web are tiny (250k or so), but here is an example with the Masm32 Windows.inc file. It reads the file into a string array, parses each of the 26,000 lines, and if it finds an EQU followed by a hex value, it converts the hex value to decimal and changes the line accordingly. Afterwards, it writes the 26,000 lines back to disk.

Of course, such a complex parsing and conversion process takes a lot of time (Converting 8071 hex equates in Windows.inc to decimals took 0.032516 seconds), but try to do the same in C# :green

include \masm32\MasmBasic\MasmBasic.inc   ; download
  Init
  call ConvertWinInc
  Exit

ConvertWinInc proc
LOCAL pos, posAfter, MyCounter
  ClearLocalVariables uses edi esi ebx
  NanoTimer()
  Recall "\Masm32\include\Windows.inc", L$()
  mov ebx, eax
  For_ n=0 To eax-1
   mov pos, Instr_(1, L$(n), "equ", 5)   ; start in pos 1, 1=case-insensitive + 4=full word
   .if pos
      mov esi, Val(Mid$(L$(n), pos+3, 99))
      .if dh                              ; binary or hex
         .if edx!=-127   ; -127 is the "doubtful" flag, we ignore it here
            .if dh==1   ; if dh is set to 1, it was a binary string, otherwise hex
               deb 1, "Found a binary value in Windows.inc", $L$(n)
               movzx edx, dl
            .endif
            movzx edx, dl   ; binary strings return usable chars in dl, plus a flag in dh
            add edx, pos
            add edx, 3
            mov posAfter, edx
            inc MyCounter
            Let L$(n)=Left$(L$(n), pos-1)+Str$("EQU %i", esi)₊Mid$(L$(n), posAfter)   ;+Str$("x%i", ebx)
         .endif
      .endif
   .endif
  Next
  Insert L$(0)
  Let L$(0)="echo ####### Thank you for using the modified version of Windows.inc #######"
  Store "MyWindows.inc", L$()         ; write all strings to file
  Print Str$("Converting %i hex equates in Windows.inc to decimals", MyCounter), Str$(" took %i µs\n", NanoTimer(µs))[/b]
  PopUses
  ret
ConvertWinInc endp
end start

[minor28]

Thank you for your answers.

Ok, I understand the hint. I was not thinking assembly. I'll take it from the beginning with the new insight.

I have a feeling that my state on C# really aroused some feelings.

[hutch--]

minor28,

Here is s simple test piece using a couple of algos from the masm32 library. The timing part run a 46 meg combined C++ header file in 156 ms on my dev 3 gig Core2 Quad. What is being timed is stripping the C/C++ comments then tokenising the entire file into lines in an array. It removes blank lines and trims both ends of each line of text.

Code Select Expand



IF 0  ; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
                      Build this template with "CONSOLE ASSEMBLE AND LINK"
ENDIF ; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    include \masm32\include\masm32rt.inc

    stripcc PROTO :DWORD,:DWORD,:DWORD

    .code

start:
   
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    call main
    inkey
    exit

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

main proc

    LOCAL pCmd  :DWORD
    LOCAL hMem  :DWORD
    LOCAL flen  :DWORD
    LOCAL hBuf  :DWORD
    LOCAL wlen  :DWORD
    LOCAL rval  :DWORD
    LOCAL hArr  :DWORD
    LOCAL acnt  :DWORD
    LOCAL cntr  :DWORD

    mov pCmd, cmd$(1)

    invoke exist,pCmd

    test eax, eax
    jnz @F
      print "No Command Line File",13,10
      ret
    @@:

    mov hMem, InputFile(pCmd)       ; load the command line file
    mov flen, ecx                   ; save its length
    mov hBuf, alloc(flen)           ; allocate an output buffer

    invoke GetTickCount
    push eax

  ; -----------------------------------
  ; strip C comments from C source file
  ; -----------------------------------
    invoke stripcc,hMem,flen,hBuf
    mov wlen, eax

  ; ---------------------------------------------
  ; tokenise stripped text file into a line array
  ; ---------------------------------------------
    invoke ltok,hBuf,ADDR hArr
    mov acnt, eax

    invoke GetTickCount
    pop ecx
    sub eax, ecx
    push eax
    print "Strip and tokenise timing = "
    pop eax
    print ustr$(eax)," ms",13,10

    inkey

    ;;; ret for testing BIG files

  ; ----------------------------------
  ; write the decommented file to disk
  ; ----------------------------------
    mov rval, OutputFile("stripped.txt",hBuf,wlen)

  ; ----------------------------
  ; display the cleaned up lines
  ; ----------------------------
    push esi
    push edi

    mov esi, hArr
    xor edi, edi
  lbl0:
    print [esi],13,10
    add esi, 4
    add edi, 1
    cmp edi, acnt
    jne lbl0

    pop edi
    pop esi

    free hArr
    free hBuf
    free hMem

    ret

main endp

; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««
;               Strip C and C++ comments from source code
; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««

stripcc proc lpsource:DWORD,lnsource:DWORD,lpresult:DWORD

; -------------------------------------------------------------
; stripcc removes C++ comments // and old style C comments
; /*------------- old style C comment -----------------*/
; removes trailing spaces on lines, with or without comments
; -------------------------------------------------------------

    push ebx
    push esi
    push edi

    mov esi, lpsource
    mov edi, lpresult
    mov ecx, lnsource
    add ecx, esi            ; exit condition in ECX

  lbl1:
    mov al, [esi]
    inc esi
    cmp al, "/"
    je comment1
  rtn:
    cmp al, 13              ; branch to trim trailing spaces
    je trimr
  nxt1:
    mov [edi], al
    inc edi
    cmp esi, ecx
    je outa_here            ; exit on source length
    jmp lbl1

  trimr:                    ; trim trailing spaces
    cmp BYTE PTR [edi-1], 32
    jne nxt1
    dec edi
    jmp trimr

  comment1:
    cmp BYTE PTR [esi], "/" ; read next character in ESI
    je cpp
    cmp BYTE PTR [esi], "*"
    je oldc
    jmp rtn                 ; if not a comment, write byte in AL to [EDI]

  cpp:
    mov al, [esi]
    inc esi
    cmp esi, ecx
    je outa_here            ; exit on source length
    cmp al, 13
    je rtn
    jmp cpp

  oldc:
    mov al, [esi]
    inc esi
    cmp esi, ecx
    je outa_here            ; exit on source length
    cmp al, "*"
    je last
    jmp oldc

  last:
    cmp BYTE PTR [esi], "/"
    jne oldc
    inc esi
    jmp lbl1

  outa_here:

    sub edi, lpresult       ; get the byte count written to [edi]
    mov eax, edi            ; set it as the return value

    pop edi
    pop esi
    pop ebx

    ret

stripcc endp

; «««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««««

end start


[minor28]

Thank you Hutch. I will take a closer look at your test piece.

This is my next approuch. Map the file to memory takes about 30ms and the rest 0 ms??? A breakpoint after second GetTickCount makes the elapsed time to show. Could it be less than 1 ms??

Code Select Expand


invoke GetTickCount
push eax

invoke CreateFile,offset szWinuser,GENERIC_READ,0,0,OPEN_EXISTING,FILE_ATTRIBUTE_ARCHIVE,0
.if eax!=INVALID_HANDLE_VALUE
mov hFile,eax
invoke GetFileSize,hFile,0
mov fileLength,eax
inc eax ; for \0

invoke HeapAlloc,hHeap,HEAP_NO_SERIALIZE,eax
mov pFileMap,eax

invoke ReadFile,hFile,pFileMap,fileLength,addr txtRead,0

invoke CloseHandle,hFile
.endif
invoke GetTickCount
pop ecx
push eax
sub eax,ecx
push eax

add txtRead,2
invoke HeapAlloc,hHeap,HEAP_ZERO_MEMORY,txtRead
mov pIncFile,eax

xor edx,edx
mov edi,pFileMap
mov ecx,txtRead
mov eax,0Ah
.while ecx>0
repne scasb
inc edx
.endw
inc edx
add edx,edx
add edx,edx
invoke HeapAlloc,hHeap,HEAP_ZERO_MEMORY,edx
mov pArray,eax

;Convert to string array (lines)
mov edi,pFileMap
mov dword ptr [eax],edi
mov pLine,edi
mov ecx,txtRead
mov lineLength,ecx
;\r\n = 0Dh 0Ah
mov eax,0Ah ;\n
@next:
repne scasb
push eax
push ecx

.if ecx== 0
.if byte ptr [edi - 2] == 0Dh
mov byte ptr [edi - 2],0
.endif
.if byte ptr [edi - 1] == 0Ah
mov byte ptr [edi - 1],0
.endif
add esp,8
jmp @end
.else
mov byte ptr [edi - 1],0
.if byte ptr [edi - 2] == 0Dh
mov byte ptr [edi - 2],0
.endif
push edi

sub lineLength,ecx
push lineLength

mov edi,pIncFile
add edi,txtRead
sub edi,ecx
sub edi,lineLength

pop ecx
mov esi,pLine
rep movsb
mov word ptr [edi-2],0A0Dh ;just for testing

pop edi
mov pLine,edi
.endif

pop ecx
mov lineLength,ecx
pop eax ;restore \n
jmp @next
@end:

invoke EdMessage,SCI_SETTEXT,0,pIncFile

invoke HeapFree,hHeap,HEAP_NO_SERIALIZE,pFileMap

invoke GetTickCount
pop edx
pop ecx
sub eax,ecx
invoke wsprintf,addr buffer,ASTR("%d ms : %d ms"),edx,eax

invoke SetWindowText,hWin,addr buffer


[jj2007]

minor28,

Here is s simple test piece using a couple of algos from the masm32 library. The timing part run a 46 meg combined C++ header file in 156 ms on my dev 3 gig Core2 Quad. What is being timed is stripping the C/C++ comments then tokenising the entire file into lines in an array. It removes blank lines and trims both ends of each line of text.

Hutch

can you zip that C file and post it somewhere?? With the output, so that we can test the correctness? Just for the fun :bg

[minor28]

Hutch,

You are using IMalloc interface function to allocate memory. Is there a special reason for that or is HeapAlloc and HeapReAlloc as good. I mean it is similar functions called to allocate, reallocate and free memory.