A matching in place tokeniser for text line data

[hutch--]

This version tokenises lines of text in place with the characteristics of skipping blank lines and trimming any leading tabs and spaces from the line. It outputs an array of pointers and the line count. This algo is well suited for setting up an array of words for sorting but equally well handles sentences, phrases etc ...

The example has the algo testing simple data for correctness, sorts the same data and a trailing benchmark from the windows.inc file which again is too small to get meaning ful timings from.

[attachment deleted by admin]

[James Ladd]

Hutch,

Very nice indeed. I learn so much from you so thanks for taking the time to do and post stuff like
this. Now if only I could get you to join me personal project?

Note: The ltok program expects /masm32/include/windows.inc to exist !

Rgs, James.

[hutch--]

I modified the original by removing the table as it could be replaced by one compare which reduces one memory access for each loop iteration and re-ordered the second loop so that the most common case has the shortest loop. Algo is smaller and should be faster. As before, the hard coded example using windows.inc is too small to produce meaningful results.

[attachment deleted by admin]