Two high speed tokenisers for PowerBASIC.

[hutch--]

Instead of paddling around trying to get some grunt out of legacy basic, here are two seperate high speed tokenisers that are written in assembler, one to tokenise words and the other to tokenise lines of CRLF terminated text. They are both "in place" tokenisers and were converted from the masm32 library. One required a string length function so it has a tweaked version of Agner Fog's original DWORD StrLen written for it.

The source text argument in both tokenisers are DWORD addresses as this allows different string formats to be used with the two functions, basic string using "StrPtr()",, ASCIIZ strings using "VarPtr()" and the direct memory address of allocated memory returned from the various memory allocation strategies in the Windows API functions.

If you need serious performance in parsing large volumes of text data, functions of this type will free you from legacy style basic and make you competitive with C/C++ compilers and assembler code.

In the "ltoktst.bas" example you will need to set the path,

    Open "\pbwin80\include\win32api.inc" for Binary as #1

to the location where your own WIN32API.INC file is located.

PS: i should have noted, these two algos are written to work on either PowerBASIC compiler, the examples are written using the console compiler so it could display results at the console. This makes the code compatible with the current two versions, pbwin80 and pbcc40.

[attachment deleted by admin]