tivars.tokenizer

package documentation
(source)

Tokenization utilities derived from the token sheets (see tokens directory)

Module decoder Token stream decoder
Module encoder Context-aware text encoder
Module state NFA implementation for context-aware tokenization with lookahead

From __init__.py:

Class Name Converter for names of vars
Class TokenizedString Converter for data sections best interpreted as strings of tokens