tivars.tokenizer

package documentation
(source)

Tokenization utilities derived from the token sheets (see tokens directory)

Module decoder Token stream decoder
Module encoder Context-aware text encoder
Module state Encoder states

From __init__.py:

Class TokenizedString Converter for data sections best interpreted as strings of tokens