tivars.tokenizer.encoder

module documentation
(source)

Context-aware text encoder

Function encode Encodes a string of token represented as text into a byte stream
Function normalize Applies NFC normalization to a given string to ensure recognition of certain Unicode characters used as token names
Function tokenize Tokenizes a string of tokens represented as text into a list of TIToken objects
Function unparse Concatenates a TIToken sequence into a bytestream
def encode(string: str, *, model: TIModel = TI_84PCE, lang: str = None, mode: str = None) -> bytes: (source)

Encodes a string of token represented as text into a byte stream

For detailed information on tokenization modes, see tivars.tokenizer.tokenize.

Parameters
string:strThe text string to encode
model:TIModelA model to target when encoding (defaults to no specific model)
lang:strThe language used in string (defaults to the locale of model, or English, en)
mode:strThe tokenization mode to use (defaults to smart)
Returns
bytesThe bytes comprising the tokens represented by string
def normalize(string: str): (source)

Applies NFC normalization to a given string to ensure recognition of certain Unicode characters used as token names

Parameters
string:strThe text to normalize
Returns
The text in string normalized
def tokenize(string: str, *, trie: TITokenTrie = None, mode: str = None, normalize: bool = True) -> list[TIToken]: (source)

Tokenizes a string of tokens represented as text into a list of TIToken objects

Tokenization is performed using one of three procedures, dictated by mode:
  • max: Always munch maximally, i.e. consume the most input possible to produce a token
  • min: Always munch minimally, i.e. consume the least input possible, which is often single characters
  • smart: Munch maximally or minimally depending on context
The smart tokenization mode uses the following contexts, munching maximally otherwise:
  • Strings: munch minimally, except when interpolating using Send( or storing to an equation
  • Program names: munch minimally up to 8 tokens
  • List names: munch minimally up to 5 tokens
In all modes:
  • Standard glyphs can be used for substituting Unicode symbols
  • xXX and uUUUU output the denoted bytes exactly, regardless of validity
  • ABCD maximally munches ABCD, regardless of surrounding context
  • Certain unprintable characters act as a hard separator for tokens
    • U+001F (unit separator): ␟
    • U+200A (hair space): ` `
    • U+200C (zero width non-joiner): ‌
For reference, here are the tokenization modes utilized by popular IDEs and other software:
  • SourceCoder: max
  • TokenIDE: max
  • TI Connect CE: ¯_(ツ)_/¯
  • TI-Planet Project Builder: smart
  • tivars_lib_cpp: smart
Parameters
string:strThe text string to encode
trie:TITokenTrieThe TokenTrie object to use for tokenization (defaults to the TI-84+CE trie)
mode:strThe tokenization mode to use (defaults to smart)
normalize:boolWhether to apply NFC normalization to the input before encoding (defaults to true)
Returns
list[TIToken]A list of TIToken objects represented by string
def unparse(tokens: Sequence[TIToken]) -> bytes: (source)

Concatenates a TIToken sequence into a bytestream

Parameters
tokens:Sequence[TIToken]The tokens to encode
Returns
bytesThe bytes comprising tokens