tivars.tokenizer.decoder

module documentation

(source)

Token stream decoder

Function	`decode`	Decodes a byte stream into a string of tokens
Function	`detokenize`	Stringifies a sequence of `TIToken` objects given a language and token representation type
Function	`parse`	Parses a byte stream into a list of `TIToken` objects

def decode(data: bytes, *, model: TIModel = TI_84PCE, lang: str = None, mode: str = None) -> str: (source) ¶

Decodes a byte stream into a string of tokens

Parameters
data:`bytes`	The token bytes to decode
model:`TIModel`	A model for which compatibility is ensured (defaults to the TI-84+CE)
lang:`str`	The language used in `string` (defaults to the locale of `model`, or English, `en`)
mode:`str`	The token representation to use for output (defaults to `display`)

Returns
`str`	A string representing the tokens in `data`

def detokenize(tokens: Sequence[TIToken], *, model: TIModel = TI_84PCE, lang: str = None, mode: str = None): (source) ¶

Stringifies a sequence of TIToken objects given a language and token representation type

Parameters
tokens:`Sequence[TIToken]`	The `TIToken` sequence to stringify
model:`TIModel`	The target model (defaults to the TI-84+CE)
lang:`str`	The target language (defaults to the locale of `model`, or English, `en`)
mode:`str`	The token representation to use for output (defaults to `display`)

Returns
A string representing `tokens`

def parse(bytestream: bytes, *, tokens: TITokens = None) -> list[TIToken]: (source) ¶

Parses a byte stream into a list of TIToken objects

Each token is represented using one of three different representations formats, dictated by mode:

display: Represents the tokens with Unicode characters matching the calculator's display
accessible: Represents the tokens with ASCII-only equivalents, often requiring multi-character glyphs
ti_ascii: Represents the tokens with their internal font indices (returns a bytes object)

Parameters
bytestream:`bytes`	The token bytes to parse
tokens:`TITokens`	The `TITokens` object to use for decoding (defaults to the TI-84+CE tokens)

Returns
`list[TIToken]`	A list of `TIToken` objects assembled from `bytestream`