tivars.trie.TITokens

class documentation

class TITokens: (source)

Constructor: TITokens(tokens)

View In Hierarchy

Data class for storing collections of TIToken instances based on tivars.tokens.scripts.Tokens

TIToken instances may be obtained from various maps:

  • The byte map is indexed by token bytes.
  • The lang map is indexed by language code, then token name.
  • The name map is indexed by token name, regardless of language.

The byte and name maps may be accessed via __getitem__.

Additionally, a trie map contains a TITokenTrie for each language, indexed by language code.

Method __getitem__ Undocumented
Method __init__ Undocumented
Instance Variable bytes Undocumented
Instance Variable langs Undocumented
Instance Variable names Undocumented
Instance Variable tries Undocumented
def __getitem__(self, item: bytes | str) -> TIToken: (source)

Undocumented

def __init__(self, tokens: Tokens): (source)

Undocumented

bytes = (source)

Undocumented

langs = (source)

Undocumented

names = (source)

Undocumented

tries = (source)

Undocumented