tivars.trie.TITokenTrie

class documentation

class TITokenTrie: (source)

View In Hierarchy

Trie for tokenizing text based on tivars.tokens.scripts.TokenTrie

Class Method from_tokens Inserts all tokens from a TITokens container into the trie
Method __init__ Undocumented
Method insert Inserts the names of a TIToken into the trie in a given language
Method match Finds all tokens which can be parsed from a given input string
Instance Variable children Undocumented
Instance Variable token Undocumented
@classmethod
def from_tokens(cls, tokens: TITokens, lang: str = None): (source)

Inserts all tokens from a TITokens container into the trie

Parameters
tokens:TITokensThe tokens to insert
lang:strThe language to insert names from (defaults to English, en)
def __init__(self): (source)

Undocumented

def insert(self, token: TIToken, lang: str = None): (source)

Inserts the names of a TIToken into the trie in a given language

Parameters
token:TITokenThe token to insert
lang:strThe language to insert names from (defaults to English, en)
def match(self, string: str) -> list[tuple[TIToken, str]]: (source)

Finds all tokens which can be parsed from a given input string

Each token is returned with the portion of the input string still remaining. Output is sorted by decreasing length of the consumed input.

Returns
list[tuple[TIToken, str]]A list of tuples each containing a TIToken and its remaining input
children: dict = (source)

Undocumented

token = (source)

Undocumented