tivars.tokenizer.state.TokenizerState

class documentation

class TokenizerState: (source)

Known subclasses: tivars.tokenizer.state.IllegalState, tivars.tokenizer.state.Line, tivars.tokenizer.state.MaxMode, tivars.tokenizer.state.MinMode, tivars.tokenizer.state.SmartMode

Constructor: TokenizerState(mode, accept, length)

View In Hierarchy

Base class for tokenizer states

Each state represents some encoding context which affects tokenization.

Method __init__ No summary
Method munch Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string
Method next Determines the next tokenizer timelines given a token
Class Variable max_length The maximum number of tokens to emit before leaving this state
Instance Variable accept Undocumented
Instance Variable length Undocumented
Instance Variable mode Undocumented
def __init__(self, mode: int, accept: bool = True, length: int = 0): (source)
overridden in tivars.tokenizer.state.MaxMode, tivars.tokenizer.state.MinMode, tivars.tokenizer.state.SmartMode
Parameters
mode:intWhether to munch maximally (0) or minimally (-1)
accept:boolWhether this state can end a timeline (defaults to True)
length:intThe current length of the input this state is going to process (defaults to 0)
def munch(self, string: str, trie: TITokenTrie) -> tuple[TIToken, str, list[list[TokenizerState]]]: (source)

Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string

Parameters
string:strThe text string to tokenize
trie:TITokenTrieThe TokenTrie object to use for tokenization
Returns
tuple[TIToken, str, list[list[TokenizerState]]]A tuple of the output Token, the remainder of string, and a list of timelines
def next(self, token: TIToken, remainder: str) -> list[list[TokenizerState]]: (source)
overridden in tivars.tokenizer.state.Line, tivars.tokenizer.state.Name, tivars.tokenizer.state.SmartMode

Determines the next tokenizer timelines given a token

Each timeline that will branch from this one is returned as a list of states to add to a copy of the stack.

If a list of states in a timeline is...
  • empty, then the timeline is exiting the current state.
  • length one, then the timeline's current state is being replaced by a new state.
  • length two, then the timeline is entering a new state, able to exit back to this one.
Parameters
token:TITokenThe current token
remainder:strThe remaining string content to tokenize
Returns
list[list[TokenizerState]]A list of timelines (each a list of states)
max_length: int = (source)
overridden in tivars.tokenizer.state.ListName, tivars.tokenizer.state.ListName, tivars.tokenizer.state.ProgramName, tivars.tokenizer.state.ProgramName

The maximum number of tokens to emit before leaving this state

accept = (source)

Undocumented

length = (source)

Undocumented

mode = (source)

Undocumented