tivars.tokenizer.state.SmartMode

class documentation

Smart tokenization mode

Method	`__init__`	No summary
Method	`next`	Determines the next tokenizer timelines given a token

Inherited from TokenizerState:

Method	`munch`	Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string
Class Variable	`max_length`	The maximum number of tokens to emit before leaving this state
Instance Variable	`accept`	Undocumented
Instance Variable	`length`	Undocumented
Instance Variable	`mode`	Undocumented

def __init__(self, mode: int = 0, accept: bool = True, length: int = 0): (source) ¶

overrides tivars.tokenizer.state.TokenizerState.__init__

Parameters
mode:`int`	Whether to munch maximally (`0`) or minimally (`-1`)
accept:`bool`	Whether this state can end a timeline (defaults to `True`)
length:`int`	The current length of the input this state is going to process (defaults to `0`)

def next(self, token: TIToken, remainder: str) -> list[list[TokenizerState]]: (source) ¶

overrides tivars.tokenizer.state.TokenizerState.next

Determines the next tokenizer timelines given a token

Each timeline that will branch from this one is returned as a list of states to add to a copy of the stack.

If a list of states in a timeline is...

empty, then the timeline is exiting the current state.
length one, then the timeline's current state is being replaced by a new state.
length two, then the timeline is entering a new state, able to exit back to this one.

Parameters
token:`TIToken`	The current token
remainder:`str`	The remaining string content to tokenize

Returns
`list[list[TokenizerState]]`	A list of timelines (each a list of states)