tivars.tokenizer.state.String

class documentation

class String(Line): (source)

Strings

Method next Determines the next tokenizer timelines given a token

Inherited from TokenizerState (via Line):

Method	`__init__`	No summary
Method	`munch`	Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string
Class Variable	`max_length`	The maximum number of tokens to emit before leaving this state
Instance Variable	`accept`	Undocumented
Instance Variable	`length`	Undocumented
Instance Variable	`mode`	Undocumented

def next(self, token: TIToken, remainder: str) -> list[list[TokenizerState]]: (source) ¶

overrides tivars.tokenizer.state.Line.next

Determines the next tokenizer timelines given a token

Each timeline that will branch from this one is returned as a list of states to add to a copy of the stack.

If a list of states in a timeline is...

empty, then the timeline is exiting the current state.
length one, then the timeline's current state is being replaced by a new state.
length two, then the timeline is entering a new state, able to exit back to this one.

Parameters
token:`TIToken`	The current token
remainder:`str`	The remaining string content to tokenize

Returns
`list[list[TokenizerState]]`	A list of timelines (each a list of states)