tivars.tokenizer.state.String

class documentation

class String(Line): (source)

Known subclasses: tivars.tokenizer.state.InterpolatedString

Constructor: String(length)

View In Hierarchy

Strings

Method next Determines the next encode state given a token
Class Variable mode Whether to munch maximally (0) or minimally (-1)

Inherited from EncoderState (via Line):

Method __init__ Undocumented
Method munch Munch the input string and determine the resulting token, encoder state, and remainder of the string
Class Variable max_length The maximum number of tokens to emit before leaving this state
Instance Variable length Undocumented
def next(self, token: TIToken) -> list[EncoderState]: (source)
overrides tivars.tokenizer.state.Line.next

Determines the next encode state given a token

The current state is popped from the stack, and the states returned by this method are pushed.

If the list of returned states is...
  • empty, then the encoder is exiting the current state.
  • length one, then the encoder's current state is being replaced by a new state.
  • length two, then the encoder is entering a new state, able to exit back to this one.
Parameters
token:TITokenThe current token
Returns
list[EncoderState]A list of encoder states to add to the stack
mode: int = (source)
overrides tivars.tokenizer.state.EncoderState.mode
overridden in tivars.tokenizer.state.InterpolatedString

Whether to munch maximally (0) or minimally (-1)