tivars.tokenizer.state.MinMode

class documentation

class MinMode(EncoderState): (source)

Constructor: MinMode(length)

View In Hierarchy

Minimal munching mode

Class Variable mode Whether to munch maximally (0) or minimally (-1)

Inherited from EncoderState:

Method __init__ Undocumented
Method munch Munch the input string and determine the resulting token, encoder state, and remainder of the string
Method next Determines the next encode state given a token
Class Variable max_length The maximum number of tokens to emit before leaving this state
Instance Variable length Undocumented
mode: int = (source)
overrides tivars.tokenizer.state.EncoderState.mode

Whether to munch maximally (0) or minimally (-1)