tivars.tokenizer.state.MaxMode

class documentation

class MaxMode(TokenizerState): (source)

Constructor: MaxMode(mode, accept, length)

View In Hierarchy

Maximal munching mode

Method __init__ No summary

Inherited from TokenizerState:

Method munch Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string
Method next Determines the next tokenizer timelines given a token
Class Variable max_length The maximum number of tokens to emit before leaving this state
Instance Variable accept Undocumented
Instance Variable length Undocumented
Instance Variable mode Undocumented
def __init__(self, mode: int = 0, accept: bool = True, length: int = 0): (source) ΒΆ
overrides tivars.tokenizer.state.TokenizerState.__init__
Parameters
mode:intWhether to munch maximally (0) or minimally (-1)
accept:boolWhether this state can end a timeline (defaults to True)
length:intThe current length of the input this state is going to process (defaults to 0)