tivars.tokenizer.state.ListName

class documentation

class ListName(Name): (source)

Constructor: ListName(length)

View In Hierarchy

List names

Class Variable max_length The maximum number of tokens to emit before leaving this state

Inherited from Name:

Method next Determines the next encode state given a token
Class Variable mode Whether to munch maximally (0) or minimally (-1)

Inherited from EncoderState (via Name, Line):

Method __init__ Undocumented
Method munch Munch the input string and determine the resulting token, encoder state, and remainder of the string
Instance Variable length Undocumented
max_length: int = (source)
overrides tivars.tokenizer.state.EncoderState.max_length

The maximum number of tokens to emit before leaving this state