tivars.tokenizer.state.ListName

class documentation

class ListName(Name): (source)

List names

Class Variable max_length The maximum number of tokens to emit before leaving this state

Inherited from Name:

Method next Determines the next tokenizer timelines given a token

Inherited from MinMode (via Name):

Method __init__ No summary

Inherited from TokenizerState (via Name, MinMode, Line):

Method	`munch`	Munch the input string and determine the resulting token, tokenizer timelines, and remainder of the string
Instance Variable	`accept`	Undocumented
Instance Variable	`length`	Undocumented
Instance Variable	`mode`	Undocumented

overrides tivars.tokenizer.state.TokenizerState.max_length

The maximum number of tokens to emit before leaving this state