tivars.tokenizer.state

module documentation

(source)

NFA implementation for context-aware tokenization with lookahead

Class	`IllegalState`	Tokenizer state which indicates its timeline must be pruned
Class	`Line`	State which is always exited after a line break or STO
Class	`ListName`	List names
Class	`MaxMode`	Maximal munching mode
Class	`MinMode`	Minimal munching mode
Class	`Name`	Valid var identifiers
Class	`ProgramName`	Program names
Class	`SmartMode`	Smart tokenization mode
Class	`String`	Strings
Class	`StringStart`	Opening quote of a string
Class	`StringSto`	STO immediately following a string
Class	`StringTarget`	STO target of a string
Class	`TokenizerState`	Base class for tokenizer states