tok.t file | source file

Tokenizer - customizable tokenizer class for use with the intrinsic class 'grammar-production' parser.

This tokenizer implementation is parameterized with a set of rules (see below); a basic set of rules is provided, but users can customize the tokenizer quite extensively simply by subclassing the Tokenizer class and overriding the 'rules_' property with a new set of rules declarations.

Summary of Classes

`Tokenizer TokenizerError TokErrorNoMatch`

Summary of Global Objects

(none)

Summary of Global Functions

`main`

Global Functions

`main (args)` | `tok.t[396]`

Test Section

TADS 3 Library Manual
Generated on 5/16/2013 from TADS version 3.1.3