Package net.sf.eBus.text
TokenLexer takes a given input and
returns a series of analyzed tokens. These tokens contain:
- A token type. May be either a pre-defined or user-defined.
- The token text value as collected from the input.
- The Java object created from the token text.
- The line number on which the token appears.
The most important difference between Lexer and
java.util.StringTokenizer is that Lexer analyzes the
tokens into types. Further, tokens are not created by
breaking a line based on inter-field separator characters or
by regular expression. Instead, Lexer searches for tokens
based on types. This lexical package is designed to work with
a higher level parser rather than a line-by-line text
processor.
-
Class Summary Class Description Token Contains the token's type, corresponding Java object, raw text and the input line number where the token was found.TokenLexer Provides a generalized token lexer capability. -
Enum Summary Enum Description TokenLexer.LexMode The lexer will either analyze the tokens identifying the type or collect raw input until a terminating delimiter is found.