Package net.sf.eBus.text

TokenLexer takes a given input and returns a series of analyzed tokens.

See: Description

Class Summary

Class	Description
Token	Contains the token's type, corresponding Java object, raw text and the input line number where the token was found.
TokenLexer	Provides a generalized token lexer capability.
TokenLexerContext

Enum Summary

Enum	Description
TokenLexer.LexMode	The lexer will either analyze the tokens identifying the type or collect raw input until a terminating delimiter is found.

Package net.sf.eBus.text Description

TokenLexer takes a given input and returns a series of analyzed tokens. These tokens contain:

• A token type. May be either a pre-defined or user-defined.
• The token text value as collected from the input.
• The Java object created from the token text.
• The line number on which the token appears.

The most important difference between Lexer and java.util.StringTokenizer is that Lexer analyzes the tokens into types. Further, tokens are not created by breaking a line based on inter-field separator characters or by regular expression. Instead, Lexer searches for tokens based on types. This lexical package is designed to work with a higher level parser rather than a line-by-line text processor.