Package net.sf.okapi.steps.tokenization.common

Interface Summary

Interface	Description
ILexer	Common set of methods to extract lexems from a string or a list of tokens.

Class Summary

Class	Description
AbstractLexer
Config
InputTokenAnnotation
LanguageAndTokenParameters
LanguageParameters
Lexem
Lexems
LexerRule
LexerRules	Lexer rules
RegexRule
RegexRules
StructureParameters
StructureParametersItem
Token
TokensAnnotation