| Package | Description |
|---|---|
| net.sf.okapi.steps.tokenization | |
| net.sf.okapi.steps.tokenization.common | |
| net.sf.okapi.steps.tokenization.engine |
| Class and Description |
|---|
| ILexer
Common set of methods to extract lexems from a string or a list of tokens.
|
| LanguageAndTokenParameters |
| LanguageParameters |
| LexerRule |
| Class and Description |
|---|
| ILexer
Common set of methods to extract lexems from a string or a list of tokens.
|
| LanguageParameters |
| Lexem |
| Lexems |
| LexerRule |
| LexerRules
Lexer rules
|
| StructureParametersItem |
| Token |
| Class and Description |
|---|
| AbstractLexer |
| ILexer
Common set of methods to extract lexems from a string or a list of tokens.
|
| Lexem |
| Lexems |
| LexerRules
Lexer rules
|
Copyright © 2021. All rights reserved.