net.sf.okapi.steps.tokenization (Okapi Step for Tokenization 1.43.0 API)

Interface Summary
Interface	Description
ITokenizer

Class Summary
Class	Description
Parameters	Tokenization step parameters
RbbiTokenizer
Token	TokenType generated by the `RbbiTokenizer`
TokenConfigs	TokenType Definitions
TokenConfigs.TokenConfig
TokenizationStep
Tokenizer
Tokens	List of `Token` and configuration for each token type.
TokensAnnotation	Holds a list of `Token`

Package net.sf.okapi.steps.tokenization