Uses of Package
net.sf.okapi.steps.tokenization

Classes in net.sf.okapi.steps.tokenization used by net.sf.okapi.steps.tokenization

Class	Description
ITokenizer
ITokenizer.TokenType
Token	TokenType generated by the RbbiTokenizer
TokenConfigs.TokenConfig
TokenizationStep
Tokens	List of Token and configuration for each token type.