All Classes (Okapi Step for Tokenization 1.44.0 API)

All Classes Interface Summary Class Summary Enum Summary
Class	Description
ITokenizer
ITokenizer.TokenType
Parameters	Tokenization step parameters
RbbiTokenizer
Token	TokenType generated by the `RbbiTokenizer`
TokenConfigs	TokenType Definitions
TokenConfigs.TokenConfig
TokenizationStep
Tokenizer
Tokens	List of `Token` and configuration for each token type.
TokensAnnotation	Holds a list of `Token`