Uses of Interface
com.aliasi.tokenizer.TokenCategorizer

Packages that use TokenCategorizer

Package	Description
com.aliasi.chunk	Classes for extracting meaningful chunks (spans) of text.
com.aliasi.tokenizer	Classes for tokenizing character sequences.

Uses of TokenCategorizer in com.aliasi.chunk

Constructors in com.aliasi.chunk with parameters of type TokenCategorizer

Constructor and Description
TrainTokenShapeChunker(TokenCategorizer categorizer, TokenizerFactory factory) Construct a trainer for a token/shape chunker based on the specified token categorizer and tokenizer factory.
TrainTokenShapeChunker(TokenCategorizer categorizer, TokenizerFactory factory, int knownMinTokenCount, int minTokenCount, int minTagCount) Construct a trainer for a token/shape chunker based on the specified token categorizer, tokenizer factory and numerical parameters.

Uses of TokenCategorizer in com.aliasi.tokenizer

Classes in com.aliasi.tokenizer that implement TokenCategorizer

Modifier and Type	Class and Description
class	CharacterTokenCategorizer Returns a category for tokens made up out of a single character.
class	IndoEuropeanTokenCategorizer A IndoEuropeanTokenCategorizer is a generic token categorizer for Indo-European languages that is based on character "shape".