Uses of Package
com.aliasi.tokenizer

Packages that use com.aliasi.tokenizer

Package	Description
com.aliasi.chunk	Classes for extracting meaningful chunks (spans) of text.
com.aliasi.classify	Classes for classifying data and evaluation.
com.aliasi.cluster	Classes for clustering data and evaluation.
com.aliasi.coref	Classes for determining entity coreference within documents.
com.aliasi.crf	Classes and interfaces for conditional random fields.
com.aliasi.dict	Classes for handling dictionaries.
com.aliasi.lm	Classes for character- and token-based language models.
com.aliasi.sentences	Classes for sentence-boundary detection.
com.aliasi.spell	Classes for spelling correction and edit distance.
com.aliasi.suffixarray	Classes for spelling correction and edit distance.
com.aliasi.test.unit.tokenizer
com.aliasi.tokenizer	Classes for tokenizing character sequences.

Classes in com.aliasi.tokenizer used by com.aliasi.chunk

Class and Description
TokenCategorizer A TokenCategorizer supplies a string-based category for string-based tokens.
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.classify

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.cluster

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.coref

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.crf

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.dict

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.lm

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.sentences

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.spell

Class and Description
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.suffixarray

Class and Description
Tokenization A Tokenization represents the result of tokenizing a string.
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.test.unit.tokenizer

Class and Description
Tokenizer The abstract class Tokenizer serves as a base for tokenizer implementations, which provide streams of tokens, whitespaces, and positions.
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.

Classes in com.aliasi.tokenizer used by com.aliasi.tokenizer

Class and Description
IndoEuropeanTokenCategorizer A IndoEuropeanTokenCategorizer is a generic token categorizer for Indo-European languages that is based on character "shape".
IndoEuropeanTokenizerFactory An IndoEuropeanTokenizerFactory creates tokenizers with built-in support for alpha-numerics, numbers, and other common constructs in Indo-European langauges.
LineTokenizerFactory A LineTokenizerFactory treats each line of an input as a token.
ModifiedTokenizerFactory A ModifiedTokenizerFactory is an abstract tokenizer factory that modifies a tokenizer returned by a base tokenizer factory.
ModifyTokenTokenizerFactory The abstract base class ModifyTokenTokenizerFactory adapts token and whitespace modifiers to modify tokenizer factories.
RegExTokenizerFactory A RegExTokenizerFactory creates a tokenizer factory out of a regular expression.
StopTokenizerFactory A StopTokenizerFactory modifies a base tokenizer factory by removing tokens in a specified stop set.
TokenCategorizer A TokenCategorizer supplies a string-based category for string-based tokens.
Tokenizer The abstract class Tokenizer serves as a base for tokenizer implementations, which provide streams of tokens, whitespaces, and positions.
TokenizerFactory A TokenizerFactory constructors tokenizers from subsequences of character arrays.