public abstract class ModifyTokenTokenizerFactory extends ModifiedTokenizerFactory implements Serializable
ModifyTokenTokenizerFactory
adapts token and whitespace modifiers to modify tokenizer
factories.
The method modifyToken(String) may be used to
modify or remove tokens from tokenizer outputs. The method
modifyWhitespace(String) may be used to modify the
whitespace returned by a tokenizer. Both methods are given
pass-through implementations in this class.
Serializable.
There are no serialization methods defined, so the default
serialization is used. There is a single reference to the base
tokenizer factory in the parent class, so a subclass will be
serializable if all of its member objects are serializable and the
base tokenizer is serializable.
It is good practice for each subclass to take completecontrol
over serialization using a serialization proxy implemented
on top of the AbstractExternalizable base
class.
| Constructor and Description |
|---|
ModifyTokenTokenizerFactory(TokenizerFactory factory)
Construct a token-modifying tokenizer factory with
the specified base factory.
|
| Modifier and Type | Method and Description |
|---|---|
Tokenizer |
modify(Tokenizer tokenizer)
Return a modified version of the specified tokenizer that
modifies tokens and whitespaces as specified by the
corresponding string modifier methods.
|
String |
modifyToken(String token)
Return a modified form of the specified token, or
null to remove it. |
String |
modifyWhitespace(String whitespace)
Return the modified form of the specified whitespace.
|
String |
toString() |
baseTokenizerFactory, tokenizerpublic ModifyTokenTokenizerFactory(TokenizerFactory factory)
factory - Base tokenizer factory.public final Tokenizer modify(Tokenizer tokenizer)
modify in class ModifiedTokenizerFactorytokenizer - Tokenizer to modify.public String modifyToken(String token)
null to remove it.
The base implementation in this class simply returns the specified token.
token - Token to modify.null to remove it.public String modifyWhitespace(String whitespace)
The base implementation in this class simply returns the specified whitespace.
whitespace - Whitespace to modify.public String toString()
toString in class ModifiedTokenizerFactoryCopyright © 2016 Alias-i, Inc.. All rights reserved.