com.aliasi.tokenizer

Class ModifyTokenTokenizerFactory

java.lang.Object

com.aliasi.tokenizer.ModifiedTokenizerFactory

com.aliasi.tokenizer.ModifyTokenTokenizerFactory

All Implemented Interfaces:

TokenizerFactory, Serializable

Direct Known Subclasses:

LowerCaseTokenizerFactory, PorterStemmerTokenizerFactory, RegExFilteredTokenizerFactory, SoundexTokenizerFactory, StopTokenizerFactory, TokenLengthTokenizerFactory, WhitespaceNormTokenizerFactory

public abstract class ModifyTokenTokenizerFactory
extends ModifiedTokenizerFactory
implements Serializable

The abstract base class ModifyTokenTokenizerFactory adapts token and whitespace modifiers to modify tokenizer factories.

The method modifyToken(String) may be used to modify or remove tokens from tokenizer outputs. The method modifyWhitespace(String) may be used to modify the whitespace returned by a tokenizer. Both methods are given pass-through implementations in this class.

Serialization

Like its parent class, this class implements Serializable. There are no serialization methods defined, so the default serialization is used. There is a single reference to the base tokenizer factory in the parent class, so a subclass will be serializable if all of its member objects are serializable and the base tokenizer is serializable.

It is good practice for each subclass to take completecontrol over serialization using a serialization proxy implemented on top of the AbstractExternalizable base class.

Thread Safety

This tokenizer factory is thread safe if the modify token and modify whitespace implementations are thread safe. The implementations provided here are thread safe.

Since:

Lingpipe3.8

Version:

4.0.1

Author:

Bob Carpenter

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ModifyTokenTokenizerFactory(TokenizerFactory factory) Construct a token-modifying tokenizer factory with the specified base factory.

Method Summary

Methods

Modifier and Type	Method and Description
Tokenizer	modify(Tokenizer tokenizer) Return a modified version of the specified tokenizer that modifies tokens and whitespaces as specified by the corresponding string modifier methods.
String	modifyToken(String token) Return a modified form of the specified token, or null to remove it.
String	modifyWhitespace(String whitespace) Return the modified form of the specified whitespace.
String	toString()

Methods inherited from class com.aliasi.tokenizer.ModifiedTokenizerFactory

baseTokenizerFactory, tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ModifyTokenTokenizerFactory

public ModifyTokenTokenizerFactory(TokenizerFactory factory)

Construct a token-modifying tokenizer factory with the specified base factory.

Parameters:

factory - Base tokenizer factory.

Method Detail

modify

public final Tokenizer modify(Tokenizer tokenizer)

Return a modified version of the specified tokenizer that modifies tokens and whitespaces as specified by the corresponding string modifier methods.

Specified by:

modify in class ModifiedTokenizerFactory

Parameters:

tokenizer - Tokenizer to modify.

Returns:

The modified tokenizer.

modifyToken

public String modifyToken(String token)

Return a modified form of the specified token, or null to remove it.

The base implementation in this class simply returns the specified token.

Parameters:

token - Token to modify.

Returns:

Modified token or null to remove it.

modifyWhitespace

public String modifyWhitespace(String whitespace)

Return the modified form of the specified whitespace.

The base implementation in this class simply returns the specified whitespace.

Parameters:

whitespace - Whitespace to modify.

Returns:

The modified whitespace.

toString

public String toString()

Overrides:

toString in class ModifiedTokenizerFactory