com.aliasi.tokenizer

Class SoundexTokenizerFactory

java.lang.Object

com.aliasi.tokenizer.ModifiedTokenizerFactory

com.aliasi.tokenizer.ModifyTokenTokenizerFactory

com.aliasi.tokenizer.SoundexTokenizerFactory

All Implemented Interfaces:

TokenizerFactory, Serializable

public class SoundexTokenizerFactory
extends ModifyTokenTokenizerFactory
implements Serializable

A SoundexTokenizerFactory modifies the output of a base tokenizer factory to produce tokens in soundex representation. Soundex replaces sequences of characters with a crude four-character approximation of their pronunciation plus initial letter.

Soundex Representations

The process for converting an input to its Soundex representation is fairly straighforward for inputs that are all ASCII letters. Soundex is case insensitive, but is only defined for strings of ASCII letters. Thus to begin, all characters that are not Latin1 letters are removed, and all Latin1 characters are stripped of their diacritics. The algorithm then proceeds according to its standard definition:

1. Normalize input by removing all characters that are not Latin1 letters, and converting all other characters to uppercase ASCII after first removing any diacritics.
2. If the input is empty, return "0000"
3. Set the first letter of the output to the first letter of the input.
4. While there are less than four letters of output do:
   1. If the next letter is a vowel, unset the last letter's code.
   2. If the next letter is A, E, I, O, U, H, W, Y, continue.
   3. If the next letter's code is equal to the previous letter's code, continue.
   4. Set the next letter of output to the current letter's code.
5. If there are fewer than four characters of output, pad the output with zeros (0)
6. Return the output string.

The table of individual character encodings is as follows:

Characters	Code
B, F, P, V	1
C, G, J, K, Q, S, X, Z	2
D, T	3
L	4
M, N	5
R	6

Here are some examples of translations from the unit tests, drawn from the sources cited below.

Tokens	Soundex Encoding	Notes
Gutierrez	G362
Pfister	P236
Jackson	J250
Tymczak	T522
Ashcraft	A261
Robert, Rupert	R163
Euler, Ellery	E460
Gauss, Ghosh	G200
Hilbert, Heilbronn	H416
Knuth, Kant	K530
Lloyd, Liddy	L300
Lukasiewicz, Lissajous	L222
Wachs, Waugh	W200

As a tokenizer filter, the SoundexFilterTokenizer simply replaces each token with its Soundex equivalent. Note that this may produce very many 0000 outputs if it is fed standard text with punctuation, numbers, etc.

Note: In order to produce a deterministic tokenizer filter, names with prefixes are coded with the prefix. Recall that Soundex considers the following set of words prefixes, and suggests providing both the Soundex computed with the prefix and the Soundex encoding computed without the prefix:

 Van, Con, De, Di, La, Le

These are not accorded any special treatment by this implementation.

Thread Safety

An English stop-listed tokenizer factory is thread safe if its base tokenizer factory is thread safe.

Serialization

An EnglishStopTokenizerFactory is serializable if its base tokenizer factory is serializable.

References and Historical Notes

Soundex was invented and patented by Robert C. Russell in 1918. The original version involved eight categories, including one for vowels, without the initial character being treated specially as to coding. The first vowel was retained in the original Soundex. Furthermore, some positional information was added, such as the deletion of final s and z.

The version in this class is the one described by Donald Knuth in The Art of Computer Programming and the one described by the United States National Archives and Records Administration version, which has been used for the United States Census.

• Knuth, D. 1973. The Art of Computer Programming Volum 3: Sorting and Searching. Addison-Wesley. 2nd Edition Pages 394-395.
• Wikipedia. Soundex.
• United States National Archives and Records Administration. Using the Census Soundex. General Information Leaflet 55.
• Robert C. Russell. 1918. United States Patent 1,261,167.
• Robert C. Russell. 1922. United States Patent 1,435,663.

Since:

Lingpipe3.8

Version:

4.0.1

Author:

Bob Carpenter

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SoundexTokenizerFactory(TokenizerFactory factory) Construct a Soundex-based tokenizer factory that converts tokens produced by the specified base factory into their soundex representations.

Method Summary

Methods

Modifier and Type	Method and Description
String	modifyToken(String token) Returns the Soundex encoding of the specified token.
static String	soundexEncoding(String token) Returns the Soundex encoding of the specified token.
String	toString()

Methods inherited from class com.aliasi.tokenizer.ModifyTokenTokenizerFactory

modify, modifyWhitespace

Methods inherited from class com.aliasi.tokenizer.ModifiedTokenizerFactory

baseTokenizerFactory, tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SoundexTokenizerFactory

public SoundexTokenizerFactory(TokenizerFactory factory)

Construct a Soundex-based tokenizer factory that converts tokens produced by the specified base factory into their soundex representations.

Parameters:

factory - Base tokenizer factory.

Method Detail

modifyToken

public String modifyToken(String token)

Returns the Soundex encoding of the specified token.

See the class documentation above for more information on the encoding.

Overrides:

modifyToken in class ModifyTokenTokenizerFactory

Parameters:

token - Input token.

Returns:

The soundex encoding of the input token.

toString

public String toString()

Overrides:

toString in class ModifyTokenTokenizerFactory

soundexEncoding

public static String soundexEncoding(String token)

Returns the Soundex encoding of the specified token.

Parameters:

token - Token to be encoded.

Returns:

The Soundex encoding of the specified token.