de.jungblut.nlp

Class TokenizerUtils

java.lang.Object

de.jungblut.nlp.TokenizerUtils

public final class TokenizerUtils
extends Object

Nifty text utility for majorly tokenizing tasks.

Author:

thomas.jungblut

Field Summary

Fields

Modifier and Type	Field and Description
static String	END_TAG
static String	SEPARATORS
static String	START_TAG

Method Summary

Modifier and Type	Method and Description
static String[]	addStartAndEndTags(String[] unigram) Adds and to the beginning of the array and the end.
static String[]	buildNGrams(String[] tokens, int size) This tokenizer uses the given tokens and then concatenates the words based on size.
static String[]	buildNGramsRange(String[] tokens, int startSize, int endSize) Builds ngrams from a range of tokens, basically a concat of all the buildNGrams(String[], int) calls within the range.
static String	concat(String[] tokens, String delimiter) Concats the given tokens with the given delimiter.
static String[]	deduplicateTokens(String[] tokens) Deduplicates the given tokens, but maintains the order.
static String[]	internStrings(String[] strings) Interns the given strings inplace.
static String[]	internStrings(String[] strings, StringPool pool) Interns the given strings inplace with the given pool.
static String	normalizeString(String token) Normalizes the token: - lower cases - removes not alphanumeric characters (since I'm german I have included äüöß as well).
static String[]	normalizeTokens(String[] tokens, boolean removeEmpty) Normalizes the tokens: - lower cases - removes not alphanumeric characters (since I'm german I have included äüöß as well).
static String[]	nShinglesTokenize(String key, int size) N-shingles tokenizer.
static String[]	qGramTokenize(String key, int size) q-gram tokenizer, which is basically a proxy to nShinglesTokenize(String, int).
static String[]	removeEmpty(String[] arr) Removes empty tokens from given array.
static String[]	removeMatchingRegex(String regex, String replacement, String[] tokens, boolean removeEmpty) Applies given regex on tokens and may optionally delete when a token gets empty.
static String[]	whiteSpaceTokenize(String text) Tokenizes on normal whitespaces "\\s+" in java regex.
static String[]	whiteSpaceTokenizeNGrams(String text, int size) This tokenizer first splits on whitespaces and then concatenates the words based on size.
static String[]	wordTokenize(String text) Tokenizes on several indicators of a word, regex is [ \r\n\t.,;:'\"()?!\\-/|]
static String[]	wordTokenize(String text, boolean keepSeperators) Tokenizes like wordTokenize(String) does, but keeps the seperators as their own token if the argument is true.
static String[]	wordTokenize(String text, String regex) Tokenizes on several indicators of a word, regex to detect these must be given.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

END_TAG

public static final String END_TAG

See Also:

Constant Field Values

START_TAG

public static final String START_TAG

See Also:

Constant Field Values

SEPARATORS

public static final String SEPARATORS

See Also:

Constant Field Values

Method Detail

removeMatchingRegex

public static String[] removeMatchingRegex(String regex,
                                           String replacement,
                                           String[] tokens,
                                           boolean removeEmpty)

Applies given regex on tokens and may optionally delete when a token gets empty.

qGramTokenize

public static String[] qGramTokenize(String key,
                                     int size)

q-gram tokenizer, which is basically a proxy to nShinglesTokenize(String, int). These are nGrams based on characters. If you want to use normal word tokenizers, then use wordTokenize(String) for unigrams. To generate bigrams out of it you need to call buildNGrams(String[], int).

Parameters:

key -

size -

Returns:

nShinglesTokenize

public static String[] nShinglesTokenize(String key,
                                         int size)

N-shingles tokenizer. N-Shingles are nGrams based on characters. If you want to use normal word tokenizers, then use wordTokenize(String) for unigrams. To generate bigrams out of it you need to call buildNGrams(String[], int).

whiteSpaceTokenize

public static String[] whiteSpaceTokenize(String text)

Tokenizes on normal whitespaces "\\s+" in java regex.

deduplicateTokens

public static String[] deduplicateTokens(String[] tokens)

Deduplicates the given tokens, but maintains the order.

wordTokenize

public static String[] wordTokenize(String text)

Tokenizes on several indicators of a word, regex is [ \r\n\t.,;:'\"()?!\\-/|]

wordTokenize

public static String[] wordTokenize(String text,
                                    boolean keepSeperators)

Tokenizes like wordTokenize(String) does, but keeps the seperators as their own token if the argument is true.

wordTokenize

public static String[] wordTokenize(String text,
                                    String regex)

Tokenizes on several indicators of a word, regex to detect these must be given.

normalizeTokens

public static String[] normalizeTokens(String[] tokens,
                                       boolean removeEmpty)

Normalizes the tokens:
- lower cases
- removes not alphanumeric characters (since I'm german I have included äüöß as well).

normalizeString

public static String normalizeString(String token)

Normalizes the token:
- lower cases
- removes not alphanumeric characters (since I'm german I have included äüöß as well).

removeEmpty

public static String[] removeEmpty(String[] arr)

Removes empty tokens from given array. The empty slots will be filled with the follow-up tokens.

whiteSpaceTokenizeNGrams

public static String[] whiteSpaceTokenizeNGrams(String text,
                                                int size)

This tokenizer first splits on whitespaces and then concatenates the words based on size.

buildNGrams

public static String[] buildNGrams(String[] tokens,
                                   int size)

This tokenizer uses the given tokens and then concatenates the words based on size.

buildNGramsRange

public static String[] buildNGramsRange(String[] tokens,
                                        int startSize,
                                        int endSize)

Builds ngrams from a range of tokens, basically a concat of all the buildNGrams(String[], int) calls within the range. Both start and end are inclusive.

internStrings

public static String[] internStrings(String[] strings)

Interns the given strings inplace.

Parameters:

strings - the strings to intern.

Returns:

an interned string array.

internStrings

public static String[] internStrings(String[] strings,
                                     StringPool pool)

Interns the given strings inplace with the given pool.

Parameters:

strings - the strings to intern.

pool - the string pool to use.

Returns:

an interned string array.

addStartAndEndTags

public static String[] addStartAndEndTags(String[] unigram)

Adds and to the beginning of the array and the end.

concat

public static String concat(String[] tokens,
                            String delimiter)

Concats the given tokens with the given delimiter.