de.jungblut.nlp

Class BigramTokenizer

java.lang.Object

de.jungblut.nlp.BigramTokenizer

All Implemented Interfaces:

Tokenizer

public final class BigramTokenizer
extends Object
implements Tokenizer

Advanced tokenizer that lowercases, adds start and end tags, deduplicates tokens and builds bigrams.

Author:

thomas.jungblut

Constructor Summary

Constructors

Constructor and Description
BigramTokenizer()

Method Summary

Modifier and Type	Method and Description
String[]	tokenize(String toTokenize) Tokenizes the given String to a array of Strings.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigramTokenizer

public BigramTokenizer()

Method Detail

tokenize

public String[] tokenize(String toTokenize)

Description copied from interface: Tokenizer

Tokenizes the given String to a array of Strings.

Specified by:

tokenize in interface Tokenizer

Parameters:

toTokenize - the string to tokenize.

Returns:

the array of tokenized tokens.