de.jungblut.nlp

Interface Tokenizer

All Known Implementing Classes:

BigramTokenizer, StandardTokenizer

public interface Tokenizer

Standard tokenizer interface. Makes multiple strings out of one.

Author:

thomas.jungblut

Method Summary

Modifier and Type	Method and Description
String[]	tokenize(String toTokenize) Tokenizes the given String to a array of Strings.

Method Detail

tokenize

String[] tokenize(String toTokenize)

Tokenizes the given String to a array of Strings.

Parameters:

toTokenize - the string to tokenize.

Returns:

the array of tokenized tokens.