Uses of Interface
de.jungblut.nlp.Tokenizer

Packages that use Tokenizer

Package	Description
de.jungblut.nlp
de.jungblut.nlp.mr

Uses of Tokenizer in de.jungblut.nlp

Classes in de.jungblut.nlp that implement Tokenizer

Modifier and Type	Class and Description
class	BigramTokenizer Advanced tokenizer that lowercases, adds start and end tags, deduplicates tokens and builds bigrams.
class	StandardTokenizer Just a basic tokenizer by certain attributes with normalization.

Uses of Tokenizer in de.jungblut.nlp.mr

Methods in de.jungblut.nlp.mr that return Tokenizer

Modifier and Type	Method and Description
static Tokenizer	WordCorpusFrequencyJob.getTokenizer(org.apache.hadoop.conf.Configuration conf) Gets a tokenizer, based on the configured class in "tokenizer.class".