de.jungblut.nlp.mr

Class WordCorpusFrequencyJob

java.lang.Object

de.jungblut.nlp.mr.WordCorpusFrequencyJob

public class WordCorpusFrequencyJob
extends Object

MapReduce job that calculates the word frequency over all documents by inverting document->words and writing the sum of the assigned documents per word and its document. Assuming that the document is small enough to fit in memory and a single line (this is not a good solution if you want to vectorize huge text books).

Author:

thomas.jungblut

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WordCorpusFrequencyJob.DocumentSumReducer Sums up all the documents per token index by docID.
static class	WordCorpusFrequencyJob.TokenMapper Write a token with its document id.
static class	WordCorpusFrequencyJob.WordCorpusCounter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DICT_OUT_PATH_KEY
static String	MIN_WORD_COUNT_KEY
static String	TOKENIZER_CLASS_KEY

Constructor Summary

Constructors

Constructor and Description
WordCorpusFrequencyJob()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.mapreduce.Job	createJob(String in, String dictOut, String out, org.apache.hadoop.conf.Configuration conf) Creates a token frequency job.
static long	getNumberOfDocuments(org.apache.hadoop.mapreduce.Job finishedJob) Gets the counter of the input lines read, in this case it should be the number of documents.
static long	getNumberOfTokens(org.apache.hadoop.mapreduce.Job finishedJob) Gets the counter of the reduce output values.
static Tokenizer	getTokenizer(org.apache.hadoop.conf.Configuration conf) Gets a tokenizer, based on the configured class in "tokenizer.class".
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DICT_OUT_PATH_KEY

public static final String DICT_OUT_PATH_KEY

See Also:

Constant Field Values

MIN_WORD_COUNT_KEY

public static final String MIN_WORD_COUNT_KEY

See Also:

Constant Field Values

TOKENIZER_CLASS_KEY

public static final String TOKENIZER_CLASS_KEY

See Also:

Constant Field Values

Constructor Detail

WordCorpusFrequencyJob

public WordCorpusFrequencyJob()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

getTokenizer

public static Tokenizer getTokenizer(org.apache.hadoop.conf.Configuration conf)

Gets a tokenizer, based on the configured class in "tokenizer.class".

getNumberOfDocuments

public static long getNumberOfDocuments(org.apache.hadoop.mapreduce.Job finishedJob)
                                 throws IOException

Gets the counter of the input lines read, in this case it should be the number of documents.

Parameters:

finishedJob - the job that has successfully finished.

Returns:

the number of map input records / number of documents.

Throws:

IOException

getNumberOfTokens

public static long getNumberOfTokens(org.apache.hadoop.mapreduce.Job finishedJob)
                              throws IOException

Gets the counter of the reduce output values. This determines the dictionary sizes.

Parameters:

finishedJob - the job that has successfully finished.

Returns:

the number of map input records / number of documents.

Throws:

IOException

createJob

public static org.apache.hadoop.mapreduce.Job createJob(String in,
                                                        String dictOut,
                                                        String out,
                                                        org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Creates a token frequency job.

Parameters:

in - the input path, may comma separate multiple paths.

dictOut - the output path of the dictionary.

out - the output directory.

conf - the configuration.

Returns:

a job with the configured propertys like name, key/value classes and input format as text.

Throws:

IOException