de.jungblut.nlp.mr

Class TfIdfCalculatorJob

java.lang.Object

de.jungblut.nlp.mr.TfIdfCalculatorJob

public class TfIdfCalculatorJob
extends Object

Job that will calculate tf-idf based on the output of the WordCorpusFrequencyJob.

Author:

thomas.jungblut

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TfIdfCalculatorJob.DocumentVectorizerReducer Calculate the sparse vector with TF-IDF.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NUMBER_OF_DOCUMENTS_KEY
static String	NUMBER_OF_TOKENS_KEY
static String	SPAM_DOCUMENT_PERCENTAGE_KEY
static String	WORD_COUNT_OUTPUT_KEY

Constructor Summary

Constructors

Constructor and Description
TfIdfCalculatorJob()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.mapreduce.Job	createJob(String in, String out, org.apache.hadoop.conf.Configuration conf, long numberOfDocuments, long numberOfTokens) Creates a tf-idf job.
static void	main(String[] args) Calculates TF-IDF vectors from text input in the following format:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUMBER_OF_DOCUMENTS_KEY

public static final String NUMBER_OF_DOCUMENTS_KEY

See Also:

Constant Field Values

NUMBER_OF_TOKENS_KEY

public static final String NUMBER_OF_TOKENS_KEY

See Also:

Constant Field Values

SPAM_DOCUMENT_PERCENTAGE_KEY

public static final String SPAM_DOCUMENT_PERCENTAGE_KEY

See Also:

Constant Field Values

WORD_COUNT_OUTPUT_KEY

public static final String WORD_COUNT_OUTPUT_KEY

See Also:

Constant Field Values

Constructor Detail

TfIdfCalculatorJob

public TfIdfCalculatorJob()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Calculates TF-IDF vectors from text input in the following format:

 documentid \t corpus
 

It will run two jobs, a first job determines the document frequency of a token, as well as its index in the resulting vector. The output is a SequenceFile with Text as key and VectorWritable as value.

Throws:

Exception

createJob

public static org.apache.hadoop.mapreduce.Job createJob(String in,
                                                        String out,
                                                        org.apache.hadoop.conf.Configuration conf,
                                                        long numberOfDocuments,
                                                        long numberOfTokens)
                                                 throws IOException

Creates a tf-idf job.

Parameters:

in - the input path, the output of the WordCorpusFrequencyJob.

out - the output directory.

conf - the configuration.

numberOfDocuments - the number of documents in the corpus per token. (map input counter value of WordCorpusFrequencyJob.)

numberOfTokens - the number of tokens in the corpus. (reduce input group counter value of WordCorpusFrequencyJob.)

Returns:

a job with the configured propertys like name, key/value classes and input format as text.

Throws:

IOException