de.jungblut.nlp.mr

Class WordCountJob

java.lang.Object

de.jungblut.nlp.mr.WordCountJob

public class WordCountJob
extends Object

MapReduce job that calculates the token frequency by an improved word count.

Author:

thomas.jungblut

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WordCountJob.WordFrequencyMapper Group the tokens in memory for each chunk, write it in the cleanup step.
static class	WordCountJob.WordFrequencyReducer Group the tokens by reducing the mappers output and summing the sums for each token.

Field Summary

Fields

Modifier and Type	Field and Description
static String	MIN_WORD_COUNT_KEY

Constructor Summary

Constructors

Constructor and Description
WordCountJob()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.mapreduce.Job	createJob(String in, String out, org.apache.hadoop.conf.Configuration conf) Creates a token frequency job.
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MIN_WORD_COUNT_KEY

public static final String MIN_WORD_COUNT_KEY

See Also:

Constant Field Values

Constructor Detail

WordCountJob

public WordCountJob()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

createJob

public static org.apache.hadoop.mapreduce.Job createJob(String in,
                                                        String out,
                                                        org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Creates a token frequency job.

Parameters:

in - the input path, may comma separate multiple paths.

out - the output directory.

conf - the configuration.

Returns:

a job with the configured propertys like name, key/value classes and in/output format as text.

Throws:

IOException