org.apache.flink.hadoopcompatibility.mapred.record.datatypes

Class HadoopFileOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapred.OutputCommitter

org.apache.hadoop.mapred.FileOutputCommitter

org.apache.flink.hadoopcompatibility.mapred.record.datatypes.HadoopFileOutputCommitter

All Implemented Interfaces:

Serializable

public class HadoopFileOutputCommitter
extends org.apache.hadoop.mapred.FileOutputCommitter
implements Serializable

Hadoop 1.2.1 FileOutputCommitter takes JobContext as input parameter. However JobContext class is package private, and in Hadoop 2.2.0 it's public. This class takes JobConf as input instead of JobContext in order to setup and commit tasks.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileOutputCommitter

LOG, SUCCEEDED_FILE_NAME, TEMP_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
HadoopFileOutputCommitter()

Method Summary

Methods

Modifier and Type	Method and Description
void	cleanupJob(org.apache.hadoop.mapred.JobConf conf)
void	commitJob(org.apache.hadoop.mapred.JobConf conf)
void	commitTask(org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)
org.apache.hadoop.fs.Path	getTempTaskOutputPath(org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)
boolean	needsTaskCommit(org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)
void	setupJob(org.apache.hadoop.mapred.JobConf conf)

Methods inherited from class org.apache.hadoop.mapred.FileOutputCommitter

abortJob, abortTask, cleanupJob, commitJob, commitTask, getTaskAttemptPath, getWorkPath, isRecoverySupported, needsTaskCommit, recoverTask, setupJob, setupTask

Methods inherited from class org.apache.hadoop.mapred.OutputCommitter

abortJob, abortTask, cleanupJob, commitJob, commitTask, needsTaskCommit, recoverTask, setupJob, setupTask

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileOutputCommitter

public HadoopFileOutputCommitter()

Method Detail

setupJob

public void setupJob(org.apache.hadoop.mapred.JobConf conf)
              throws IOException

Throws:

IOException

commitTask

public void commitTask(org.apache.hadoop.mapred.JobConf conf,
              org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)
                throws IOException

Throws:

IOException

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapred.JobConf conf,
                      org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)
                        throws IOException

Throws:

IOException

getTempTaskOutputPath

public org.apache.hadoop.fs.Path getTempTaskOutputPath(org.apache.hadoop.mapred.JobConf conf,
                                              org.apache.hadoop.mapred.TaskAttemptID taskAttemptID)

cleanupJob

public void cleanupJob(org.apache.hadoop.mapred.JobConf conf)
                throws IOException

Throws:

IOException

commitJob

public void commitJob(org.apache.hadoop.mapred.JobConf conf)
               throws IOException

Throws:

IOException