org.apache.hadoop.tools.mapred

Class CopyCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

org.apache.hadoop.tools.mapred.CopyCommitter

public class CopyCommitter
extends org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

The CopyCommitter class is DistCp's OutputCommitter implementation. It is responsible for handling the completion/cleanup of the DistCp run. Specifically, it does the following: 1. Cleanup of the meta-folder (where DistCp maintains its file-list, etc.) 2. Preservation of user/group/replication-factor on any directories that have been copied. (Files are taken care of in their map-tasks.) 3. Atomic-move of data from the temporary work-folder to the final path (if atomic-commit was opted for). 4. Deletion of files from the target that are missing at source (if opted for). 5. Cleanup of any partially copied files, from previous, failed attempts.

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT, FILEOUTPUTCOMMITTER_CLEANUP_FAILURES_IGNORED, FILEOUTPUTCOMMITTER_CLEANUP_FAILURES_IGNORED_DEFAULT, FILEOUTPUTCOMMITTER_CLEANUP_SKIPPED, FILEOUTPUTCOMMITTER_CLEANUP_SKIPPED_DEFAULT, FILEOUTPUTCOMMITTER_FAILURE_ATTEMPTS, FILEOUTPUTCOMMITTER_FAILURE_ATTEMPTS_DEFAULT, FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED, FILEOUTPUTCOMMITTER_TASK_CLEANUP_ENABLED_DEFAULT, PENDING_DIR_NAME, SUCCEEDED_FILE_NAME, SUCCESSFUL_JOB_OUTPUT_DIR_MARKER, TEMP_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
CopyCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a output committer

Method Summary

Methods

Modifier and Type	Method and Description
void	abortJob(org.apache.hadoop.mapreduce.JobContext jobContext, org.apache.hadoop.mapreduce.JobStatus.State state)
void	commitJob(org.apache.hadoop.mapreduce.JobContext jobContext)

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

abortTask, abortTask, cleanupJob, commitJobInternal, commitTask, commitTask, getCommittedTaskPath, getCommittedTaskPath, getCommittedTaskPath, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getTaskAttemptPath, getTaskAttemptPath, getWorkPath, isCommitJobRepeatable, isRecoverySupported, needsTaskCommit, needsTaskCommit, recoverTask, setupJob, setupTask

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CopyCommitter

public CopyCommitter(org.apache.hadoop.fs.Path outputPath,
             org.apache.hadoop.mapreduce.TaskAttemptContext context)
              throws IOException

Create a output committer

Parameters:

outputPath - the job's output path

context - the task's context

Throws:

IOException - - Exception if any

Method Detail

commitJob

public void commitJob(org.apache.hadoop.mapreduce.JobContext jobContext)
               throws IOException

Overrides:

commitJob in class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

Throws:

IOException

abortJob

public void abortJob(org.apache.hadoop.mapreduce.JobContext jobContext,
            org.apache.hadoop.mapreduce.JobStatus.State state)
              throws IOException

Overrides:

abortJob in class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

Throws:

IOException