org.apache.hadoop.tools

Class DistCpOptions

java.lang.Object

org.apache.hadoop.tools.DistCpOptions

public class DistCpOptions
extends Object

The Options class encapsulates all DistCp options. These may be set from command-line (via the OptionsParser) or may be set manually.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DistCpOptions.FileAttribute

Field Summary

Fields

Modifier and Type	Field and Description
static int	maxNumListstatusThreads

Constructor Summary

Constructors

Constructor and Description
DistCpOptions(DistCpOptions that) Copy constructor.
DistCpOptions(List<org.apache.hadoop.fs.Path> sourcePaths, org.apache.hadoop.fs.Path targetPath) Constructor, to initialize source/target paths.
DistCpOptions(org.apache.hadoop.fs.Path sourceFileListing, org.apache.hadoop.fs.Path targetPath) Constructor, to initialize source/target paths.

Method Summary

Methods

Modifier and Type	Method and Description
void	appendToConf(org.apache.hadoop.conf.Configuration conf) Add options to configuration.
protected DistCpOptions	clone()
org.apache.hadoop.fs.Path	getAtomicWorkPath() Get work path for atomic commit.
int	getBlocksPerChunk()
int	getCopyBufferSize()
String	getCopyStrategy() Get the copy strategy to use.
String	getFiltersFile() File path that contains the list of patterns for paths to be filtered from the file copy.
String	getFromSnapshot()
org.apache.hadoop.fs.Path	getLogPath() Get output directory for writing distcp logs.
int	getMapBandwidth() Get the map bandwidth in MB
int	getMaxMaps() Get the max number of maps to use for this copy
int	getNumListstatusThreads() Get the number of threads to use for listStatus
org.apache.hadoop.fs.Path	getSourceFileListing() File path (hdfs:// or file://) that contains the list of actual files to copy
List<org.apache.hadoop.fs.Path>	getSourcePaths() Getter for sourcePaths.
String	getSslConfigurationFile() Get path where the ssl configuration file is present to use for hftps://
org.apache.hadoop.fs.Path	getTargetPath() Getter for the targetPath.
boolean	getTargetPathExists() Getter for the targetPathExists.
String	getToSnapshot()
void	preserve(DistCpOptions.FileAttribute fileAttribute) Add file attributes that need to be preserved.
Iterator<DistCpOptions.FileAttribute>	preserveAttributes() Returns an iterator with the list of file attributes to preserve
void	preserveRawXattrs() Indicate that raw.* xattrs should be preserved
void	setAppend(boolean append) Set if we want to append new data to target files.
void	setAtomicCommit(boolean atomicCommit) Set if data need to be committed automatically
void	setAtomicWorkPath(org.apache.hadoop.fs.Path atomicWorkPath) Set the work path for atomic commit
void	setBlocking(boolean blocking) Set if Disctp should run blocking or non-blocking
void	setBlocksPerChunk(int csize)
void	setCopyBufferSize(int newCopyBufferSize)
void	setCopyStrategy(String copyStrategy) Set the copy strategy to use.
void	setDeleteMissing(boolean deleteMissing) Set if files only present in target should be deleted
void	setFiltersFile(String filtersFilename) Set filtersFile.
void	setIgnoreFailures(boolean ignoreFailures) Set if failures during copy be ignored
void	setLogPath(org.apache.hadoop.fs.Path logPath) Set the log path where distcp output logs are stored Uses JobStagingDir/_logs by default
void	setMapBandwidth(int mapBandwidth) Set per map bandwidth
void	setMaxMaps(int maxMaps) Set the max number of maps to use for copy
void	setNumListstatusThreads(int numThreads) Set the number of threads to use for listStatus.
void	setOverwrite(boolean overwrite) Set if files should always be overwritten on target
void	setSkipCRC(boolean skipCRC) Set if checksum comparison should be skipped while determining if source and destination files are identical
void	setSourcePaths(List<org.apache.hadoop.fs.Path> sourcePaths) Setter for sourcePaths.
void	setSslConfigurationFile(String sslConfigurationFile) Set the SSL configuration file path to use with hftps:// (local path)
void	setSyncFolder(boolean syncFolder) Set if source and target folder contents be sync'ed up
boolean	setTargetPathExists(boolean targetPathExists) Set targetPathExists.
void	setUseDiff(String fromSS, String toSS)
void	setUseRdiff(String fromSS, String toSS)
void	setVerboseLog(boolean newVerboseLog)
boolean	shouldAppend()
boolean	shouldAtomicCommit() Should the data be committed atomically?
boolean	shouldBlock() Should DistCp be running in blocking mode
boolean	shouldDeleteMissing() Should target files missing in source should be deleted?
boolean	shouldIgnoreFailures() Should failures be logged and ignored during copy?
boolean	shouldOverwrite() Should files be overwritten always?
boolean	shouldPreserve(DistCpOptions.FileAttribute attribute) Checks if the input attribute should be preserved or not.
boolean	shouldPreserveRawXattrs() Return true if raw.* xattrs should be preserved.
boolean	shouldSkipCRC() Should CRC/checksum check be skipped while checking files are identical
boolean	shouldSyncFolder() Should the data be sync'ed between source and target paths?
boolean	shouldUseDiff()
boolean	shouldUseRdiff()
boolean	shouldUseSnapshotDiff()
boolean	shouldVerboseLog()
boolean	splitLargeFile()
String	toString() Utility to easily string-ify Options, for logging.
void	validate(DistCpOptionSwitch option, boolean value)

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

maxNumListstatusThreads

public static final int maxNumListstatusThreads

See Also:

Constant Field Values

Constructor Detail

DistCpOptions

public DistCpOptions(List<org.apache.hadoop.fs.Path> sourcePaths,
             org.apache.hadoop.fs.Path targetPath)

Constructor, to initialize source/target paths.

Parameters:

sourcePaths - List of source-paths (including wildcards) to be copied to target.

targetPath - Destination path for the dist-copy.

DistCpOptions

public DistCpOptions(org.apache.hadoop.fs.Path sourceFileListing,
             org.apache.hadoop.fs.Path targetPath)

Constructor, to initialize source/target paths.

Parameters:

sourceFileListing - File containing list of source paths

targetPath - Destination path for the dist-copy.

DistCpOptions

public DistCpOptions(DistCpOptions that)

Copy constructor.

Parameters:

that - DistCpOptions being copied from.

Method Detail

shouldAtomicCommit

public boolean shouldAtomicCommit()

Should the data be committed atomically?

Returns:

true if data should be committed automically. false otherwise

setAtomicCommit

public void setAtomicCommit(boolean atomicCommit)

Set if data need to be committed automatically

Parameters:

atomicCommit - - boolean switch

shouldSyncFolder

public boolean shouldSyncFolder()

Should the data be sync'ed between source and target paths?

Returns:

true if data should be sync'ed up. false otherwise

setSyncFolder

public void setSyncFolder(boolean syncFolder)

Set if source and target folder contents be sync'ed up

Parameters:

syncFolder - - boolean switch

shouldDeleteMissing

public boolean shouldDeleteMissing()

Should target files missing in source should be deleted?

Returns:

true if zoombie target files to be removed. false otherwise

setDeleteMissing

public void setDeleteMissing(boolean deleteMissing)

Set if files only present in target should be deleted

Parameters:

deleteMissing - - boolean switch

shouldIgnoreFailures

public boolean shouldIgnoreFailures()

Should failures be logged and ignored during copy?

Returns:

true if failures are to be logged and ignored. false otherwise

setIgnoreFailures

public void setIgnoreFailures(boolean ignoreFailures)

Set if failures during copy be ignored

Parameters:

ignoreFailures - - boolean switch

shouldBlock

public boolean shouldBlock()

Should DistCp be running in blocking mode

Returns:

true if should run in blocking, false otherwise

setBlocking

public void setBlocking(boolean blocking)

Set if Disctp should run blocking or non-blocking

Parameters:

blocking - - boolean switch

shouldOverwrite

public boolean shouldOverwrite()

Should files be overwritten always?

Returns:

true if files in target that may exist before distcp, should always be overwritten. false otherwise

setOverwrite

public void setOverwrite(boolean overwrite)

Set if files should always be overwritten on target

Parameters:

overwrite - - boolean switch

shouldAppend

public boolean shouldAppend()

Returns:

whether we can append new data to target files

setAppend

public void setAppend(boolean append)

Set if we want to append new data to target files. This is valid only with update option and CRC is not skipped.

shouldUseDiff

public boolean shouldUseDiff()

shouldUseRdiff

public boolean shouldUseRdiff()

shouldUseSnapshotDiff

public boolean shouldUseSnapshotDiff()

getFromSnapshot

public String getFromSnapshot()

getToSnapshot

public String getToSnapshot()

setUseDiff

public void setUseDiff(String fromSS,
              String toSS)

setUseRdiff

public void setUseRdiff(String fromSS,
               String toSS)

shouldSkipCRC

public boolean shouldSkipCRC()

Should CRC/checksum check be skipped while checking files are identical

Returns:

true if checksum check should be skipped while checking files are identical. false otherwise

setSkipCRC

public void setSkipCRC(boolean skipCRC)

Set if checksum comparison should be skipped while determining if source and destination files are identical

Parameters:

skipCRC - - boolean switch

getNumListstatusThreads

public int getNumListstatusThreads()

Get the number of threads to use for listStatus

Returns:

Number of threads to do listStatus

setNumListstatusThreads

public void setNumListstatusThreads(int numThreads)

Set the number of threads to use for listStatus. We allow max 40 threads. Setting numThreads to zero signify we should use the value from conf properties.

Parameters:

numThreads - - Number of threads

getMaxMaps

public int getMaxMaps()

Get the max number of maps to use for this copy

Returns:

Max number of maps

setMaxMaps

public void setMaxMaps(int maxMaps)

Set the max number of maps to use for copy

Parameters:

maxMaps - - Number of maps

getMapBandwidth

public int getMapBandwidth()

Get the map bandwidth in MB

Returns:

Bandwidth in MB

setMapBandwidth

public void setMapBandwidth(int mapBandwidth)

Set per map bandwidth

Parameters:

mapBandwidth - - per map bandwidth

getSslConfigurationFile

public String getSslConfigurationFile()

Get path where the ssl configuration file is present to use for hftps://

Returns:

Path on local file system

setSslConfigurationFile

public void setSslConfigurationFile(String sslConfigurationFile)

Set the SSL configuration file path to use with hftps:// (local path)

Parameters:

sslConfigurationFile - - Local ssl config file path

preserveAttributes

public Iterator<DistCpOptions.FileAttribute> preserveAttributes()

Returns an iterator with the list of file attributes to preserve

Returns:

iterator of file attributes to preserve

shouldPreserve

public boolean shouldPreserve(DistCpOptions.FileAttribute attribute)

Checks if the input attribute should be preserved or not.

Parameters:

attribute - - Attribute to check

Returns:

True if attribute should be preserved, false otherwise

preserve

public void preserve(DistCpOptions.FileAttribute fileAttribute)

Add file attributes that need to be preserved. This method may be called multiple times to add attributes.

Parameters:

fileAttribute - - Attribute to add, one at a time

shouldPreserveRawXattrs

public boolean shouldPreserveRawXattrs()

Return true if raw.* xattrs should be preserved.

Returns:

true if raw.* xattrs should be preserved.

preserveRawXattrs

public void preserveRawXattrs()

Indicate that raw.* xattrs should be preserved

getAtomicWorkPath

public org.apache.hadoop.fs.Path getAtomicWorkPath()

Get work path for atomic commit. If null, the work path would be parentOf(targetPath) + "/._WIP_" + nameOf(targetPath)

Returns:

Atomic work path on the target cluster. Null if not set

setAtomicWorkPath

public void setAtomicWorkPath(org.apache.hadoop.fs.Path atomicWorkPath)

Set the work path for atomic commit

Parameters:

atomicWorkPath - - Path on the target cluster

getLogPath

public org.apache.hadoop.fs.Path getLogPath()

Get output directory for writing distcp logs. Otherwise logs are temporarily written to JobStagingDir/_logs and deleted upon job completion

Returns:

Log output path on the cluster where distcp job is run

setLogPath

public void setLogPath(org.apache.hadoop.fs.Path logPath)

Set the log path where distcp output logs are stored Uses JobStagingDir/_logs by default

Parameters:

logPath - - Path where logs will be saved

getCopyStrategy

public String getCopyStrategy()

Get the copy strategy to use. Uses appropriate input format

Returns:

copy strategy to use

setCopyStrategy

public void setCopyStrategy(String copyStrategy)

Set the copy strategy to use. Should map to a strategy implementation in distp-default.xml

Parameters:

copyStrategy - - copy Strategy to use

getSourceFileListing

public org.apache.hadoop.fs.Path getSourceFileListing()

File path (hdfs:// or file://) that contains the list of actual files to copy

Returns:

- Source listing file path

getSourcePaths

public List<org.apache.hadoop.fs.Path> getSourcePaths()

Getter for sourcePaths.

Returns:

List of source-paths.

setSourcePaths

public void setSourcePaths(List<org.apache.hadoop.fs.Path> sourcePaths)

Setter for sourcePaths.

Parameters:

sourcePaths - The new list of source-paths.

getTargetPath

public org.apache.hadoop.fs.Path getTargetPath()

Getter for the targetPath.

Returns:

The target-path.

getTargetPathExists

public boolean getTargetPathExists()

Getter for the targetPathExists.

Returns:

The target-path.

setTargetPathExists

public boolean setTargetPathExists(boolean targetPathExists)

Set targetPathExists.

Parameters:

targetPathExists - Whether the target path of distcp exists.

getFiltersFile

public final String getFiltersFile()

File path that contains the list of patterns for paths to be filtered from the file copy.

Returns:

- Filter file path.

setFiltersFile

public final void setFiltersFile(String filtersFilename)

Set filtersFile.

Parameters:

filtersFilename - The path to a list of patterns to exclude from copy.

setBlocksPerChunk

public final void setBlocksPerChunk(int csize)

getBlocksPerChunk

public final int getBlocksPerChunk()

splitLargeFile

public final boolean splitLargeFile()

setCopyBufferSize

public final void setCopyBufferSize(int newCopyBufferSize)

getCopyBufferSize

public int getCopyBufferSize()

setVerboseLog

public void setVerboseLog(boolean newVerboseLog)

shouldVerboseLog

public boolean shouldVerboseLog()

validate

public void validate(DistCpOptionSwitch option,
            boolean value)

appendToConf

public void appendToConf(org.apache.hadoop.conf.Configuration conf)

Add options to configuration. These will be used in the Mapper/committer

Parameters:

conf - - Configuration object to which the options need to be added

toString

public String toString()

Utility to easily string-ify Options, for logging.

Overrides:

toString in class Object

Returns:

String representation of the Options.

clone

protected DistCpOptions clone()
                       throws CloneNotSupportedException

Overrides:

clone in class Object

Throws:

CloneNotSupportedException