Package org.apache.nifi.processors.hadoop

Class GetHDFS

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

org.apache.nifi.processors.hadoop.GetHDFS

All Implemented Interfaces:

org.apache.nifi.components.ClassloaderIsolationKeyProvider, org.apache.nifi.components.ConfigurableComponent, org.apache.nifi.processor.Processor

Direct Known Subclasses:

GetHDFSSequenceFile

@TriggerWhenEmpty
@InputRequirement(INPUT_FORBIDDEN)
@Tags({"hadoop","HCFS","HDFS","get","fetch","ingest","source","filesystem"})
@CapabilityDescription("Fetch files from Hadoop Distributed File System (HDFS) into FlowFiles. This Processor will delete the file from HDFS after fetching it.")
@WritesAttribute(attribute="filename",description="The name of the file that was read from HDFS.") @WritesAttribute(attribute="path",description="The path is set to the relative path of the file\'s directory on HDFS. For example, if the Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to \"./\". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to \"abc/1/2/3\".")
@SeeAlso({PutHDFS.class,ListHDFS.class})
@Restriction(requiredPermission=READ_DISTRIBUTED_FILESYSTEM,explanation="Provides operator the ability to retrieve any file that NiFi has access to in HDFS or the local filesystem.") @Restriction(requiredPermission=WRITE_DISTRIBUTED_FILESYSTEM,explanation="Provides operator the ability to delete any file that NiFi has access to in HDFS or the local filesystem.")
public class GetHDFS
extends AbstractHadoopProcessor

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	GetHDFS.ProcessorConfiguration	Holder for a snapshot in time of some processor properties that are passed around.

Nested classes/interfaces inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

AbstractHadoopProcessor.ValidationResources

Field Summary

Fields

Modifier and Type	Field	Description
static final org.apache.nifi.components.PropertyDescriptor	BATCH_SIZE
static final org.apache.nifi.components.PropertyDescriptor	BUFFER_SIZE
static final int	BUFFER_SIZE_DEFAULT
static final String	BUFFER_SIZE_KEY
static final org.apache.nifi.components.PropertyDescriptor	FILE_FILTER_REGEX
private final BlockingQueue<org.apache.hadoop.fs.Path>	filePathQueue
static final org.apache.nifi.components.PropertyDescriptor	FILTER_MATCH_NAME_ONLY
static final org.apache.nifi.components.PropertyDescriptor	IGNORE_DOTTED_FILES
static final org.apache.nifi.components.PropertyDescriptor	KEEP_SOURCE_FILE
private final AtomicLong	lastPollTime
private final Lock	listingLock
private final AtomicLong	logEmptyListing
static final org.apache.nifi.components.PropertyDescriptor	MAX_AGE
static final int	MAX_WORKING_QUEUE_SIZE
static final org.apache.nifi.components.PropertyDescriptor	MIN_AGE
static final org.apache.nifi.components.PropertyDescriptor	POLLING_INTERVAL
private final BlockingQueue<org.apache.hadoop.fs.Path>	processing
protected GetHDFS.ProcessorConfiguration	processorConfig
private static final List<org.apache.nifi.components.PropertyDescriptor>	PROPERTY_DESCRIPTORS
private final Lock	queueLock
static final org.apache.nifi.components.PropertyDescriptor	RECURSE_SUBDIRS
static final org.apache.nifi.processor.Relationship	REL_SUCCESS
private static final Set<org.apache.nifi.processor.Relationship>	RELATIONSHIPS

Fields inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

ABSOLUTE_HDFS_PATH_ATTRIBUTE, ADDITIONAL_CLASSPATH_RESOURCES, COMPRESSION_CODEC, DIRECTORY, HADOOP_CONFIGURATION_RESOURCES, HADOOP_FILE_URL_ATTRIBUTE, hdfsResources, KERBEROS_USER_SERVICE, TARGET_HDFS_DIR_CREATED_ATTRIBUTE

Constructor Summary

Constructors

Constructor	Description
GetHDFS()

Method Summary

Modifier and Type	Method	Description
protected Collection<org.apache.nifi.components.ValidationResult>	customValidate(org.apache.nifi.components.ValidationContext context)
Set<org.apache.nifi.processor.Relationship>	getRelationships()
protected List<org.apache.nifi.components.PropertyDescriptor>	getSupportedPropertyDescriptors()
void	onScheduled(org.apache.nifi.processor.ProcessContext context)
void	onTrigger(org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session)
protected Set<org.apache.hadoop.fs.Path>	performListing(org.apache.nifi.processor.ProcessContext context)	Do a listing of HDFS if the POLLING_INTERVAL has lapsed.
protected void	processBatchOfFiles(List<org.apache.hadoop.fs.Path> files, org.apache.nifi.processor.ProcessContext context, org.apache.nifi.processor.ProcessSession session)
protected Set<org.apache.hadoop.fs.Path>	selectFiles(org.apache.hadoop.fs.FileSystem hdfs, org.apache.hadoop.fs.Path dir, Set<org.apache.hadoop.fs.Path> filesVisited)	Poll HDFS for files to process that match the configured file filters.

Methods inherited from class org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

abstractOnScheduled, abstractOnStopped, checkHdfsUriForTimeout, findCause, getClassloaderIsolationKey, getCommonPropertyDescriptors, getCompressionCodec, getConfigLocations, getConfiguration, getFileSystem, getFileSystem, getFileSystemAsUser, getHadoopConfigurationForValidation, getNormalizedPath, getNormalizedPath, getNormalizedPath, getPathDifference, getUserGroupInformation, handleAuthErrors, init, isFileSystemAccessDenied, isLocalFileSystemAccessDenied, migrateProperties, preProcessConfiguration, resetHDFSResources, validateFileSystem

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Methods inherited from interface org.apache.nifi.processor.Processor

isStateful, migrateRelationships

Field Details

BUFFER_SIZE_KEY

public static final String BUFFER_SIZE_KEY

See Also:

• Constant Field Values

BUFFER_SIZE_DEFAULT

public static final int BUFFER_SIZE_DEFAULT

See Also:

• Constant Field Values

MAX_WORKING_QUEUE_SIZE

public static final int MAX_WORKING_QUEUE_SIZE

See Also:

• Constant Field Values

REL_SUCCESS

public static final org.apache.nifi.processor.Relationship REL_SUCCESS

RECURSE_SUBDIRS

public static final org.apache.nifi.components.PropertyDescriptor RECURSE_SUBDIRS

KEEP_SOURCE_FILE

public static final org.apache.nifi.components.PropertyDescriptor KEEP_SOURCE_FILE

FILE_FILTER_REGEX

public static final org.apache.nifi.components.PropertyDescriptor FILE_FILTER_REGEX

FILTER_MATCH_NAME_ONLY

public static final org.apache.nifi.components.PropertyDescriptor FILTER_MATCH_NAME_ONLY

IGNORE_DOTTED_FILES

public static final org.apache.nifi.components.PropertyDescriptor IGNORE_DOTTED_FILES

MIN_AGE

public static final org.apache.nifi.components.PropertyDescriptor MIN_AGE

MAX_AGE

public static final org.apache.nifi.components.PropertyDescriptor MAX_AGE

BATCH_SIZE

public static final org.apache.nifi.components.PropertyDescriptor BATCH_SIZE

POLLING_INTERVAL

public static final org.apache.nifi.components.PropertyDescriptor POLLING_INTERVAL

BUFFER_SIZE

public static final org.apache.nifi.components.PropertyDescriptor BUFFER_SIZE

RELATIONSHIPS

private static final Set<org.apache.nifi.processor.Relationship> RELATIONSHIPS

PROPERTY_DESCRIPTORS

private static final List<org.apache.nifi.components.PropertyDescriptor> PROPERTY_DESCRIPTORS

processorConfig

protected GetHDFS.ProcessorConfiguration processorConfig

logEmptyListing

private final AtomicLong logEmptyListing

lastPollTime

private final AtomicLong lastPollTime

listingLock

private final Lock listingLock

queueLock

private final Lock queueLock

filePathQueue

private final BlockingQueue<org.apache.hadoop.fs.Path> filePathQueue

processing

private final BlockingQueue<org.apache.hadoop.fs.Path> processing

Constructor Details

GetHDFS

public GetHDFS()

Method Details

getRelationships

public Set<org.apache.nifi.processor.Relationship> getRelationships()

Specified by:

getRelationships in interface org.apache.nifi.processor.Processor

Overrides:

getRelationships in class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getSupportedPropertyDescriptors

protected List<org.apache.nifi.components.PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractHadoopProcessor

customValidate

protected Collection<org.apache.nifi.components.ValidationResult> customValidate(org.apache.nifi.components.ValidationContext context)

Overrides:

customValidate in class AbstractHadoopProcessor

onScheduled

@OnScheduled
public void onScheduled(org.apache.nifi.processor.ProcessContext context)
                 throws IOException

Throws:

IOException

onTrigger

public void onTrigger(org.apache.nifi.processor.ProcessContext context,
 org.apache.nifi.processor.ProcessSession session)
               throws org.apache.nifi.processor.exception.ProcessException

Specified by:

onTrigger in class org.apache.nifi.processor.AbstractProcessor

Throws:

org.apache.nifi.processor.exception.ProcessException

processBatchOfFiles

protected void processBatchOfFiles(List<org.apache.hadoop.fs.Path> files,
 org.apache.nifi.processor.ProcessContext context,
 org.apache.nifi.processor.ProcessSession session)

performListing

protected Set<org.apache.hadoop.fs.Path> performListing(org.apache.nifi.processor.ProcessContext context)
                                                 throws IOException,
InterruptedException

Do a listing of HDFS if the POLLING_INTERVAL has lapsed. Will return null if POLLING_INTERVAL has not lapsed. Will return an empty set if no files were found on HDFS that matched the configured filters.

Parameters:

context - context

Returns:

null if POLLING_INTERVAL has not lapsed. Will return an empty set if no files were found on HDFS that matched the configured filters

Throws:

IOException - ex

InterruptedException

selectFiles

protected Set<org.apache.hadoop.fs.Path> selectFiles(org.apache.hadoop.fs.FileSystem hdfs,
 org.apache.hadoop.fs.Path dir,
 Set<org.apache.hadoop.fs.Path> filesVisited)
                                              throws IOException,
InterruptedException

Poll HDFS for files to process that match the configured file filters.

Parameters:

hdfs - hdfs

dir - dir

filesVisited - filesVisited

Returns:

files to process

Throws:

IOException - ex

InterruptedException