org.apache.nifi.processors.hadoop

Class AbstractHadoopProcessor

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.hadoop.AbstractHadoopProcessor

All Implemented Interfaces:

ClassloaderIsolationKeyProvider, ConfigurableComponent, Processor

@RequiresInstanceClassLoading(cloneAncestorResources=true)
public abstract class AbstractHadoopProcessor
extends AbstractProcessor
implements ClassloaderIsolationKeyProvider

This is a base class that is helpful when building processors interacting with HDFS.

As of Apache NiFi 1.5.0, the Relogin Period property is no longer used in the configuration of a Hadoop processor. Due to changes made to SecurityUtil.loginKerberos(Configuration, String, String), which is used by this class to authenticate a principal with Kerberos, Hadoop components no longer attempt relogins explicitly. For more information, please read the documentation for SecurityUtil.loginKerberos(Configuration, String, String).

See Also:

SecurityUtil.loginKerberos(Configuration, String, String)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AbstractHadoopProcessor.ValidationResources

Field Summary

Fields

Modifier and Type	Field and Description
static String	ABSOLUTE_HDFS_PATH_ATTRIBUTE
static PropertyDescriptor	ADDITIONAL_CLASSPATH_RESOURCES
private static String	ALLOW_EXPLICIT_KEYTAB
static PropertyDescriptor	COMPRESSION_CODEC
private static String	DENY_LFS_ACCESS
private static String	DENY_LFS_EXPLANATION
static PropertyDescriptor	DIRECTORY
private static HdfsResources	EMPTY_HDFS_RESOURCES
static PropertyDescriptor	HADOOP_CONFIGURATION_RESOURCES
private AtomicReference<HdfsResources>	hdfsResources
static PropertyDescriptor	KERBEROS_CREDENTIALS_SERVICE
static PropertyDescriptor	KERBEROS_RELOGIN_PERIOD
(package private) static PropertyDescriptor	KERBEROS_USER_SERVICE
private File	kerberosConfigFile
protected KerberosProperties	kerberosProperties
private static Pattern	LOCAL_FILE_SYSTEM_URI
private static String	NORMALIZE_ERROR_WITH_PROPERTY
private static String	NORMALIZE_ERROR_WITHOUT_PROPERTY
protected List<PropertyDescriptor>	properties
private static Object	RESOURCES_LOCK
protected static String	TARGET_HDFS_DIR_CREATED_ATTRIBUTE
private AtomicReference<AbstractHadoopProcessor.ValidationResources>	validationResourceHolder

Constructor Summary

Constructors

Constructor and Description
AbstractHadoopProcessor()

Method Summary

Modifier and Type	Method and Description
void	abstractOnScheduled(ProcessContext context) If your subclass also has an @OnScheduled annotated method and you need hdfsResources in that method, then be sure to call super.abstractOnScheduled(context)
void	abstractOnStopped()
protected void	checkHdfsUriForTimeout(org.apache.hadoop.conf.Configuration config)
protected Collection<ValidationResult>	customValidate(ValidationContext validationContext)
String	getClassloaderIsolationKey(PropertyContext context)
protected org.apache.hadoop.io.compress.CompressionCodec	getCompressionCodec(ProcessContext context, org.apache.hadoop.conf.Configuration configuration) Returns the configured CompressionCodec, or null if none is configured.
protected List<String>	getConfigLocations(PropertyContext context)
protected org.apache.hadoop.conf.Configuration	getConfiguration()
private static org.apache.hadoop.conf.Configuration	getConfigurationFromResources(org.apache.hadoop.conf.Configuration config, List<String> locations)
protected org.apache.hadoop.fs.FileSystem	getFileSystem()
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration config) This exists in order to allow unit tests to override it so that they don't take several minutes waiting for UDP packets to be received
protected org.apache.hadoop.fs.FileSystem	getFileSystemAsUser(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.security.UserGroupInformation ugi)
protected org.apache.hadoop.conf.Configuration	getHadoopConfigurationForValidation(List<String> locations)
protected KerberosProperties	getKerberosProperties(File kerberosConfigFile)
private KerberosUser	getKerberosUser(ProcessContext context)
protected org.apache.hadoop.fs.Path	getNormalizedPath(ProcessContext context, PropertyDescriptor property)
protected org.apache.hadoop.fs.Path	getNormalizedPath(ProcessContext context, PropertyDescriptor property, FlowFile flowFile)
protected org.apache.hadoop.fs.Path	getNormalizedPath(String rawPath)
private org.apache.hadoop.fs.Path	getNormalizedPath(String rawPath, Optional<String> propertyName)
static String	getPathDifference(org.apache.hadoop.fs.Path root, org.apache.hadoop.fs.Path child) Returns the relative path of the child that does not include the filename or the root path.
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
protected org.apache.hadoop.security.UserGroupInformation	getUserGroupInformation()
protected void	init(ProcessorInitializationContext context)
private void	interruptStatisticsThread(org.apache.hadoop.fs.FileSystem fileSystem)
(package private) boolean	isAllowExplicitKeytab()
protected boolean	isFileSystemAccessDenied(URI fileSystemUri)
(package private) boolean	isLocalFileSystemAccessDenied()
protected void	preProcessConfiguration(org.apache.hadoop.conf.Configuration config, ProcessContext context) This method will be called after the Configuration has been created, but before the FileSystem is created, allowing sub-classes to take further action on the Configuration before creating the FileSystem.
(package private) HdfsResources	resetHDFSResources(List<String> resourceLocations, ProcessContext context)
protected Collection<ValidationResult>	validateFileSystem(org.apache.hadoop.conf.Configuration configuration)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger, onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, getRelationships, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.processor.Processor

isStateful

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate

Field Detail

ALLOW_EXPLICIT_KEYTAB

private static final String ALLOW_EXPLICIT_KEYTAB

See Also:

Constant Field Values

DENY_LFS_ACCESS

private static final String DENY_LFS_ACCESS

See Also:

Constant Field Values

DENY_LFS_EXPLANATION

private static final String DENY_LFS_EXPLANATION

LOCAL_FILE_SYSTEM_URI

private static final Pattern LOCAL_FILE_SYSTEM_URI

NORMALIZE_ERROR_WITH_PROPERTY

private static final String NORMALIZE_ERROR_WITH_PROPERTY

See Also:

Constant Field Values

NORMALIZE_ERROR_WITHOUT_PROPERTY

private static final String NORMALIZE_ERROR_WITHOUT_PROPERTY

See Also:

Constant Field Values

HADOOP_CONFIGURATION_RESOURCES

public static final PropertyDescriptor HADOOP_CONFIGURATION_RESOURCES

DIRECTORY

public static final PropertyDescriptor DIRECTORY

COMPRESSION_CODEC

public static final PropertyDescriptor COMPRESSION_CODEC

KERBEROS_RELOGIN_PERIOD

public static final PropertyDescriptor KERBEROS_RELOGIN_PERIOD

ADDITIONAL_CLASSPATH_RESOURCES

public static final PropertyDescriptor ADDITIONAL_CLASSPATH_RESOURCES

KERBEROS_CREDENTIALS_SERVICE

public static final PropertyDescriptor KERBEROS_CREDENTIALS_SERVICE

KERBEROS_USER_SERVICE

static final PropertyDescriptor KERBEROS_USER_SERVICE

ABSOLUTE_HDFS_PATH_ATTRIBUTE

public static final String ABSOLUTE_HDFS_PATH_ATTRIBUTE

See Also:

Constant Field Values

TARGET_HDFS_DIR_CREATED_ATTRIBUTE

protected static final String TARGET_HDFS_DIR_CREATED_ATTRIBUTE

See Also:

Constant Field Values

RESOURCES_LOCK

private static final Object RESOURCES_LOCK

EMPTY_HDFS_RESOURCES

private static final HdfsResources EMPTY_HDFS_RESOURCES

kerberosProperties

protected KerberosProperties kerberosProperties

properties

protected List<PropertyDescriptor> properties

kerberosConfigFile

private volatile File kerberosConfigFile

hdfsResources

private final AtomicReference<HdfsResources> hdfsResources

validationResourceHolder

private final AtomicReference<AbstractHadoopProcessor.ValidationResources> validationResourceHolder

Constructor Detail

AbstractHadoopProcessor

public AbstractHadoopProcessor()

Method Detail

init

protected void init(ProcessorInitializationContext context)

Overrides:

init in class AbstractSessionFactoryProcessor

getKerberosProperties

protected KerberosProperties getKerberosProperties(File kerberosConfigFile)

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

getClassloaderIsolationKey

public String getClassloaderIsolationKey(PropertyContext context)

Specified by:

getClassloaderIsolationKey in interface ClassloaderIsolationKeyProvider

customValidate

protected Collection<ValidationResult> customValidate(ValidationContext validationContext)

Overrides:

customValidate in class AbstractConfigurableComponent

validateFileSystem

protected Collection<ValidationResult> validateFileSystem(org.apache.hadoop.conf.Configuration configuration)

getHadoopConfigurationForValidation

protected org.apache.hadoop.conf.Configuration getHadoopConfigurationForValidation(List<String> locations)
                                                                            throws IOException

Throws:

IOException

abstractOnScheduled

@OnScheduled
public final void abstractOnScheduled(ProcessContext context)
                                            throws IOException

If your subclass also has an @OnScheduled annotated method and you need hdfsResources in that method, then be sure to call super.abstractOnScheduled(context)

Throws:

IOException

getConfigLocations

protected List<String> getConfigLocations(PropertyContext context)

abstractOnStopped

@OnStopped
public final void abstractOnStopped()

interruptStatisticsThread

private void interruptStatisticsThread(org.apache.hadoop.fs.FileSystem fileSystem)
                                throws NoSuchFieldException,
                                       IllegalAccessException

Throws:

NoSuchFieldException

IllegalAccessException

getConfigurationFromResources

private static org.apache.hadoop.conf.Configuration getConfigurationFromResources(org.apache.hadoop.conf.Configuration config,
                                                                                  List<String> locations)
                                                                           throws IOException

Throws:

IOException

resetHDFSResources

HdfsResources resetHDFSResources(List<String> resourceLocations,
                                 ProcessContext context)
                          throws IOException

Throws:

IOException

getKerberosUser

private KerberosUser getKerberosUser(ProcessContext context)

preProcessConfiguration

protected void preProcessConfiguration(org.apache.hadoop.conf.Configuration config,
                                       ProcessContext context)

This method will be called after the Configuration has been created, but before the FileSystem is created, allowing sub-classes to take further action on the Configuration before creating the FileSystem.

Parameters:

config - the Configuration that will be used to create the FileSystem

context - the context that can be used to retrieve additional values

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration config)
                                                 throws IOException

This exists in order to allow unit tests to override it so that they don't take several minutes waiting for UDP packets to be received

Parameters:

config - the configuration to use

Returns:

the FileSystem that is created for the given Configuration

Throws:

IOException - if unable to create the FileSystem

getFileSystemAsUser

protected org.apache.hadoop.fs.FileSystem getFileSystemAsUser(org.apache.hadoop.conf.Configuration config,
                                                              org.apache.hadoop.security.UserGroupInformation ugi)
                                                       throws IOException

Throws:

IOException

checkHdfsUriForTimeout

protected void checkHdfsUriForTimeout(org.apache.hadoop.conf.Configuration config)
                               throws IOException

Throws:

IOException

getCompressionCodec

protected org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec(ProcessContext context,
                                                                             org.apache.hadoop.conf.Configuration configuration)

Returns the configured CompressionCodec, or null if none is configured.

Parameters:

context - the ProcessContext

configuration - the Hadoop Configuration

Returns:

CompressionCodec or null

getPathDifference

public static String getPathDifference(org.apache.hadoop.fs.Path root,
                                       org.apache.hadoop.fs.Path child)

Returns the relative path of the child that does not include the filename or the root path.

Parameters:

root - the path to relativize from

child - the path to relativize

Returns:

the relative path

getConfiguration

protected org.apache.hadoop.conf.Configuration getConfiguration()

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem()

getUserGroupInformation

protected org.apache.hadoop.security.UserGroupInformation getUserGroupInformation()

isAllowExplicitKeytab

boolean isAllowExplicitKeytab()

isLocalFileSystemAccessDenied

boolean isLocalFileSystemAccessDenied()

isFileSystemAccessDenied

protected boolean isFileSystemAccessDenied(URI fileSystemUri)

getNormalizedPath

protected org.apache.hadoop.fs.Path getNormalizedPath(ProcessContext context,
                                                      PropertyDescriptor property)

getNormalizedPath

protected org.apache.hadoop.fs.Path getNormalizedPath(String rawPath)

getNormalizedPath

protected org.apache.hadoop.fs.Path getNormalizedPath(ProcessContext context,
                                                      PropertyDescriptor property,
                                                      FlowFile flowFile)

getNormalizedPath

private org.apache.hadoop.fs.Path getNormalizedPath(String rawPath,
                                                    Optional<String> propertyName)