org.apache.nifi.processor.util.list

Class AbstractListProcessor<T extends ListableEntity>

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processor.util.list.AbstractListProcessor<T>

All Implemented Interfaces:

ConfigurableComponent, Processor

@TriggerSerially
 @Stateful(scopes={LOCAL,CLUSTER},
          description="After a listing of resources is performed, the latest timestamp of any of the resources is stored in the component\'s state. The scope used depends on the implementation.")
public abstract class AbstractListProcessor<T extends ListableEntity>
extends AbstractProcessor

An Abstract Processor that is intended to simplify the coding required in order to perform Listing operations of remote resources. Those remote resources may be files, "objects", "messages", or any other sort of entity that may need to be listed in such a way that we identity the entity only once. Each of these objects, messages, etc. is referred to as an "entity" for the scope of this Processor.

This class is responsible for triggering the listing to occur, filtering the results returned such that only new (unlisted) entities or entities that have been modified will be emitted from the Processor.

In order to make use of this abstract class, the entities listed must meet the following criteria:

• Entity must have a timestamp associated with it. This timestamp is used to determine if entities are "new" or not. Any entity that is returned by the listing will be considered "new" if the timestamp is later than the latest timestamp pulled.
• If the timestamp of an entity is before OR equal to the latest timestamp pulled, then the entity is not considered new. If the timestamp is later than the last timestamp pulled, then the entity is considered new.
• Entity must have a user-readable name that can be used for logging purposes.

This class persists state across restarts so that even if NiFi is restarted, duplicates will not be pulled from the target system given the above criteria. This is performed using the StateManager. This allows the system to be restarted and begin processing where it left off. The state that is stored is the latest timestamp that has been pulled (as determined by the timestamps of the entities that are returned). See the section above for information about how this information isused in order to determine new entities.

NOTE: This processor performs migrations of legacy state mechanisms inclusive of locally stored, file-based state and the optional utilization of the Distributed Cache Service property to the new StateManager functionality. Upon successful migration, the associated data from one or both of the legacy mechanisms is purged.

For each new entity that is listed, the Processor will send a FlowFile to the 'success' relationship. The FlowFile will have no content but will have some set of attributes (defined by the concrete implementation) that can be used to fetch those remote resources or interact with them in whatever way makes sense for the configured dataflow.

Subclasses are responsible for the following:

• Perform a listing of remote resources. The subclass will implement the performListing(ProcessContext, Long) method, which creates a listing of all entities on the remote system that have timestamps later than the provided timestamp. If the entities returned have a timestamp before the provided one, those entities will be filtered out. It is therefore not necessary to perform the filtering of timestamps but is provided in order to give the implementation the ability to filter those resources on the server side rather than pulling back all of the information, if it makes sense to do so in the concrete implementation.
• Creating a Map of attributes that are applicable for an entity. The attributes that are assigned to each FlowFile are exactly those returned by the createAttributes(ListableEntity, ProcessContext).
• Returning the configured path. Many resources can be comprised of a "path" (or a "container" or "bucket", etc.) as well as name or identifier that is unique only within that path. The getPath(ProcessContext) method is responsible for returning the path that is currently being polled for entities. If this does concept does not apply for the concrete implementation, it is recommended that the concrete implementation return "." or "/" for all invocations of this method.
• Determining when the listing must be cleared. It is sometimes necessary to clear state about which entities have already been ingested, as the result of a user changing a property value. The isListingResetNecessary(PropertyDescriptor) method is responsible for determining when the listing needs to be reset by returning a boolean indicating whether or not a change in the value of the provided property should trigger the timestamp and identifier information to be cleared.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	AbstractListProcessor.StringSerDe

Field Summary

Fields

Modifier and Type	Field and Description
static PropertyDescriptor	DISTRIBUTED_CACHE_SERVICE
private boolean	justElectedPrimaryNode
private Long	lastListingTime
private Long	lastProcessedTime
private Long	lastRunTime
static long	LISTING_LAG_NANOS
(package private) static String	LISTING_TIMESTAMP_KEY
(package private) static String	PROCESSED_TIMESTAMP_KEY
static Relationship	REL_SUCCESS
private boolean	resetState

Constructor Summary

Constructors

Constructor and Description
AbstractListProcessor()

Method Summary

Modifier and Type	Method and Description
protected abstract Map<String,String>	createAttributes(T entity, ProcessContext context) Creates a Map of attributes that should be applied to the FlowFile to represent this entity.
private EntityListing	deserialize(String serializedState)
protected String	getKey(String directory)
protected abstract String	getPath(ProcessContext context) Returns the path to perform a listing on.
File	getPersistenceFile()
Set<Relationship>	getRelationships()
protected abstract Scope	getStateScope(ProcessContext context) Returns a Scope that specifies where the state should be managed for this Processor
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
protected abstract boolean	isListingResetNecessary(PropertyDescriptor property) Determines whether or not the listing must be reset if the value of the given property is changed
private void	migrateState(String path, DistributedMapCacheClient client, StateManager stateManager, Scope scope) This processor used to use the DistributedMapCacheClient in order to store cluster-wide state, before the introduction of the StateManager.
void	onPrimaryNodeChange(PrimaryNodeState newState)
void	onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue)
void	onTrigger(ProcessContext context, ProcessSession session)
protected abstract List<T>	performListing(ProcessContext context, Long minTimestamp) Performs a listing of the remote entities that can be pulled.
private void	persist(long listingTimestamp, long processedTimestamp, StateManager stateManager, Scope scope)
private void	resetTimeStates()
void	updateState(ProcessContext context)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

customValidate, equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, validate

Field Detail

DISTRIBUTED_CACHE_SERVICE

public static final PropertyDescriptor DISTRIBUTED_CACHE_SERVICE

REL_SUCCESS

public static final Relationship REL_SUCCESS

lastListingTime

private volatile Long lastListingTime

lastProcessedTime

private volatile Long lastProcessedTime

lastRunTime

private volatile Long lastRunTime

justElectedPrimaryNode

private volatile boolean justElectedPrimaryNode

resetState

private volatile boolean resetState

LISTING_LAG_NANOS

public static final long LISTING_LAG_NANOS

LISTING_TIMESTAMP_KEY

static final String LISTING_TIMESTAMP_KEY

See Also:

Constant Field Values

PROCESSED_TIMESTAMP_KEY

static final String PROCESSED_TIMESTAMP_KEY

See Also:

Constant Field Values

Constructor Detail

AbstractListProcessor

public AbstractListProcessor()

Method Detail

getPersistenceFile

public File getPersistenceFile()

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

onPropertyModified

public void onPropertyModified(PropertyDescriptor descriptor,
                               String oldValue,
                               String newValue)

Specified by:

onPropertyModified in interface ConfigurableComponent

Overrides:

onPropertyModified in class AbstractConfigurableComponent

getRelationships

public Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

onPrimaryNodeChange

@OnPrimaryNodeStateChange
public void onPrimaryNodeChange(PrimaryNodeState newState)

updateState

@OnScheduled
public final void updateState(ProcessContext context)
                                    throws IOException

Throws:

IOException

migrateState

private void migrateState(String path,
                          DistributedMapCacheClient client,
                          StateManager stateManager,
                          Scope scope)
                   throws IOException

This processor used to use the DistributedMapCacheClient in order to store cluster-wide state, before the introduction of the StateManager. This method will migrate state from that DistributedMapCacheClient, or from a local file, to the StateManager, if any state already exists. More specifically, this will extract out the relevant timestamp for when the processor last ran

Parameters:

path - the path to migrate state for

client - the DistributedMapCacheClient that is capable of obtaining the current state

stateManager - the StateManager to use in order to store the new state

scope - the scope to use

Throws:

IOException - if unable to retrieve or store the state

persist

private void persist(long listingTimestamp,
                     long processedTimestamp,
                     StateManager stateManager,
                     Scope scope)
              throws IOException

Throws:

IOException

getKey

protected String getKey(String directory)

deserialize

private EntityListing deserialize(String serializedState)
                           throws org.codehaus.jackson.JsonParseException,
                                  org.codehaus.jackson.map.JsonMappingException,
                                  IOException

Throws:

org.codehaus.jackson.JsonParseException

org.codehaus.jackson.map.JsonMappingException

IOException

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

resetTimeStates

private void resetTimeStates()

createAttributes

protected abstract Map<String,String> createAttributes(T entity,
                                                       ProcessContext context)

Creates a Map of attributes that should be applied to the FlowFile to represent this entity. This processor will emit a FlowFile for each "new" entity (see the documentation for this class for a discussion of how this class determines whether or not an entity is "new"). The FlowFile will contain no content. The attributes that will be included are exactly the attributes that are returned by this method.

Parameters:

entity - the entity represented by the FlowFile

context - the ProcessContext for obtaining configuration information

Returns:

a Map of attributes for this entity

getPath

protected abstract String getPath(ProcessContext context)

Returns the path to perform a listing on. Many resources can be comprised of a "path" (or a "container" or "bucket", etc.) as well as name or identifier that is unique only within that path. This method is responsible for returning the path that is currently being polled for entities. If this does concept does not apply for the concrete implementation, it is recommended that the concrete implementation return "." or "/" for all invocations of this method.

Parameters:

context - the ProcessContex to use in order to obtain configuration

Returns:

the path that is to be used to perform the listing, or null if not applicable.

performListing

protected abstract List<T> performListing(ProcessContext context,
                                          Long minTimestamp)
                                   throws IOException

Performs a listing of the remote entities that can be pulled. If any entity that is returned has already been "discovered" or "emitted" by this Processor, it will be ignored. A discussion of how the Processor determines those entities that have already been emitted is provided above in the documentation for this class. Any entity that is returned by this method with a timestamp prior to the minTimestamp will be filtered out by the Processor. Therefore, it is not necessary that implementations perform this filtering but can be more efficient if the filtering can be performed on the server side prior to retrieving the information.

Parameters:

context - the ProcessContex to use in order to pull the appropriate entities

minTimestamp - the minimum timestamp of entities that should be returned.

Returns:

a Listing of entities that have a timestamp >= minTimestamp

Throws:

IOException

isListingResetNecessary

protected abstract boolean isListingResetNecessary(PropertyDescriptor property)

Determines whether or not the listing must be reset if the value of the given property is changed

Parameters:

property - the property that has changed

Returns:

true if a change in value of the given property necessitates that the listing be reset, false otherwise.

getStateScope

protected abstract Scope getStateScope(ProcessContext context)

Returns a Scope that specifies where the state should be managed for this Processor

Parameters:

context - the ProcessContext to use in order to make a determination

Returns:

a Scope that specifies where the state should be managed for this Processor