Package org.apache.nifi.controller.repository

Class StandardProcessSession

java.lang.Object
org.apache.nifi.controller.repository.StandardProcessSession
All Implemented Interfaces:
ProvenanceEventEnricher, org.apache.nifi.processor.ProcessSession
public class StandardProcessSession extends Object implements org.apache.nifi.processor.ProcessSession, ProvenanceEventEnricher

Provides a ProcessSession that ensures all accesses, changes and transfers occur in an atomic manner for all FlowFiles including their contents and attributes

NOT THREAD SAFE

  • Field Details

  • Constructor Details

  • Method Details

    • verifyTaskActive

      private void verifyTaskActive()

    • getRepositoryContext

      protected RepositoryContext getRepositoryContext()

    • getSessionId

      protected long getSessionId()

    • closeStreams

      private void closeStreams(Map<FlowFile,? extends Closeable> streamMap, String action, String streamType)

    • checkpoint

      public void checkpoint()

    • validateCommitState

      private void validateCommitState()

    • checkpoint

      private void checkpoint(boolean copyCollections)

    • isRetry

      private boolean isRetry(StandardRepositoryRecord record)

    • retry

      private void retry(StandardRepositoryRecord record, long maxBackoffMillis)

    • getRetries

      private int getRetries(FlowFile flowFile)

    • calculateBackoffTime

      private long calculateBackoffTime(int retryCount, long maxBackoffPeriod, long baseBackoffTime)

    • commit

      public void commit()
      Specified by:
      commit in interface org.apache.nifi.processor.ProcessSession

    • commitAsync

      public void commitAsync()
      Specified by:
      commitAsync in interface org.apache.nifi.processor.ProcessSession

    • commitAsync

      public void commitAsync(Runnable onSuccess, Consumer<Throwable> onFailure)
      Specified by:
      commitAsync in interface org.apache.nifi.processor.ProcessSession

    • commit

      private void commit(boolean asynchronous)

    • commit

      protected void commit(StandardProcessSession.Checkpoint checkpoint, boolean asynchronous)
      Commits the given checkpoint, updating repositories as necessary, and performing any necessary cleanup of resources, etc. Subclasses may choose to perform these tasks asynchronously if the asynchronous flag indicates that it is acceptable to do so. However, this implementation will perform the commit synchronously, regardless of the asynchronous flag.
      Parameters:
      checkpoint - the session checkpoint to commit
      asynchronous - whether or not the commit is allowed to be performed asynchronously.

    • updateEventRepository

      private void updateEventRepository(StandardProcessSession.Checkpoint checkpoint)

    • combineCounters

      private Map<String,Long> combineCounters(Map<String,Long> first, Map<String,Long> second)

    • addEventType

      private void addEventType(Map<String,BitSet> map, String id, ProvenanceEventType eventType)

    • getRecord

      private StandardRepositoryRecord getRecord(FlowFile flowFile)

    • updateProvenanceRepo

      protected void updateProvenanceRepo(StandardProcessSession.Checkpoint checkpoint)

    • updateEventContentClaims

      private void updateEventContentClaims(ProvenanceEventBuilder builder, FlowFile flowFile, StandardRepositoryRecord repoRecord)

    • enrich

      public ProvenanceEventRecord enrich(ProvenanceEventRecord rawEvent, FlowFile flowFile, long commitNanos)
      Description copied from interface: ProvenanceEventEnricher
      Returns a new Provenance event that has been updated to contain the original and updated FlowFile attributes and content claim information.
      Specified by:
      enrich in interface ProvenanceEventEnricher
      Parameters:
      rawEvent - record
      flowFile - flowfile
      commitNanos - the time (in nanoseconds) when the associated session was committed
      Returns:
      new event record

    • enrich

      private ProvenanceEventRecord enrich(ProvenanceEventRecord rawEvent, Map<String,FlowFileRecord> flowFileRecordMap, Map<Long,StandardRepositoryRecord> records, boolean updateAttributesAndContent, long commitNanos)

    • isSpuriousForkEvent

      private boolean isSpuriousForkEvent(ProvenanceEventRecord event, Set<String> removedFlowFiles)
      Checks if the given event is a spurious FORK, meaning that the FORK has a single child and that child was removed in this session. This happens when a Processor calls #create(FlowFile) and then removes the created FlowFile.
      Parameters:
      event - event
      Returns:
      true if spurious fork

    • isSpuriousRouteEvent

      private boolean isSpuriousRouteEvent(ProvenanceEventRecord event, Map<Long,StandardRepositoryRecord> records)
      Checks if the given event is a spurious ROUTE, meaning that the ROUTE indicates that a FlowFile was routed to a relationship with only 1 connection and that Connection is the Connection from which the FlowFile was pulled. I.e., the FlowFile was really routed nowhere.
      Parameters:
      event - event
      records - records
      Returns:
      true if spurious route

    • rollback

      public void rollback()
      Specified by:
      rollback in interface org.apache.nifi.processor.ProcessSession

    • rollback

      public void rollback(boolean penalize)
      Specified by:
      rollback in interface org.apache.nifi.processor.ProcessSession

    • rollback

      protected void rollback(boolean penalize, boolean rollbackCheckpoint)

    • rollbackRecord

      protected void rollbackRecord(StandardRepositoryRecord record, boolean penalize)
      Rolls back the Record in a manner that is appropriate for the context. The default implementation is to place the queue back on its original queue, if it exists, or just ignore it if it has no original queue. However, subclasses may wish to change the behavior for how Records are handled when a rollback occurs.
      Parameters:
      record - the Record that is to be rolled back
      penalize - whether or not the Record should be penalized

    • loggableFlowfileInfo

      private String loggableFlowfileInfo()

    • decrementClaimCount

      private void decrementClaimCount(ContentClaim claim)

    • destroyContent

      private void destroyContent(ContentClaim claim, StandardRepositoryRecord repoRecord)
      Destroys a ContentClaim that was being written to but is no longer needed
      Parameters:
      claim - claim to destroy

    • resetState

      private void resetState()

    • acknowledgeRecords

      private void acknowledgeRecords()

    • migrate

      public void migrate(org.apache.nifi.processor.ProcessSession newOwner)
      Specified by:
      migrate in interface org.apache.nifi.processor.ProcessSession

    • migrate

      public void migrate(org.apache.nifi.processor.ProcessSession newOwner, Collection<FlowFile> flowFiles)
      Specified by:
      migrate in interface org.apache.nifi.processor.ProcessSession

    • migrate

      private void migrate(StandardProcessSession newOwner, Collection<FlowFile> flowFiles)

    • summarizeEvents

      private String summarizeEvents(StandardProcessSession.Checkpoint checkpoint)

    • formatNanos

      private void formatNanos(long nanos, StringBuilder sb)

    • incrementConnectionInputCounts

      private void incrementConnectionInputCounts(Connection connection, RepositoryRecord record)

    • incrementConnectionInputCounts

      private void incrementConnectionInputCounts(String connectionId, int flowFileCount, long bytes)

    • incrementConnectionOutputCounts

      private void incrementConnectionOutputCounts(Connection connection, FlowFileRecord record)

    • incrementConnectionOutputCounts

      private void incrementConnectionOutputCounts(String connectionId, int flowFileCount, long bytes)

    • registerDequeuedRecord

      private void registerDequeuedRecord(FlowFileRecord flowFile, Connection connection)

    • handleConflictingId

      private void handleConflictingId(FlowFileRecord flowFile, Connection connection, StandardRepositoryRecord conflict)

    • adjustCounter

      public void adjustCounter(String name, long delta, boolean immediate)
      Specified by:
      adjustCounter in interface org.apache.nifi.processor.ProcessSession

    • adjustCounter

      private void adjustCounter(String name, long delta, Map<String,Long> map)

    • get

      public FlowFile get()
      Specified by:
      get in interface org.apache.nifi.processor.ProcessSession

    • get

      public List<FlowFile> get(int maxResults)
      Specified by:
      get in interface org.apache.nifi.processor.ProcessSession

    • get

      public List<FlowFile> get(org.apache.nifi.processor.FlowFileFilter filter)
      Specified by:
      get in interface org.apache.nifi.processor.ProcessSession

    • get

      private List<FlowFile> get(StandardProcessSession.ConnectionPoller poller, boolean lockAllQueues)

    • getQueueSize

      public QueueSize getQueueSize()
      Specified by:
      getQueueSize in interface org.apache.nifi.processor.ProcessSession

    • create

      public FlowFile create()
      Specified by:
      create in interface org.apache.nifi.processor.ProcessSession

    • create

      public FlowFile create(FlowFile parent)
      Specified by:
      create in interface org.apache.nifi.processor.ProcessSession

    • create

      public FlowFile create(Collection<FlowFile> parents)
      Specified by:
      create in interface org.apache.nifi.processor.ProcessSession

    • clone

      public FlowFile clone(FlowFile example)
      Specified by:
      clone in interface org.apache.nifi.processor.ProcessSession

    • clone

      public FlowFile clone(FlowFile example, long offset, long size)
      Specified by:
      clone in interface org.apache.nifi.processor.ProcessSession

    • registerForkEvent

      private void registerForkEvent(FlowFile parent, FlowFile child)

    • registerJoinEvent

      private void registerJoinEvent(FlowFile child, Collection<FlowFile> parents)

    • penalize

      public FlowFile penalize(FlowFile flowFile)
      Specified by:
      penalize in interface org.apache.nifi.processor.ProcessSession

    • penalize

      public FlowFile penalize(FlowFile flowFile, long period, TimeUnit timeUnit)

    • putAttribute

      public FlowFile putAttribute(FlowFile flowFile, String key, String value)
      Specified by:
      putAttribute in interface org.apache.nifi.processor.ProcessSession

    • putAllAttributes

      public FlowFile putAllAttributes(FlowFile flowFile, Map<String,String> attributes)
      Specified by:
      putAllAttributes in interface org.apache.nifi.processor.ProcessSession

    • removeAttribute

      public FlowFile removeAttribute(FlowFile flowFile, String key)
      Specified by:
      removeAttribute in interface org.apache.nifi.processor.ProcessSession

    • removeAllAttributes

      public FlowFile removeAllAttributes(FlowFile flowFile, Set<String> keys)
      Specified by:
      removeAllAttributes in interface org.apache.nifi.processor.ProcessSession

    • removeAllAttributes

      public FlowFile removeAllAttributes(FlowFile flowFile, Pattern keyPattern)
      Specified by:
      removeAllAttributes in interface org.apache.nifi.processor.ProcessSession

    • updateLastQueuedDate

      private void updateLastQueuedDate(StandardRepositoryRecord record, Long lastQueueDate)

    • updateLastQueuedDate

      private void updateLastQueuedDate(StandardRepositoryRecord record)

    • transfer

      public void transfer(FlowFile flowFile, org.apache.nifi.processor.Relationship relationship)
      Specified by:
      transfer in interface org.apache.nifi.processor.ProcessSession

    • transfer

      public void transfer(FlowFile flowFile)
      Specified by:
      transfer in interface org.apache.nifi.processor.ProcessSession

    • transfer

      public void transfer(Collection<FlowFile> flowFiles)
      Specified by:
      transfer in interface org.apache.nifi.processor.ProcessSession

    • transfer

      public void transfer(Collection<FlowFile> flowFiles, org.apache.nifi.processor.Relationship relationship)
      Specified by:
      transfer in interface org.apache.nifi.processor.ProcessSession

    • remove

      public void remove(FlowFile flowFile)
      Specified by:
      remove in interface org.apache.nifi.processor.ProcessSession

    • remove

      public void remove(Collection<FlowFile> flowFiles)
      Specified by:
      remove in interface org.apache.nifi.processor.ProcessSession

    • removeForkEvents

      private void removeForkEvents(FlowFile flowFile)

    • expireFlowFiles

      public void expireFlowFiles()

    • removeExpired

      private void removeExpired(Set<FlowFileRecord> flowFiles, Connection connection)

    • getInputStream

      private InputStream getInputStream(FlowFile flowFile, ContentClaim claim, long contentClaimOffset, boolean allowCachingOfStream) throws ContentNotFoundException
      Throws:
      ContentNotFoundException

    • read

      public void read(FlowFile source, org.apache.nifi.processor.io.InputStreamCallback reader)
      Specified by:
      read in interface org.apache.nifi.processor.ProcessSession

    • read

      public InputStream read(FlowFile source)
      Specified by:
      read in interface org.apache.nifi.processor.ProcessSession

    • createTaskTerminationStream

      private InputStream createTaskTerminationStream(InputStream delegate)

    • createTaskTerminationStream

      private OutputStream createTaskTerminationStream(OutputStream delegate)

    • incrementReadCount

      private void incrementReadCount(FlowFile flowFile)

    • decrementReadCount

      private void decrementReadCount(FlowFile flowFile)

    • merge

      public FlowFile merge(Collection<FlowFile> sources, FlowFile destination)
      Specified by:
      merge in interface org.apache.nifi.processor.ProcessSession

    • merge

      public FlowFile merge(Collection<FlowFile> sources, FlowFile destination, byte[] header, byte[] footer, byte[] demarcator)
      Specified by:
      merge in interface org.apache.nifi.processor.ProcessSession

    • ensureNotAppending

      private void ensureNotAppending(ContentClaim claim) throws IOException
      Throws:
      IOException

    • write

      public OutputStream write(FlowFile source)
      Specified by:
      write in interface org.apache.nifi.processor.ProcessSession

    • write

      public FlowFile write(FlowFile source, org.apache.nifi.processor.io.OutputStreamCallback writer)
      Specified by:
      write in interface org.apache.nifi.processor.ProcessSession

    • append

      public FlowFile append(FlowFile source, org.apache.nifi.processor.io.OutputStreamCallback writer)
      Specified by:
      append in interface org.apache.nifi.processor.ProcessSession

    • removeTemporaryClaim

      private void removeTemporaryClaim(StandardRepositoryRecord record)
      Checks if the ContentClaim associated with this record should be removed, since the record is about to be updated to point to a new content claim. If so, removes the working claim. This happens if invalid input: '&' only if the content of this FlowFile has been modified since it was last committed to the FlowFile repository, because this indicates that the content is no longer needed and should be cleaned up.
      Parameters:
      record - record

    • resetWriteClaims

      private void resetWriteClaims()

    • resetWriteClaims

      private void resetWriteClaims(boolean suppressExceptions)

    • resetReadClaim

      private void resetReadClaim()

    • write

      public FlowFile write(FlowFile source, org.apache.nifi.processor.io.StreamCallback writer)
      Specified by:
      write in interface org.apache.nifi.processor.ProcessSession

    • importFrom

      public FlowFile importFrom(Path source, boolean keepSourceFile, FlowFile destination)
      Specified by:
      importFrom in interface org.apache.nifi.processor.ProcessSession

    • importFrom

      public FlowFile importFrom(InputStream source, FlowFile destination)
      Specified by:
      importFrom in interface org.apache.nifi.processor.ProcessSession

    • exportTo

      public void exportTo(FlowFile source, Path destination, boolean append)
      Specified by:
      exportTo in interface org.apache.nifi.processor.ProcessSession

    • exportTo

      public void exportTo(FlowFile source, OutputStream destination)
      Specified by:
      exportTo in interface org.apache.nifi.processor.ProcessSession

    • handleContentNotFound

      private void handleContentNotFound(ContentNotFoundException nfe, Collection<StandardRepositoryRecord> suspectRecords)

    • handleContentNotFound

      private void handleContentNotFound(ContentNotFoundException nfe, StandardRepositoryRecord suspectRecord)

    • validateRecordState

      private FlowFile validateRecordState(FlowFile flowFile)

    • validateRecordState

      private FlowFile validateRecordState(FlowFile flowFile, boolean allowRecursiveRead)

    • validateRecordState

      private List<FlowFile> validateRecordState(Collection<FlowFile> flowFiles)

    • isFlowFileKnown

      boolean isFlowFileKnown(FlowFile flowFile)
      Checks if a FlowFile is known in this session.
      Parameters:
      flowFile - the FlowFile to check
      Returns:
      true if the FlowFile is known in this session, false otherwise.

    • getMostRecent

      private FlowFile getMostRecent(FlowFile flowFile)

    • intersectAttributes

      private static Map<String,String> intersectAttributes(Collection<FlowFile> flowFileList)
      Returns the attributes that are common to every FlowFile given. The key and value must match exactly.
      Parameters:
      flowFileList - a list of FlowFiles
      Returns:
      the common attributes

    • getProvenanceReporter

      public ProvenanceReporter getProvenanceReporter()
      Specified by:
      getProvenanceReporter in interface org.apache.nifi.processor.ProcessSession

    • setState

      public void setState(Map<String,String> state, Scope scope) throws IOException
      Specified by:
      setState in interface org.apache.nifi.processor.ProcessSession
      Throws:
      IOException

    • setState

      private void setState(StateMap stateMap, Scope scope)

    • getState

      public StateMap getState(Scope scope) throws IOException
      Specified by:
      getState in interface org.apache.nifi.processor.ProcessSession
      Throws:
      IOException

    • replaceState

      public boolean replaceState(StateMap oldValue, Map<String,String> newValue, Scope scope) throws IOException
      Specified by:
      replaceState in interface org.apache.nifi.processor.ProcessSession
      Throws:
      IOException

    • clearState

      public void clearState(Scope scope)
      Specified by:
      clearState in interface org.apache.nifi.processor.ProcessSession

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getIncrementedVersion

      private String getIncrementedVersion(String currentVersion)