org.apache.flink.runtime.io.network.partition

Class PipelinedResultPartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultPartition

org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition

org.apache.flink.runtime.io.network.partition.PipelinedResultPartition

All Implemented Interfaces:

AutoCloseable, AvailabilityProvider, ResultPartitionWriter, ChannelStateHolder, CheckpointedResultPartition

public class PipelinedResultPartition
extends BufferWritingResultPartition
implements CheckpointedResultPartition, ChannelStateHolder

A result output of a task, pipelined (streamed) to the receivers.

This result partition implementation is used both in batch and streaming. For streaming it supports low latency transfers (ensure data is sent within x milliseconds) or unconstrained while for batch it transfers only once a buffer is full. Additionally, for streaming use this typically limits the length of the buffer backlog to not have too much data in flight, while for batch we do not constrain this.

Specifics of the PipelinedResultPartition

The PipelinedResultPartition cannot reconnect once a consumer disconnects (finished or errored). Once all consumers have disconnected (released the subpartition, notified via the call onConsumedSubpartition(int)) then the partition as a whole is disposed and all buffers are freed.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from class org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition

subpartitions

Fields inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition

bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numSubpartitions, partitionId, partitionManager, partitionType

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
PipelinedResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, BufferCompressor bufferCompressor, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

Method Summary

Modifier and Type	Method and Description
void	close() Closes the partition writer which releases the allocated resource, for example the buffer pool.
void	finishReadRecoveredState(boolean notifyAndBlockOnCompletion)
void	flush(int targetSubpartition) Manually trigger the consumption of data from the given subpartitions.
void	flushAll() Manually trigger the consumption of data from all subpartitions.
CheckpointedResultSubpartition	getCheckpointedSubpartition(int subpartitionIndex) Gets the checkpointed subpartition with the given subpartitionIndex.
void	setChannelStateWriter(ChannelStateWriter channelStateWriter) Injects the ChannelStateWriter.
String	toString()

Methods inherited from class org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition

broadcastEvent, broadcastRecord, createSubpartitionView, emitRecord, finish, flushAllSubpartitions, flushSubpartition, getAllPartitions, getBackPressuredTimeMsPerSecond, getNumberOfQueuedBuffers, getNumberOfQueuedBuffers, releaseInternal, setMetricGroup, setup

Methods inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition

canBeCompressed, checkInProduceState, fail, getAvailableFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, isFinished, isReleased, release, release

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Constructor Detail

PipelinedResultPartition

public PipelinedResultPartition(String owningTaskName,
                                int partitionIndex,
                                ResultPartitionID partitionId,
                                ResultPartitionType partitionType,
                                ResultSubpartition[] subpartitions,
                                int numTargetKeyGroups,
                                ResultPartitionManager partitionManager,
                                @Nullable
                                BufferCompressor bufferCompressor,
                                org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)

Method Detail

setChannelStateWriter

public void setChannelStateWriter(ChannelStateWriter channelStateWriter)

Description copied from interface: ChannelStateHolder

Injects the ChannelStateWriter. Must only be called once.

Specified by:

setChannelStateWriter in interface ChannelStateHolder

getCheckpointedSubpartition

public CheckpointedResultSubpartition getCheckpointedSubpartition(int subpartitionIndex)

Description copied from interface: CheckpointedResultPartition

Gets the checkpointed subpartition with the given subpartitionIndex.

Specified by:

getCheckpointedSubpartition in interface CheckpointedResultPartition

flushAll

public void flushAll()

Description copied from interface: ResultPartitionWriter

Manually trigger the consumption of data from all subpartitions.

Specified by:

flushAll in interface ResultPartitionWriter

flush

public void flush(int targetSubpartition)

Description copied from interface: ResultPartitionWriter

Manually trigger the consumption of data from the given subpartitions.

Specified by:

flush in interface ResultPartitionWriter

toString

public String toString()

Overrides:

toString in class ResultPartition

finishReadRecoveredState

public void finishReadRecoveredState(boolean notifyAndBlockOnCompletion)
                              throws IOException

Specified by:

finishReadRecoveredState in interface CheckpointedResultPartition

Throws:

IOException

close

public void close()

Description copied from interface: ResultPartitionWriter

Closes the partition writer which releases the allocated resource, for example the buffer pool.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ResultPartitionWriter

Overrides:

close in class BufferWritingResultPartition