org.apache.flink.streaming.api.operators.collect

类 CollectSinkFunction<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.operators.collect.CollectSinkFunction<IN>

类型参数:

IN - type of results to be written into the sink.

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.runtime.state.CheckpointListener, CheckpointedFunction, SinkFunction<IN>

@Internal
public class CollectSinkFunction<IN>
extends RichSinkFunction<IN>
implements CheckpointedFunction, org.apache.flink.runtime.state.CheckpointListener

A sink function that collects query results and sends them back to the client.

This sink works by limiting the number of results buffered in it (can be configured) so that when the buffer is full, it back-pressures the job until the client consumes some results.

NOTE: When using this sink, make sure that its parallelism is 1, and make sure that it is used in a StreamTask.

Communication Protocol Explanation

We maintain the following variables in this communication protocol

1. version: This variable will be set to a random value when the sink opens. Client discovers that the sink has restarted if this variable is different.
2. offset: This indicates that client has successfully received the results before this offset. Sink can safely throw these results away.
3. lastCheckpointedOffset: This is the value of offset when the checkpoint happens. This value will be restored from the checkpoint and set back to offset when the sink restarts. Clients who need exactly-once semantics need to rely on this value for the position to revert when a failover happens.

Client will put version and offset into the request, indicating that it thinks what the current version is and it has received this much results.

Sink will check the validity of the request. If version mismatches or offset is smaller than expected, sink will send back the current version and lastCheckpointedOffset with an empty result list.

If the request is valid, sink prepares some results starting from offset and sends them back to the client with lastCheckpointedOffset. If there is currently no results starting from offset, sink will not wait but will instead send back an empty result list.

For client who wants exactly-once semantics, when receiving the response, the client will check for the following conditions:

1. If the version mismatches, client knows that sink has restarted. It will throw away all uncheckpointed results after lastCheckpointedOffset.
2. If lastCheckpointedOffset increases, client knows that a checkpoint happens. It can now move all results before this offset to a user-visible buffer.
3. If the response also contains new results, client will now move these new results into uncheckpointed buffer.

Note that

1. user can only see results before a lastCheckpointedOffset, and
2. client will go back to the latest lastCheckpointedOffset when sink restarts,

client will never throw away results in user-visible buffer. So this communication protocol achieves exactly-once semantics.

In order not to block job finishing/cancelling, if there are still results in sink's buffer when job terminates, these results will be sent back to client through accumulators.

另请参阅:

序列化表格

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.sink.SinkFunction

SinkFunction.Context<T>

构造器概要

构造器

构造器和说明
CollectSinkFunction(org.apache.flink.api.common.typeutils.TypeSerializer<IN> serializer, int maxResultsPerBatch, String accumulatorName)

方法概要

限定符和类型	方法和说明
void	accumulateFinalResults()
void	close()
static <T> org.apache.flink.api.java.tuple.Tuple2<Long,CollectCoordinationResponse<T>>	deserializeAccumulatorResult(byte[] serializedAccResults)
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) This method is called when the parallel function instance is created during distributed execution.
void	invoke(IN value, SinkFunction.Context context) Writes the given value to the sink.
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration parameters)
static <T> byte[]	serializeAccumulatorResult(long offset, String version, long lastCheckpointedOffset, List<T> bufferedResults, org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer)
void	setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway eventGateway)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) This method is called when a snapshot for a checkpoint is requested.

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.functions.sink.SinkFunction

invoke

构造器详细资料

CollectSinkFunction

public CollectSinkFunction(org.apache.flink.api.common.typeutils.TypeSerializer<IN> serializer,
                           int maxResultsPerBatch,
                           String accumulatorName)

方法详细资料

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

从接口复制的说明: CheckpointedFunction

This method is called when the parallel function instance is created during distributed execution. Functions typically set up their state storing data structures in this method.

指定者:

initializeState 在接口中 CheckpointedFunction

参数:

context - the context for initializing the operator

抛出:

Exception - Thrown, if state could not be created ot restored.

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

从接口复制的说明: CheckpointedFunction

This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.

指定者:

snapshotState 在接口中 CheckpointedFunction

参数:

context - the context for drawing a snapshot of the operator

抛出:

Exception - Thrown, if state could not be created ot restored.

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

指定者:

open 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

open 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

invoke

public void invoke(IN value,
                   SinkFunction.Context context)
            throws Exception

从接口复制的说明: SinkFunction

Writes the given value to the sink. This function is called for every record.

You have to override this method when implementing a SinkFunction, this is a default method for backward compatibility with the old-style method only.

指定者:

invoke 在接口中 SinkFunction<IN>

参数:

value - The input record.

context - Additional context about the input record.

抛出:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

close 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

accumulateFinalResults

public void accumulateFinalResults()
                            throws Exception

抛出:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

指定者:

notifyCheckpointComplete 在接口中 org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

指定者:

notifyCheckpointAborted 在接口中 org.apache.flink.runtime.state.CheckpointListener

setOperatorEventGateway

public void setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway eventGateway)

serializeAccumulatorResult

@VisibleForTesting
public static <T> byte[] serializeAccumulatorResult(long offset,
                                                                       String version,
                                                                       long lastCheckpointedOffset,
                                                                       List<T> bufferedResults,
                                                                       org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer)
                                                                throws IOException

抛出:

IOException

deserializeAccumulatorResult

public static <T> org.apache.flink.api.java.tuple.Tuple2<Long,CollectCoordinationResponse<T>> deserializeAccumulatorResult(byte[] serializedAccResults)
                                                                                                                    throws IOException

抛出:

IOException