org.apache.flink.streaming.api.functions.sink

类 TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>

类型参数:

IN - Input type for SinkFunction.

TXN - Transaction to store all of the information required to handle a transaction.

CONTEXT - Context that will be shared across all invocations for the given TwoPhaseCommitSinkFunction instance. Context is created once

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.runtime.state.CheckpointListener, CheckpointedFunction, SinkFunction<IN>

@PublicEvolving
public abstract class TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>
extends RichSinkFunction<IN>
implements CheckpointedFunction, org.apache.flink.runtime.state.CheckpointListener

This is a recommended base class for all of the SinkFunction that intend to implement exactly-once semantic. It does that by implementing two phase commit algorithm on top of the CheckpointedFunction and CheckpointListener. User should provide custom TXN (transaction handle) and implement abstract methods handling this transaction handle.

另请参阅:

序列化表格

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	TwoPhaseCommitSinkFunction.State<TXN,CONTEXT> State POJO class coupling pendingTransaction, context and pendingCommitTransactions.
static class	TwoPhaseCommitSinkFunction.StateSerializer<TXN,CONTEXT> Custom TypeSerializer for the sink state.
static class	TwoPhaseCommitSinkFunction.StateSerializerConfigSnapshot<TXN,CONTEXT> 已过时。 this snapshot class is no longer in use, and is maintained only for backwards compatibility purposes. It is fully replaced by TwoPhaseCommitSinkFunction.StateSerializerSnapshot.
static class	TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN,CONTEXT> Snapshot for the TwoPhaseCommitSinkFunction.StateSerializer.
static class	TwoPhaseCommitSinkFunction.TransactionHolder<TXN> Adds metadata (currently only the start time of the transaction) to the transaction object.

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.sink.SinkFunction

SinkFunction.Context<T>

字段概要

字段

限定符和类型	字段和说明
protected LinkedHashMap<Long,TwoPhaseCommitSinkFunction.TransactionHolder<TXN>>	pendingCommitTransactions
protected org.apache.flink.api.common.state.ListState<TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>>	state
protected Optional<CONTEXT>	userContext

构造器概要

构造器

构造器和说明
TwoPhaseCommitSinkFunction(org.apache.flink.api.common.typeutils.TypeSerializer<TXN> transactionSerializer, org.apache.flink.api.common.typeutils.TypeSerializer<CONTEXT> contextSerializer) Use default ListStateDescriptor for internal state serialization.

方法概要

限定符和类型	方法和说明
protected abstract void	abort(TXN transaction) Abort a transaction.
protected abstract TXN	beginTransaction() Method that starts a new transaction.
void	close()
protected abstract void	commit(TXN transaction) Commit a pre-committed transaction.
protected TXN	currentTransaction()
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	enableTransactionTimeoutWarnings(double warningRatio) Enables logging of warnings if a transaction's elapsed time reaches a specified ratio of the transactionTimeout.
protected void	finishRecoveringContext(Collection<TXN> handledTransactions) Callback for subclasses which is called after restoring (each) user context.
protected Optional<CONTEXT>	getUserContext()
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	ignoreFailuresAfterTransactionTimeout() If called, the sink will only log but not propagate exceptions thrown in recoverAndCommit(Object) if the transaction is older than a specified transaction timeout.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context) This method is called when the parallel function instance is created during distributed execution.
protected Optional<CONTEXT>	initializeUserContext()
void	invoke(IN value) This should not be implemented by subclasses.
void	invoke(IN value, SinkFunction.Context context) Writes the given value to the sink.
protected abstract void	invoke(TXN transaction, IN value, SinkFunction.Context context) Write value within a transaction.
void	notifyCheckpointComplete(long checkpointId)
protected java.util.stream.Stream<Map.Entry<Long,TXN>>	pendingTransactions()
protected abstract void	preCommit(TXN transaction) Pre commit previously created transaction.
protected void	recoverAndAbort(TXN transaction) Abort a transaction that was rejected by a coordinator after a failure.
protected void	recoverAndCommit(TXN transaction) Invoked on recovered transactions after a failure.
protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT>	setTransactionTimeout(long transactionTimeout) Sets the transaction timeout.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) This method is called when a snapshot for a checkpoint is requested.

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

pendingCommitTransactions

protected final LinkedHashMap<Long,TwoPhaseCommitSinkFunction.TransactionHolder<TXN>> pendingCommitTransactions

userContext

protected transient Optional<CONTEXT> userContext

state

protected transient org.apache.flink.api.common.state.ListState<TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>> state

构造器详细资料

TwoPhaseCommitSinkFunction

public TwoPhaseCommitSinkFunction(org.apache.flink.api.common.typeutils.TypeSerializer<TXN> transactionSerializer,
                                  org.apache.flink.api.common.typeutils.TypeSerializer<CONTEXT> contextSerializer)

Use default ListStateDescriptor for internal state serialization. Helpful utilities for using this constructor are TypeInformation.of(Class), TypeHint and TypeInformation.of(TypeHint). Example:

 
 TwoPhaseCommitSinkFunction(TypeInformation.of(new TypeHint<State<TXN, CONTEXT>>() {}));
 
 

参数:

transactionSerializer - TypeSerializer for the transaction type of this sink

contextSerializer - TypeSerializer for the context type of this sink

方法详细资料

initializeUserContext

protected Optional<CONTEXT> initializeUserContext()

getUserContext

protected Optional<CONTEXT> getUserContext()

currentTransaction

@Nullable
protected TXN currentTransaction()

pendingTransactions

@Nonnull
protected java.util.stream.Stream<Map.Entry<Long,TXN>> pendingTransactions()

invoke

protected abstract void invoke(TXN transaction,
                               IN value,
                               SinkFunction.Context context)
                        throws Exception

Write value within a transaction.

抛出:

Exception

beginTransaction

protected abstract TXN beginTransaction()
                                 throws Exception

Method that starts a new transaction.

返回:

newly created transaction.

抛出:

Exception

preCommit

protected abstract void preCommit(TXN transaction)
                           throws Exception

Pre commit previously created transaction. Pre commit must make all of the necessary steps to prepare the transaction for a commit that might happen in the future. After this point the transaction might still be aborted, but underlying implementation must ensure that commit calls on already pre committed transactions will always succeed.

Usually implementation involves flushing the data.

抛出:

Exception

commit

protected abstract void commit(TXN transaction)

Commit a pre-committed transaction. If this method fail, Flink application will be restarted and recoverAndCommit(Object) will be called again for the same transaction.

recoverAndCommit

protected void recoverAndCommit(TXN transaction)

Invoked on recovered transactions after a failure. User implementation must ensure that this call will eventually succeed. If it fails, Flink application will be restarted and it will be invoked again. If it does not succeed eventually, a data loss will occur. Transactions will be recovered in an order in which they were created.

abort

protected abstract void abort(TXN transaction)

Abort a transaction.

recoverAndAbort

protected void recoverAndAbort(TXN transaction)

Abort a transaction that was rejected by a coordinator after a failure.

finishRecoveringContext

protected void finishRecoveringContext(Collection<TXN> handledTransactions)

Callback for subclasses which is called after restoring (each) user context.

参数:

handledTransactions - transactions which were already committed or aborted and do not need further handling

invoke

public final void invoke(IN value)
                  throws Exception

This should not be implemented by subclasses.

指定者:

invoke 在接口中 SinkFunction<IN>

抛出:

Exception

invoke

public final void invoke(IN value,
                         SinkFunction.Context context)
                  throws Exception

从接口复制的说明: SinkFunction

Writes the given value to the sink. This function is called for every record.

You have to override this method when implementing a SinkFunction, this is a default method for backward compatibility with the old-style method only.

指定者:

invoke 在接口中 SinkFunction<IN>

参数:

value - The input record.

context - Additional context about the input record.

抛出:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

notifyCheckpointComplete

public final void notifyCheckpointComplete(long checkpointId)
                                    throws Exception

指定者:

notifyCheckpointComplete 在接口中 org.apache.flink.runtime.state.CheckpointListener

抛出:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

从接口复制的说明: CheckpointedFunction

This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.

指定者:

snapshotState 在接口中 CheckpointedFunction

参数:

context - the context for drawing a snapshot of the operator

抛出:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

从接口复制的说明: CheckpointedFunction

This method is called when the parallel function instance is created during distributed execution. Functions typically set up their state storing data structures in this method.

指定者:

initializeState 在接口中 CheckpointedFunction

参数:

context - the context for initializing the operator

抛出:

Exception

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

close 在类中 org.apache.flink.api.common.functions.AbstractRichFunction

抛出:

Exception

setTransactionTimeout

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> setTransactionTimeout(long transactionTimeout)

Sets the transaction timeout. Setting only the transaction timeout has no effect in itself.

参数:

transactionTimeout - The transaction timeout in ms.

另请参阅:

ignoreFailuresAfterTransactionTimeout(), enableTransactionTimeoutWarnings(double)

ignoreFailuresAfterTransactionTimeout

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> ignoreFailuresAfterTransactionTimeout()

If called, the sink will only log but not propagate exceptions thrown in recoverAndCommit(Object) if the transaction is older than a specified transaction timeout. The start time of an transaction is determined by System.currentTimeMillis(). By default, failures are propagated.

enableTransactionTimeoutWarnings

protected TwoPhaseCommitSinkFunction<IN,TXN,CONTEXT> enableTransactionTimeoutWarnings(double warningRatio)

Enables logging of warnings if a transaction's elapsed time reaches a specified ratio of the transactionTimeout. If warningRatio is 0, a warning will be always logged when committing the transaction.

参数:

warningRatio - A value in the range [0,1].

返回: