org.apache.flink.streaming.api.functions.source

Class MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<Type>

org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase<Type,UId>

org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>

Type Parameters:

Type - The type of the messages created by the source.

UId - The type of the unique IDs which are consistent across sessions.

SessionId - The type of the IDs that are used for acknowledging elements (ids valid during session).

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.runtime.state.CheckpointListener, Checkpointed<org.apache.flink.runtime.state.SerializedCheckpointData[]>, SourceFunction<Type>

@PublicEvolving
public abstract class MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>
extends MessageAcknowledgingSourceBase<Type,UId>

Abstract base class for data sources that receive elements from a message queue and acknowledge them back by IDs. In contrast to MessageAcknowledgingSourceBase, this source handles two types of ids: 1) Session ids 2) Unique message ids Session ids are used to acknowledge messages in a session. When a checkpoint is restored, unacknowledged messages are redelivered. Duplicates are detected using the unique message ids which are checkpointed.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected List<SessionId>	sessionIds
protected Deque<org.apache.flink.api.java.tuple.Tuple2<Long,List<SessionId>>>	sessionIdsPerSnapshot

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MultipleIdsMessageAcknowledgingSourceBase(Class<UId> idClass) Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
protected	MultipleIdsMessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo) Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Method Summary

Modifier and Type	Method and Description
protected void	acknowledgeIDs(long checkpointId, List<UId> uniqueIds) Acknowledges the session ids.
protected abstract void	acknowledgeSessionIDs(List<SessionId> sessionIds) Acknowledges the session ids.
void	close()
void	open(org.apache.flink.configuration.Configuration parameters)
org.apache.flink.runtime.state.SerializedCheckpointData[]	snapshotState(long checkpointId, long checkpointTimestamp) Gets the current state of the function of operator.

Methods inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

addId, notifyCheckpointComplete, restoreState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

cancel, run

Field Detail

sessionIdsPerSnapshot

protected transient Deque<org.apache.flink.api.java.tuple.Tuple2<Long,List<SessionId>>> sessionIdsPerSnapshot

sessionIds

protected transient List<SessionId> sessionIds

Constructor Detail

MultipleIdsMessageAcknowledgingSourceBase

protected MultipleIdsMessageAcknowledgingSourceBase(Class<UId> idClass)

Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Parameters:

idClass - The class of the message ID type, used to create a serializer for the message IDs.

MultipleIdsMessageAcknowledgingSourceBase

protected MultipleIdsMessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)

Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Parameters:

idTypeInfo - The type information of the message ID type, used to create a serializer for the message IDs.

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class MessageAcknowledgingSourceBase<Type,UId>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class MessageAcknowledgingSourceBase<Type,UId>

Throws:

Exception

acknowledgeIDs

protected final void acknowledgeIDs(long checkpointId,
                                    List<UId> uniqueIds)

Acknowledges the session ids.

Specified by:

acknowledgeIDs in class MessageAcknowledgingSourceBase<Type,UId>

Parameters:

checkpointId - The id of the current checkout to acknowledge ids for.

uniqueIds - The checkpointed unique ids which are ignored here. They only serve as a means of de-duplicating messages when the acknowledgment after a checkpoint fails.

acknowledgeSessionIDs

protected abstract void acknowledgeSessionIDs(List<SessionId> sessionIds)

Acknowledges the session ids.

Parameters:

sessionIds - The message ids for this session.

snapshotState

public org.apache.flink.runtime.state.SerializedCheckpointData[] snapshotState(long checkpointId,
                                                                               long checkpointTimestamp)
                                                                        throws Exception

Description copied from interface: Checkpointed

Gets the current state of the function of operator. The state must reflect the result of all prior invocations to this function.

Specified by:

snapshotState in interface Checkpointed<org.apache.flink.runtime.state.SerializedCheckpointData[]>

Overrides:

snapshotState in class MessageAcknowledgingSourceBase<Type,UId>

Parameters:

checkpointId - The ID of the checkpoint.

checkpointTimestamp - The timestamp of the checkpoint, as derived by System.currentTimeMillis() on the JobManager.

Returns:

A snapshot of the operator state.

Throws:

Exception - Thrown if the creation of the state object failed. This causes the checkpoint to fail. The system may decide to fail the operation (and trigger recovery), or to discard this checkpoint attempt and to continue running and to try again with the next checkpoint attempt.