Type - The type of the messages created by the source.UId - The type of unique IDs which may be used to acknowledge elements.@PublicEvolving public abstract class MessageAcknowledgingSourceBase<Type,UId> extends RichSourceFunction<Type> implements Checkpointed<org.apache.flink.runtime.state.SerializedCheckpointData[]>, org.apache.flink.runtime.state.CheckpointListener
The mechanism for this source assumes that messages are identified by a unique ID. When messages are taken from the message queue, the message must not be dropped immediately, but must be retained until acknowledged. Messages that are not acknowledged within a certain time interval will be served again (to a different connection, established by the recovered source).
Note that this source can give no guarantees about message order in the case of failures, because messages that were retrieved but not yet acknowledged will be returned later again, after a set of messages that was not retrieved before the failure.
Internally, this source gathers the IDs of elements it emits. Per checkpoint, the IDs are stored and acknowledged when the checkpoint is complete. That way, no message is acknowledged unless it is certain that it has been successfully processed throughout the topology and the updates to any state caused by that message are persistent.
All messages that are emitted and successfully processed by the streaming program will eventually be acknowledged. In corner cases, the source may receive certain IDs multiple times, if a failure occurs while acknowledging. To cope with this situation, an additional Set stores all processed IDs. IDs are only removed after they have been acknowledged.
A typical way to use this base in a source function is by implementing a run() method as follows:
public void run(SourceContext<Type> ctx) throws Exception {
while (running) {
Message msg = queue.retrieve();
synchronized (ctx.getCheckpointLock()) {
ctx.collect(msg.getMessageData());
addId(msg.getMessageId());
}
}
}
SourceFunction.SourceContext<T>| Modifier | Constructor and Description |
|---|---|
protected |
MessageAcknowledgingSourceBase(Class<UId> idClass)
Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
|
protected |
MessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)
Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
|
| Modifier and Type | Method and Description |
|---|---|
protected abstract void |
acknowledgeIDs(long checkpointId,
List<UId> UIds)
This method must be implemented to acknowledge the given set of IDs back to the message queue.
|
protected boolean |
addId(UId uid)
Adds an ID to be stored with the current checkpoint.
|
void |
close() |
void |
notifyCheckpointComplete(long checkpointId) |
void |
open(org.apache.flink.configuration.Configuration parameters) |
void |
restoreState(org.apache.flink.runtime.state.SerializedCheckpointData[] state)
Restores the state of the function or operator to that of a previous checkpoint.
|
org.apache.flink.runtime.state.SerializedCheckpointData[] |
snapshotState(long checkpointId,
long checkpointTimestamp)
Gets the current state of the function of operator.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContextclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitcancel, runprotected MessageAcknowledgingSourceBase(Class<UId> idClass)
idClass - The class of the message ID type, used to create a serializer for the message IDs.protected MessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)
idTypeInfo - The type information of the message ID type, used to create a serializer for the message IDs.public void open(org.apache.flink.configuration.Configuration parameters)
throws Exception
open in interface org.apache.flink.api.common.functions.RichFunctionopen in class org.apache.flink.api.common.functions.AbstractRichFunctionExceptionpublic void close()
throws Exception
close in interface org.apache.flink.api.common.functions.RichFunctionclose in class org.apache.flink.api.common.functions.AbstractRichFunctionExceptionprotected abstract void acknowledgeIDs(long checkpointId,
List<UId> UIds)
UIds - The list od IDs to acknowledge.protected boolean addId(UId uid)
uid - The ID to add.public org.apache.flink.runtime.state.SerializedCheckpointData[] snapshotState(long checkpointId,
long checkpointTimestamp)
throws Exception
CheckpointedsnapshotState in interface Checkpointed<org.apache.flink.runtime.state.SerializedCheckpointData[]>checkpointId - The ID of the checkpoint.checkpointTimestamp - The timestamp of the checkpoint, as derived by
System.currentTimeMillis() on the JobManager.Exception - Thrown if the creation of the state object failed. This causes the
checkpoint to fail. The system may decide to fail the operation (and trigger
recovery), or to discard this checkpoint attempt and to continue running
and to try again with the next checkpoint attempt.public void restoreState(org.apache.flink.runtime.state.SerializedCheckpointData[] state)
throws Exception
CheckpointedrestoreState in interface Checkpointed<org.apache.flink.runtime.state.SerializedCheckpointData[]>state - The state to be restored.ExceptionCopyright © 2014–2016 The Apache Software Foundation. All rights reserved.