org.apache.flink.streaming.connectors.rabbitmq

Class RMQSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<Type>

org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase<Type,UId>

org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

org.apache.flink.streaming.connectors.rabbitmq.RMQSource<OUT>

Type Parameters:

OUT - The type of the data read from RabbitMQ.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class RMQSource<OUT>
extends org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>

RabbitMQ source (consumer) which reads from a queue and acknowledges messages on checkpoints. When checkpointing is enabled, it guarantees exactly-once processing semantics.

RabbitMQ requires messages to be acknowledged. On failures, RabbitMQ will re-resend all messages which have not been acknowledged previously. When a failure occurs directly after a completed checkpoint, all messages part of this checkpoint might be processed again because they couldn't be acknowledged before failure. This case is handled by the MessageAcknowledgingSourceBase base class which deduplicates the messages using the correlation id.

RabbitMQ's Delivery Tags do NOT represent unique ids / offsets. That's why the source uses the Correlation ID in the message properties to check for duplicate messages. Note that the correlation id has to be set at the producer. If the correlation id is not set, messages may be produced more than once in corner cases.

This source can be operated in three different modes:

1) Exactly-once (when checkpointed) with RabbitMQ transactions and messages with unique correlation IDs. 2) At-least-once (when checkpointed) with RabbitMQ transactions but no deduplication mechanism (correlation id is not set). 3) No strong delivery guarantees (without checkpointing) with RabbitMQ auto-commit mode.

Users may overwrite the setupConnectionFactory() method to pass their setup their own ConnectionFactory in case the constructor parameters are not sufficient.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	autoAck
protected com.rabbitmq.client.Channel	channel
protected com.rabbitmq.client.Connection	connection
protected com.rabbitmq.client.QueueingConsumer	consumer
protected String	queueName
protected org.apache.flink.api.common.serialization.DeserializationSchema<OUT>	schema

Fields inherited from class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

sessionIds, sessionIdsPerSnapshot

Fields inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

pendingCheckpoints

Constructor Summary

Constructors

Constructor and Description
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, boolean usesCorrelationId, org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema) Creates a new RabbitMQ source.
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema) Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled.

Method Summary

Modifier and Type	Method and Description
protected void	acknowledgeSessionIDs(List<Long> sessionIds)
void	cancel()
void	close()
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getProducedType()
void	open(org.apache.flink.configuration.Configuration config)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
protected com.rabbitmq.client.ConnectionFactory	setupConnectionFactory() Initializes the connection to RMQ with a default connection factory.
protected void	setupQueue() Sets up the queue.

Methods inherited from class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

acknowledgeIDs, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

addId, initializeState, notifyCheckpointComplete

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

queueName

protected final String queueName

schema

protected org.apache.flink.api.common.serialization.DeserializationSchema<OUT> schema

connection

protected transient com.rabbitmq.client.Connection connection

channel

protected transient com.rabbitmq.client.Channel channel

consumer

protected transient com.rabbitmq.client.QueueingConsumer consumer

autoAck

protected transient boolean autoAck

Constructor Detail

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)

Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled. No strong delivery guarantees when checkpointing is disabled.

For exactly-once, please use the constructor RMQSource(RMQConnectionConfig, String, boolean, DeserializationSchema).

Parameters:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 boolean usesCorrelationId,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)

Creates a new RabbitMQ source. For exactly-once, you must set the correlation ids of messages at the producer. The correlation id must be unique. Otherwise the behavior of the source is undefined. If in doubt, set usesCorrelationId to false. When correlation ids are not used, this source has at-least-once processing semantics when checkpointing is enabled.

Parameters:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

usesCorrelationId - Whether the messages received are supplied with a unique id to deduplicate messages (in case of failed acknowledgments). Only used when checkpointing is enabled.

deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

Method Detail

setupConnectionFactory

protected com.rabbitmq.client.ConnectionFactory setupConnectionFactory()
                                                                throws Exception

Initializes the connection to RMQ with a default connection factory. The user may override this method to setup and configure their own ConnectionFactory.

Throws:

Exception

setupQueue

protected void setupQueue()
                   throws IOException

Sets up the queue. The default implementation just declares the queue. The user may override this method to have a custom setup for the queue (i.e. binding the queue to an exchange or defining custom queue parameters)

Throws:

IOException

open

public void open(org.apache.flink.configuration.Configuration config)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

acknowledgeSessionIDs

protected void acknowledgeSessionIDs(List<Long> sessionIds)

Specified by:

acknowledgeSessionIDs in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>