org.apache.flink.streaming.connectors.rabbitmq

类 RMQSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<Type>

org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase<Type,UId>

org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

org.apache.flink.streaming.connectors.rabbitmq.RMQSource<OUT>

类型参数:

OUT - The type of the data read from RabbitMQ.

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class RMQSource<OUT>
extends org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>

RabbitMQ source (consumer) which reads from a queue and acknowledges messages on checkpoints. When checkpointing is enabled, it guarantees exactly-once processing semantics.

RabbitMQ requires messages to be acknowledged. On failures, RabbitMQ will re-resend all messages which have not been acknowledged previously. When a failure occurs directly after a completed checkpoint, all messages part of this checkpoint might be processed again because they couldn't be acknowledged before failure. This case is handled by the MessageAcknowledgingSourceBase base class which deduplicates the messages using the correlation id.

RabbitMQ's Delivery Tags do NOT represent unique ids / offsets. That's why the source uses the Correlation ID in the message properties to check for duplicate messages. Note that the correlation id has to be set at the producer. If the correlation id is not set, messages may be produced more than once in corner cases.

This source can be operated in three different modes:

1) Exactly-once (when checkpointed) with RabbitMQ transactions and messages with unique correlation IDs. 2) At-least-once (when checkpointed) with RabbitMQ transactions but no deduplication mechanism (correlation id is not set). 3) No strong delivery guarantees (without checkpointing) with RabbitMQ auto-commit mode.

Users may overwrite the setupConnectionFactory() method to pass their setup their own ConnectionFactory in case the constructor parameters are not sufficient.

另请参阅:

序列化表格

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

字段概要

字段

限定符和类型	字段和说明
protected boolean	autoAck
protected com.rabbitmq.client.Channel	channel
protected com.rabbitmq.client.Connection	connection
protected org.apache.flink.streaming.connectors.rabbitmq.QueueingConsumer	consumer
protected RMQDeserializationSchema<OUT>	deliveryDeserializer
protected String	queueName

从类继承的字段 org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

sessionIds, sessionIdsPerSnapshot

从类继承的字段 org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

pendingCheckpoints

构造器概要

构造器

构造器和说明
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, boolean usesCorrelationId, org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema) Creates a new RabbitMQ source.
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, boolean usesCorrelationId, RMQDeserializationSchema<OUT> deliveryDeserializer) Creates a new RabbitMQ source.
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema) Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled.
RMQSource(RMQConnectionConfig rmqConnectionConfig, String queueName, RMQDeserializationSchema<OUT> deliveryDeserializer) Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled.

方法概要

限定符和类型	方法和说明
protected void	acknowledgeSessionIDs(List<Long> sessionIds)
void	cancel()
void	close()
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getProducedType()
void	open(org.apache.flink.configuration.Configuration config)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
protected com.rabbitmq.client.Connection	setupConnection() Initializes the connection to RMQ using the default connection factory from setupConnectionFactory().
protected com.rabbitmq.client.ConnectionFactory	setupConnectionFactory() Initializes the connection to RMQ with a default connection factory.
protected void	setupQueue() Sets up the queue.

从类继承的方法 org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

acknowledgeIDs, snapshotState

从类继承的方法 org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

addId, initializeState, notifyCheckpointAborted, notifyCheckpointComplete

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

queueName

protected final String queueName

deliveryDeserializer

protected RMQDeserializationSchema<OUT> deliveryDeserializer

connection

protected transient com.rabbitmq.client.Connection connection

channel

protected transient com.rabbitmq.client.Channel channel

consumer

protected transient org.apache.flink.streaming.connectors.rabbitmq.QueueingConsumer consumer

autoAck

protected transient boolean autoAck

构造器详细资料

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)

Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled. No strong delivery guarantees when checkpointing is disabled.

For exactly-once, please use the constructor RMQSource(RMQConnectionConfig, String, boolean, DeserializationSchema).

参数:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 boolean usesCorrelationId,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)

Creates a new RabbitMQ source. For exactly-once, you must set the correlation ids of messages at the producer. The correlation id must be unique. Otherwise the behavior of the source is undefined. If in doubt, set usesCorrelationId to false. When correlation ids are not used, this source has at-least-once processing semantics when checkpointing is enabled.

参数:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

usesCorrelationId - Whether the messages received are supplied with a unique id to deduplicate messages (in case of failed acknowledgments). Only used when checkpointing is enabled.

deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 RMQDeserializationSchema<OUT> deliveryDeserializer)

Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled. No strong delivery guarantees when checkpointing is disabled.

For exactly-once, please use the constructor RMQSource(RMQConnectionConfig, String, boolean, RMQDeserializationSchema).

It also uses the provided RMQDeserializationSchema to parse both the correlationID and the message.

参数:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

deliveryDeserializer - A RMQDeserializationSchema for parsing the RMQDelivery.

RMQSource

public RMQSource(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 boolean usesCorrelationId,
                 RMQDeserializationSchema<OUT> deliveryDeserializer)

Creates a new RabbitMQ source. For exactly-once, you must set the correlation ids of messages at the producer. The correlation id must be unique. Otherwise the behavior of the source is undefined. If in doubt, set usesCorrelationId to false. When correlation ids are not used, this source has at-least-once processing semantics when checkpointing is enabled.

It also uses the provided RMQDeserializationSchema to parse both the correlationID and the message.

参数:

rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.

queueName - The queue to receive messages from.

usesCorrelationId - Whether the messages received are supplied with a unique id to deduplicate messages (in case of failed acknowledgments). Only used when checkpointing is enabled.

deliveryDeserializer - A RMQDeserializationSchema for parsing the RMQDelivery.

方法详细资料

setupConnectionFactory

protected com.rabbitmq.client.ConnectionFactory setupConnectionFactory()
                                                                throws Exception

Initializes the connection to RMQ with a default connection factory. The user may override this method to setup and configure their own ConnectionFactory.

抛出:

Exception

setupConnection

@VisibleForTesting
protected com.rabbitmq.client.Connection setupConnection()
                                                                     throws Exception

Initializes the connection to RMQ using the default connection factory from setupConnectionFactory(). The user may override this method to setup and configure their own Connection.

抛出:

Exception

setupQueue

@VisibleForTesting
protected void setupQueue()
                                      throws IOException

Sets up the queue. The default implementation just declares the queue. The user may override this method to have a custom setup for the queue (i.e. binding the queue to an exchange or defining custom queue parameters)

抛出:

IOException

open

public void open(org.apache.flink.configuration.Configuration config)
          throws Exception

指定者:

open 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

open 在类中 org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

抛出:

Exception

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.api.common.functions.RichFunction

覆盖:

close 在类中 org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

抛出:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
         throws Exception

指定者:

run 在接口中 org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

抛出:

Exception

cancel

public void cancel()

指定者:

cancel 在接口中 org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

acknowledgeSessionIDs

protected void acknowledgeSessionIDs(List<Long> sessionIds)

指定者:

acknowledgeSessionIDs 在类中 org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,String,Long>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()

指定者:

getProducedType 在接口中 org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>