org.apache.flink.streaming.connectors.pulsar.internal

Class PulsarFetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.pulsar.internal.PulsarFetcher<T>

Type Parameters:

T - The type of elements deserialized from Pulsar messages, and emitted into the Flink data stream.

public class PulsarFetcher<T>
extends Object

Implements the logic around emitting records and tracking offsets, as well as around the optional timestamp assignment and watermark generation.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.pulsar.client.impl.conf.ClientConfigurationData	clientConf
protected PulsarDeserializationSchema<T>	deserializer
protected Set<TopicRange>	excludeStartMessageIds
protected PulsarMetadataReader	metadataReader
protected int	pollTimeoutMs
protected Map<String,Object>	readerConf
protected Map<TopicRange,org.apache.pulsar.client.api.MessageId>	seedTopicsWithInitialOffsets
protected org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>	sourceContext The source context to emit records and watermarks to.
protected List<PulsarTopicState<T>>	subscribedPartitionStates All partitions (and their state) that this fetcher is subscribed to.
protected ClosableBlockingQueue<PulsarTopicState<T>>	unassignedPartitionsQueue Queue of partitions that are not yet assigned to any reader thread for consuming.
protected org.apache.flink.api.common.eventtime.WatermarkOutput	watermarkOutput Wrapper around our SourceContext for allowing the WatermarkGenerator to emit watermarks and mark idleness.

Constructor Summary

Constructors

Constructor and Description
PulsarFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Map<String,Object> readerConf, int pollTimeoutMs, PulsarDeserializationSchema<T> deserializer, PulsarMetadataReader metadataReader, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)
PulsarFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets, Set<TopicRange> excludeStartMessageIds, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Map<String,Object> readerConf, int pollTimeoutMs, int commitMaxRetries, PulsarDeserializationSchema<T> deserializer, PulsarMetadataReader metadataReader, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
void	addDiscoveredTopics(Set<TopicRange> newTopics)
void	cancel()
void	commitOffsetToPulsar(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset, PulsarCommitCallback offsetCommitCallback)
Map<TopicRange,ReaderThread<T>>	createAndStartReaderThread(List<PulsarTopicState<T>> states, ExceptionProxy exceptionProxy)
protected ReaderThread<T>	createReaderThread(ExceptionProxy exceptionProxy, PulsarTopicState state)
protected void	doCommitOffsetToPulsar(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset, PulsarCommitCallback offsetCommitCallback)
protected void	emitRecordsWithTimestamps(T record, PulsarTopicState<T> partitionState, org.apache.pulsar.client.api.MessageId offset, long pulsarEventTimestamp) Emits a record attaching a timestamp to it.
PulsarMetadataReader	getMetaDataReader()
protected List<PulsarTopicState<T>>	getSubscribedTopicStates()
Map<TopicRange,org.apache.pulsar.client.api.MessageId>	removeEarliestAndLatest(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)
void	runFetchLoop()
Map<TopicRange,org.apache.pulsar.client.api.MessageId>	snapshotCurrentState() Takes a snapshot of the partition offsets.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sourceContext

protected final org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext

The source context to emit records and watermarks to.

seedTopicsWithInitialOffsets

protected final Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets

excludeStartMessageIds

protected final Set<TopicRange> excludeStartMessageIds

subscribedPartitionStates

protected final List<PulsarTopicState<T>> subscribedPartitionStates

All partitions (and their state) that this fetcher is subscribed to.

unassignedPartitionsQueue

protected final ClosableBlockingQueue<PulsarTopicState<T>> unassignedPartitionsQueue

Queue of partitions that are not yet assigned to any reader thread for consuming.

All partitions added to this queue are guaranteed to have been added to subscribedPartitionStates already.

clientConf

protected final org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf

readerConf

protected final Map<String,Object> readerConf

deserializer

protected final PulsarDeserializationSchema<T> deserializer

pollTimeoutMs

protected final int pollTimeoutMs

metadataReader

protected final PulsarMetadataReader metadataReader

watermarkOutput

protected final org.apache.flink.api.common.eventtime.WatermarkOutput watermarkOutput

Wrapper around our SourceContext for allowing the WatermarkGenerator to emit watermarks and mark idleness.

Constructor Detail

PulsarFetcher

public PulsarFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                     Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets,
                     org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                     org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                     long autoWatermarkInterval,
                     ClassLoader userCodeClassLoader,
                     org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext,
                     org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                     Map<String,Object> readerConf,
                     int pollTimeoutMs,
                     PulsarDeserializationSchema<T> deserializer,
                     PulsarMetadataReader metadataReader,
                     org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                     boolean useMetrics)
              throws Exception

Throws:

Exception

PulsarFetcher

public PulsarFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                     Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets,
                     Set<TopicRange> excludeStartMessageIds,
                     org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                     org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                     long autoWatermarkInterval,
                     ClassLoader userCodeClassLoader,
                     org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext,
                     org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                     Map<String,Object> readerConf,
                     int pollTimeoutMs,
                     int commitMaxRetries,
                     PulsarDeserializationSchema<T> deserializer,
                     PulsarMetadataReader metadataReader,
                     org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                     boolean useMetrics)
              throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Throws:

Exception

emitRecordsWithTimestamps

protected void emitRecordsWithTimestamps(T record,
                                         PulsarTopicState<T> partitionState,
                                         org.apache.pulsar.client.api.MessageId offset,
                                         long pulsarEventTimestamp)

Emits a record attaching a timestamp to it.

Parameters:

record - The record to emit

partitionState - The state of the pulsar partition from which the record was fetched

offset - The offset of the corresponding pulsar record

pulsarEventTimestamp - The timestamp of the pulsar record

cancel

public void cancel()
            throws Exception

Throws:

Exception

commitOffsetToPulsar

public void commitOffsetToPulsar(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset,
                                 PulsarCommitCallback offsetCommitCallback)
                          throws InterruptedException

Throws:

InterruptedException

doCommitOffsetToPulsar

protected void doCommitOffsetToPulsar(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset,
                                      PulsarCommitCallback offsetCommitCallback)
                               throws InterruptedException

Throws:

InterruptedException

removeEarliestAndLatest

public Map<TopicRange,org.apache.pulsar.client.api.MessageId> removeEarliestAndLatest(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)

addDiscoveredTopics

public void addDiscoveredTopics(Set<TopicRange> newTopics)
                         throws IOException,
                                ClassNotFoundException

Throws:

IOException

ClassNotFoundException

snapshotCurrentState

public Map<TopicRange,org.apache.pulsar.client.api.MessageId> snapshotCurrentState()

Takes a snapshot of the partition offsets.

Important: This method must be called under the checkpoint lock.

Returns:

A map from partition to current offset.

createAndStartReaderThread

public Map<TopicRange,ReaderThread<T>> createAndStartReaderThread(List<PulsarTopicState<T>> states,
                                                                  ExceptionProxy exceptionProxy)

getSubscribedTopicStates

protected List<PulsarTopicState<T>> getSubscribedTopicStates()

createReaderThread

protected ReaderThread<T> createReaderThread(ExceptionProxy exceptionProxy,
                                             PulsarTopicState state)

getMetaDataReader

public PulsarMetadataReader getMetaDataReader()