org.apache.flink.streaming.connectors.pulsar

Class FlinkPulsarSource<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSource<T>

Type Parameters:

T - The type of records produced by this data source.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>

public class FlinkPulsarSource<T>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Pulsar data source.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected String	adminUrl
protected Map<String,String>	caseInsensitiveParams
protected org.apache.pulsar.client.impl.conf.ClientConfigurationData	clientConfigurationData
protected int	commitMaxRetries
protected PulsarDeserializationSchema<T>	deserializer
static String	KEY_DISABLE_METRICS Boolean configuration key to disable metrics tracking.
static int	MAX_NUM_PENDING_CHECKPOINTS The maximum number of pending non-committed checkpoints to track, to avoid memory leaks.
protected PulsarMetadataReader	metadataReader The partition discoverer, used to find new partitions.
protected int	pollTimeoutMs
protected Properties	properties
protected Map<String,Object>	readerConf
protected UUID	uuid

Constructor Summary

Constructors

Constructor and Description
FlinkPulsarSource(String adminUrl, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, PulsarDeserializationSchema<T> deserializer, Properties properties)
FlinkPulsarSource(String serviceUrl, String adminUrl, org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializer, Properties properties)
FlinkPulsarSource(String serviceUrl, String adminUrl, PulsarDeserializationSchema<T> deserializer, Properties properties)

Method Summary

Modifier and Type	Method and Description
FlinkPulsarSource<T>	assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T> assigner) Deprecated.
FlinkPulsarSource<T>	assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T> assigner) Deprecated.
FlinkPulsarSource<T>	assignTimestampsAndWatermarks(org.apache.flink.api.common.eventtime.WatermarkStrategy<T> watermarkStrategy) Sets the given WatermarkStrategy on this consumer.
void	cancel()
void	close()
protected PulsarFetcher<T>	createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, org.apache.flink.streaming.api.operators.StreamingRuntimeContext streamingRuntime, boolean useMetrics, Set<TopicRange> excludeStartMessageIds)
protected PulsarMetadataReader	createMetadataReader()
Map<TopicRange,org.apache.pulsar.client.api.MessageId>	getOwnedTopicStarts()
Map<Long,Map<TopicRange,org.apache.pulsar.client.api.MessageId>>	getPendingOffsetsToCommit()
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
protected String	getSubscriptionName()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	joinDiscoveryLoopThread()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
Map<TopicRange,org.apache.pulsar.client.api.MessageId>	offsetForEachTopic(Set<TopicRange> topics, StartupMode mode, Map<TopicRange,org.apache.pulsar.client.api.MessageId> specificStartupOffsets)
void	open(org.apache.flink.configuration.Configuration parameters)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> ctx)
void	runWithTopicsDiscovery()
FlinkPulsarSource<T>	setStartFromEarliest()
FlinkPulsarSource<T>	setStartFromLatest()
FlinkPulsarSource<T>	setStartFromSpecificOffsets(Map<String,org.apache.pulsar.client.api.MessageId> specificStartupOffsets)
FlinkPulsarSource<T>	setStartFromSubscription(String externalSubscriptionName)
FlinkPulsarSource<T>	setStartFromSubscription(String externalSubscriptionName, org.apache.pulsar.client.api.MessageId subscriptionPosition)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_NUM_PENDING_CHECKPOINTS

public static final int MAX_NUM_PENDING_CHECKPOINTS

The maximum number of pending non-committed checkpoints to track, to avoid memory leaks.

See Also:

Constant Field Values

KEY_DISABLE_METRICS

public static final String KEY_DISABLE_METRICS

Boolean configuration key to disable metrics tracking.

See Also:

Constant Field Values

adminUrl

protected String adminUrl

clientConfigurationData

protected org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConfigurationData

caseInsensitiveParams

protected final Map<String,String> caseInsensitiveParams

readerConf

protected final Map<String,Object> readerConf

deserializer

protected volatile PulsarDeserializationSchema<T> deserializer

pollTimeoutMs

protected final int pollTimeoutMs

commitMaxRetries

protected final int commitMaxRetries

properties

protected final Properties properties

uuid

protected final UUID uuid

metadataReader

protected transient volatile PulsarMetadataReader metadataReader

The partition discoverer, used to find new partitions.

Constructor Detail

FlinkPulsarSource

public FlinkPulsarSource(String adminUrl,
                         org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                         PulsarDeserializationSchema<T> deserializer,
                         Properties properties)

FlinkPulsarSource

public FlinkPulsarSource(String serviceUrl,
                         String adminUrl,
                         PulsarDeserializationSchema<T> deserializer,
                         Properties properties)

FlinkPulsarSource

public FlinkPulsarSource(String serviceUrl,
                         String adminUrl,
                         org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializer,
                         Properties properties)

Method Detail

assignTimestampsAndWatermarks

@Deprecated
public FlinkPulsarSource<T> assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T> assigner)

Deprecated.

Specifies an AssignerWithPunctuatedWatermarks to emit watermarks in a punctuated manner. The watermark extractor will run per Pulsar partition, watermarks will be merged across partitions in the same way as in the Flink runtime, when streams are merged.

When a subtask of a FlinkPulsarSource source reads multiple Pulsar partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Pulsar partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more that one partition.

Running timestamp extractors / watermark generators directly inside the Pulsar source, per Pulsar partition, allows users to let them exploit the per-partition characteristics.

Note: One can use either an AssignerWithPunctuatedWatermarks or an AssignerWithPeriodicWatermarks, not both at the same time.

Parameters:

assigner - The timestamp assigner / watermark generator to use.

Returns:

The reader object, to allow function chaining.

assignTimestampsAndWatermarks

@Deprecated
public FlinkPulsarSource<T> assignTimestampsAndWatermarks(org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T> assigner)

Deprecated.

Specifies an AssignerWithPunctuatedWatermarks to emit watermarks in a punctuated manner. The watermark extractor will run per Pulsar partition, watermarks will be merged across partitions in the same way as in the Flink runtime, when streams are merged.

When a subtask of a FlinkPulsarSource source reads multiple Pulsar partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Pulsar partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more that one partition.

Running timestamp extractors / watermark generators directly inside the Pulsar source, per Pulsar partition, allows users to let them exploit the per-partition characteristics.

Note: One can use either an AssignerWithPunctuatedWatermarks or an AssignerWithPeriodicWatermarks, not both at the same time.

Parameters:

assigner - The timestamp assigner / watermark generator to use.

Returns:

The reader object, to allow function chaining.

assignTimestampsAndWatermarks

public FlinkPulsarSource<T> assignTimestampsAndWatermarks(org.apache.flink.api.common.eventtime.WatermarkStrategy<T> watermarkStrategy)

Sets the given WatermarkStrategy on this consumer. These will be used to assign timestamps to records and generates watermarks to signal event time progress.

Running timestamp extractors / watermark generators directly inside the Pulsar source (which you can do by using this method), per Pulsar partition, allows users to let them exploit the per-partition characteristics.

When a subtask of a FlinkPulsarSource reads multiple pulsar partitions, the streams from the partitions are unioned in a "first come first serve" fashion. Per-partition characteristics are usually lost that way. For example, if the timestamps are strictly ascending per Pulsar partition, they will not be strictly ascending in the resulting Flink DataStream, if the parallel source subtask reads more than one partition.

Common watermark generation patterns can be found as static methods in the WatermarkStrategy class.

Returns:

The consumer object, to allow function chaining.

setStartFromEarliest

public FlinkPulsarSource<T> setStartFromEarliest()

setStartFromLatest

public FlinkPulsarSource<T> setStartFromLatest()

setStartFromSpecificOffsets

public FlinkPulsarSource<T> setStartFromSpecificOffsets(Map<String,org.apache.pulsar.client.api.MessageId> specificStartupOffsets)

setStartFromSubscription

public FlinkPulsarSource<T> setStartFromSubscription(String externalSubscriptionName)

setStartFromSubscription

public FlinkPulsarSource<T> setStartFromSubscription(String externalSubscriptionName,
                                                     org.apache.pulsar.client.api.MessageId subscriptionPosition)

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

getSubscriptionName

protected String getSubscriptionName()

createMetadataReader

protected PulsarMetadataReader createMetadataReader()
                                             throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> ctx)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<T>

Throws:

Exception

createFetcher

protected PulsarFetcher<T> createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                                         Map<TopicRange,org.apache.pulsar.client.api.MessageId> seedTopicsWithInitialOffsets,
                                         org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                                         org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                                         long autoWatermarkInterval,
                                         ClassLoader userCodeClassLoader,
                                         org.apache.flink.streaming.api.operators.StreamingRuntimeContext streamingRuntime,
                                         boolean useMetrics,
                                         Set<TopicRange> excludeStartMessageIds)
                                  throws Exception

Throws:

Exception

joinDiscoveryLoopThread

public void joinDiscoveryLoopThread()
                             throws InterruptedException

Throws:

InterruptedException

runWithTopicsDiscovery

public void runWithTopicsDiscovery()
                            throws Exception

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<T>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)
                             throws Exception

Specified by:

notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

Throws:

Exception

offsetForEachTopic

public Map<TopicRange,org.apache.pulsar.client.api.MessageId> offsetForEachTopic(Set<TopicRange> topics,
                                                                                 StartupMode mode,
                                                                                 Map<TopicRange,org.apache.pulsar.client.api.MessageId> specificStartupOffsets)

getPendingOffsetsToCommit

public Map<Long,Map<TopicRange,org.apache.pulsar.client.api.MessageId>> getPendingOffsetsToCommit()

getOwnedTopicStarts

public Map<TopicRange,org.apache.pulsar.client.api.MessageId> getOwnedTopicStarts()