org.apache.flink.streaming.connectors.pulsar

Class FlinkPulsarSink<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSink<T>

Type Parameters:

T - Type of the record.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

public class FlinkPulsarSink<T>
extends org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Write data to Flink.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlinkPulsarSinkBase.PulsarTransactionState<T> State for handling transactions.
static class	FlinkPulsarSinkBase.TransactionStateSerializer<T> TypeSerializer for FlinkPulsarSinkBase.PulsarTransactionState.

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializer<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializerConfigSnapshot<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.TransactionHolder<TXN>

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.pulsar.client.admin.PulsarAdmin	admin
protected String	adminUrl
protected Map<String,String>	caseInsensitiveParams
protected org.apache.pulsar.client.impl.conf.ClientConfigurationData	clientConfigurationData
protected String	defaultTopic
protected Throwable	failedWrite
protected boolean	failOnWrite
protected boolean	flushOnCheckpoint
protected boolean	forcedTopic
protected long	maxBlockTimeMs
protected org.apache.pulsar.client.api.MessageRouter	messageRouter
protected long	pendingRecords Number of unacknowledged records.
protected org.apache.flink.util.SerializableObject	pendingRecordsLock Lock for accessing the pending records.
protected Map<String,Object>	producerConf
protected Properties	properties
protected PulsarSinkSemantic	semantic Semantic chosen for this instance.
protected BiConsumer<org.apache.pulsar.client.api.MessageId,Throwable>	sendCallback
protected int	sendTimeOutMs
protected org.apache.pulsar.client.api.Producer<T>	singleProducer
protected ConcurrentHashMap<org.apache.pulsar.client.api.transaction.TxnID,List<CompletableFuture<org.apache.pulsar.client.api.MessageId>>>	tid2FuturesMap
protected ConcurrentHashMap<org.apache.pulsar.client.api.transaction.TxnID,List<org.apache.pulsar.client.api.MessageId>>	tid2MessagesMap
protected Map<String,org.apache.pulsar.client.api.Producer<T>>	topic2Producer
protected long	transactionTimeout

Fields inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

pendingCommitTransactions, state, userContext

Constructor Summary

Constructors

Constructor and Description
FlinkPulsarSink(String adminUrl, Optional<String> defaultTopicName, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Properties properties, PulsarSerializationSchema serializationSchema)
FlinkPulsarSink(String adminUrl, Optional<String> defaultTopicName, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Properties properties, PulsarSerializationSchema serializationSchema, org.apache.pulsar.client.api.MessageRouter messageRouter, PulsarSinkSemantic semantic)
FlinkPulsarSink(String adminUrl, Optional<String> defaultTopicName, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Properties properties, PulsarSerializationSchema serializationSchema, PulsarSinkSemantic semantic)
FlinkPulsarSink(String serviceUrl, String adminUrl, Optional<String> defaultTopicName, Properties properties, PulsarSerializationSchema<T> serializationSchema)

Method Summary

Modifier and Type	Method and Description
protected void	abort(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState)
protected FlinkPulsarSinkBase.PulsarTransactionState<T>	beginTransaction()
protected void	checkErroneous()
void	close()
protected void	commit(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState)
protected org.apache.pulsar.client.api.Producer<T>	createProducer(org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, Map<String,Object> producerConf, String topic, org.apache.pulsar.client.api.Schema<T> schema)
protected org.apache.pulsar.client.api.Producer<T>	getProducer(String topic)
protected void	initializeSendCallback()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	invoke(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState, T value, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
void	open(org.apache.flink.configuration.Configuration parameters)
protected void	preCommit(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
protected void	producerClose()
void	producerFlush(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
protected void	recoverAndAbort(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
protected void	recoverAndCommit(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

currentTransaction, enableTransactionTimeoutWarnings, finishRecoveringContext, getUserContext, ignoreFailuresAfterTransactionTimeout, initializeUserContext, invoke, invoke, notifyCheckpointAborted, notifyCheckpointComplete, pendingTransactions, setTransactionTimeout

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

adminUrl

protected String adminUrl

clientConfigurationData

protected org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConfigurationData

caseInsensitiveParams

protected final Map<String,String> caseInsensitiveParams

producerConf

protected final Map<String,Object> producerConf

properties

protected final Properties properties

flushOnCheckpoint

protected boolean flushOnCheckpoint

failOnWrite

protected boolean failOnWrite

transactionTimeout

protected long transactionTimeout

maxBlockTimeMs

protected long maxBlockTimeMs

sendTimeOutMs

protected int sendTimeOutMs

pendingRecordsLock

protected final org.apache.flink.util.SerializableObject pendingRecordsLock

Lock for accessing the pending records.

pendingRecords

protected long pendingRecords

Number of unacknowledged records.

tid2MessagesMap

protected ConcurrentHashMap<org.apache.pulsar.client.api.transaction.TxnID,List<org.apache.pulsar.client.api.MessageId>> tid2MessagesMap

tid2FuturesMap

protected ConcurrentHashMap<org.apache.pulsar.client.api.transaction.TxnID,List<CompletableFuture<org.apache.pulsar.client.api.MessageId>>> tid2FuturesMap

forcedTopic

protected final boolean forcedTopic

defaultTopic

protected final String defaultTopic

messageRouter

protected final org.apache.pulsar.client.api.MessageRouter messageRouter

failedWrite

protected transient volatile Throwable failedWrite

admin

protected transient org.apache.pulsar.client.admin.PulsarAdmin admin

sendCallback

protected transient BiConsumer<org.apache.pulsar.client.api.MessageId,Throwable> sendCallback

semantic

protected PulsarSinkSemantic semantic

Semantic chosen for this instance.

singleProducer

protected transient org.apache.pulsar.client.api.Producer<T> singleProducer

topic2Producer

protected transient Map<String,org.apache.pulsar.client.api.Producer<T>> topic2Producer

Constructor Detail

FlinkPulsarSink

public FlinkPulsarSink(String adminUrl,
                       Optional<String> defaultTopicName,
                       org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                       Properties properties,
                       PulsarSerializationSchema serializationSchema,
                       org.apache.pulsar.client.api.MessageRouter messageRouter,
                       PulsarSinkSemantic semantic)

FlinkPulsarSink

public FlinkPulsarSink(String adminUrl,
                       Optional<String> defaultTopicName,
                       org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                       Properties properties,
                       PulsarSerializationSchema serializationSchema,
                       PulsarSinkSemantic semantic)

FlinkPulsarSink

public FlinkPulsarSink(String adminUrl,
                       Optional<String> defaultTopicName,
                       org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                       Properties properties,
                       PulsarSerializationSchema serializationSchema)

FlinkPulsarSink

public FlinkPulsarSink(String serviceUrl,
                       String adminUrl,
                       Optional<String> defaultTopicName,
                       Properties properties,
                       PulsarSerializationSchema<T> serializationSchema)

Method Detail

invoke

public void invoke(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState,
                   T value,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws Exception

Specified by:

invoke in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Overrides:

snapshotState in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Overrides:

initializeState in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

initializeSendCallback

protected void initializeSendCallback()

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

getProducer

protected org.apache.pulsar.client.api.Producer<T> getProducer(String topic)

createProducer

protected org.apache.pulsar.client.api.Producer<T> createProducer(org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                                                                  Map<String,Object> producerConf,
                                                                  String topic,
                                                                  org.apache.pulsar.client.api.Schema<T> schema)

producerFlush

public void producerFlush(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
                   throws Exception

Throws:

Exception

beginTransaction

protected FlinkPulsarSinkBase.PulsarTransactionState<T> beginTransaction()
                                                                  throws Exception

Specified by:

beginTransaction in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

preCommit

protected void preCommit(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)
                  throws Exception

Specified by:

preCommit in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

Throws:

Exception

commit

protected void commit(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState)

Specified by:

commit in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

abort

protected void abort(FlinkPulsarSinkBase.PulsarTransactionState<T> transactionState)

Specified by:

abort in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

recoverAndCommit

protected void recoverAndCommit(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)

Overrides:

recoverAndCommit in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

recoverAndAbort

protected void recoverAndAbort(FlinkPulsarSinkBase.PulsarTransactionState<T> transaction)

Overrides:

recoverAndAbort in class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<T,FlinkPulsarSinkBase.PulsarTransactionState<T>,Void>

producerClose

protected void producerClose()
                      throws Exception

Throws:

Exception

checkErroneous

protected void checkErroneous()
                       throws Exception

Throws:

Exception