org.apache.flink.connector.pulsar.source.config

Class SourceConfiguration

java.lang.Object

org.apache.flink.api.common.ExecutionConfig.GlobalJobParameters

org.apache.flink.configuration.Configuration

org.apache.flink.configuration.UnmodifiableConfiguration

org.apache.flink.connector.pulsar.common.config.PulsarConfiguration

org.apache.flink.connector.pulsar.source.config.SourceConfiguration

All Implemented Interfaces:

Serializable, Cloneable, org.apache.flink.configuration.ReadableConfig, org.apache.flink.configuration.WritableConfig, org.apache.flink.core.io.IOReadableWritable

@PublicEvolving
public class SourceConfiguration
extends PulsarConfiguration

The configure class for pulsar source.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.configuration.Configuration

confData

Constructor Summary

Constructors

Constructor and Description
SourceConfiguration(org.apache.flink.configuration.Configuration configuration)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object o)
long	getAutoCommitCursorInterval() The interval in millis for acknowledge message when you enable isEnableAutoAcknowledgeMessage() and use SubscriptionType.Failover or SubscriptionType.Exclusive as your consuming subscription type.
int	getMaxFetchRecords() The fetch counts for a split reader.
java.time.Duration	getMaxFetchTime() The fetch time for flink split reader polling message.
int	getMessageQueueCapacity() The capacity of the element queue in the source reader.
long	getPartitionDiscoveryIntervalMs() The interval in millis for flink querying topic partition information.
String	getSubscriptionDesc() Convert the subscription into a readable str.
org.apache.pulsar.client.api.SubscriptionMode	getSubscriptionMode() The pulsar's subscription mode for this flink source.
String	getSubscriptionName() The pulsar's subscription name for this flink source.
org.apache.pulsar.client.api.SubscriptionType	getSubscriptionType() The pulsar's subscription type for this flink source.
long	getTransactionTimeoutMillis() Pulsar's transaction have a timeout mechanism for uncommitted transaction.
CursorVerification	getVerifyInitialOffsets() Validate the CursorPosition generated by StartCursor.
int	hashCode()
boolean	isEnableAutoAcknowledgeMessage() This is used for all subscription type.
boolean	isEnablePartitionDiscovery()
boolean	isEnableSchemaEvolution() If we should deserialize the message with a specified Pulsar Schema instead the default Schema.BYTES.

Methods inherited from class org.apache.flink.connector.pulsar.common.config.PulsarConfiguration

get, getProperties, useOption, useOption

Methods inherited from class org.apache.flink.configuration.UnmodifiableConfiguration

addAll, addAll, addAllToProperties, removeConfig

Methods inherited from class org.apache.flink.configuration.Configuration

clone, contains, containsKey, fromMap, get, getBoolean, getBoolean, getBoolean, getBytes, getClass, getDouble, getDouble, getDouble, getEnum, getFloat, getFloat, getFloat, getInteger, getInteger, getInteger, getLong, getLong, getLong, getOptional, getString, getString, getString, getValue, keySet, read, set, setBoolean, setBoolean, setBytes, setClass, setDouble, setDouble, setFloat, setFloat, setInteger, setInteger, setLong, setLong, setString, setString, toMap, toString, write

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

SourceConfiguration

public SourceConfiguration(org.apache.flink.configuration.Configuration configuration)

Method Detail

getMessageQueueCapacity

public int getMessageQueueCapacity()

The capacity of the element queue in the source reader.

isEnablePartitionDiscovery

public boolean isEnablePartitionDiscovery()

getPartitionDiscoveryIntervalMs

public long getPartitionDiscoveryIntervalMs()

The interval in millis for flink querying topic partition information.

isEnableAutoAcknowledgeMessage

public boolean isEnableAutoAcknowledgeMessage()

This is used for all subscription type. But the behavior may not be the same among them. If you don't enable the flink checkpoint, make sure this option is set to true.

• SubscriptionType.Shared and SubscriptionType.Key_Shared would immediately acknowledge the message after consuming it.
• SubscriptionType.Failover and SubscriptionType.Exclusive would perform a incremental acknowledge in a fixed getAutoCommitCursorInterval().

isEnableSchemaEvolution

public boolean isEnableSchemaEvolution()

If we should deserialize the message with a specified Pulsar Schema instead the default Schema.BYTES. This switch is only used for PulsarSchemaWrapper.

getAutoCommitCursorInterval

public long getAutoCommitCursorInterval()

The interval in millis for acknowledge message when you enable isEnableAutoAcknowledgeMessage() and use SubscriptionType.Failover or SubscriptionType.Exclusive as your consuming subscription type.

getTransactionTimeoutMillis

public long getTransactionTimeoutMillis()

Pulsar's transaction have a timeout mechanism for uncommitted transaction. We use transaction for SubscriptionType.Shared and SubscriptionType.Key_Shared when user disable isEnableAutoAcknowledgeMessage() and enable flink checkpoint. Since the checkpoint interval couldn't be acquired from SourceReaderContext.getConfiguration(), we have to expose this option. Make sure this value is greater than the checkpoint interval.

getMaxFetchTime

public java.time.Duration getMaxFetchTime()

The fetch time for flink split reader polling message. We would stop polling message and return the message in RecordsWithSplitIds when timeout or exceed the getMaxFetchRecords().

getMaxFetchRecords

public int getMaxFetchRecords()

The fetch counts for a split reader. We would stop polling message and return the message in RecordsWithSplitIds when timeout getMaxFetchTime() or exceed this value.

getVerifyInitialOffsets

public CursorVerification getVerifyInitialOffsets()

Validate the CursorPosition generated by StartCursor.

getSubscriptionName

public String getSubscriptionName()

The pulsar's subscription name for this flink source. All the readers would share this subscription name.

See Also:

ConsumerBuilder.subscriptionName(java.lang.String)

getSubscriptionType

public org.apache.pulsar.client.api.SubscriptionType getSubscriptionType()

The pulsar's subscription type for this flink source. All the readers would share this subscription type.

See Also:

SubscriptionType

getSubscriptionMode

public org.apache.pulsar.client.api.SubscriptionMode getSubscriptionMode()

The pulsar's subscription mode for this flink source. All the readers would share this subscription mode.

See Also:

SubscriptionMode

getSubscriptionDesc

public String getSubscriptionDesc()

Convert the subscription into a readable str.

equals

public boolean equals(Object o)

Overrides:

equals in class org.apache.flink.configuration.Configuration

hashCode

public int hashCode()

Overrides:

hashCode in class org.apache.flink.configuration.Configuration