org.apache.flink.streaming.connectors.pulsar.internal

Class PulsarMetadataReader

java.lang.Object

org.apache.flink.streaming.connectors.pulsar.internal.PulsarMetadataReader

All Implemented Interfaces:

AutoCloseable

public class PulsarMetadataReader
extends Object
implements AutoCloseable

A Helper class that talks to Pulsar Admin API. - getEarliest / Latest / Specific MessageIds - guarantee message existence using subscription by setup, move and remove

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PulsarMetadataReader.ClosedException Designate the close of the metadata reader.

Constructor Summary

Constructors

Constructor and Description
PulsarMetadataReader(String adminUrl, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, String subscriptionName, Map<String,String> caseInsensitiveParams, int indexOfThisSubtask, int numParallelSubtasks)
PulsarMetadataReader(String adminUrl, org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf, String subscriptionName, Map<String,String> caseInsensitiveParams, int indexOfThisSubtask, int numParallelSubtasks, boolean useExternalSubscription)

Method Summary

Modifier and Type	Method and Description
boolean	checkCursorAvailable(String topic, org.apache.pulsar.client.impl.MessageIdImpl startMessageId)
void	close()
void	commitOffsetToCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)
void	createNamespace(String ns)
void	createNamespace(String ns, boolean retain)
void	createTenant(String tenant)
void	createTopic(String topicName, int partitionNum)
void	deleteNamespace(String ns)
void	deleteSchema(String topicName)
void	deleteTopic(String topicName)
Set<TopicRange>	discoverTopicChanges()
org.apache.pulsar.client.api.MessageId	getLastMessageId(String topic)
org.apache.pulsar.client.api.MessageId	getPositionFromSubscription(TopicRange topic, org.apache.pulsar.client.api.MessageId defaultPosition)
org.apache.pulsar.common.schema.SchemaInfo	getPulsarSchema(List<String> topics)
org.apache.pulsar.common.schema.SchemaInfo	getPulsarSchema(String topic)
SerializableRange	getRange()
Set<TopicRange>	getTopicPartitionRanges() Get all TopicRange that should be consumed by the subTask.
Set<String>	getTopicPartitions() Get topic partitions all.
List<String>	getTopics(String ns)
List<String>	listNamespaces()
boolean	namespaceExists(String ns)
void	removeCursor(Set<TopicRange> topics)
void	resetCursor(TopicRange topicRange, org.apache.pulsar.client.api.MessageId messageId)
void	setupCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)
void	setupCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset, boolean failOnDataLoss)
boolean	tenantExists(String tenant)
boolean	topicExists(String topicName)
void	uploadSchema(String topicName, org.apache.pulsar.common.schema.SchemaInfo schemaInfo)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PulsarMetadataReader

public PulsarMetadataReader(String adminUrl,
                            org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                            String subscriptionName,
                            Map<String,String> caseInsensitiveParams,
                            int indexOfThisSubtask,
                            int numParallelSubtasks,
                            boolean useExternalSubscription)
                     throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

PulsarMetadataReader

public PulsarMetadataReader(String adminUrl,
                            org.apache.pulsar.client.impl.conf.ClientConfigurationData clientConf,
                            String subscriptionName,
                            Map<String,String> caseInsensitiveParams,
                            int indexOfThisSubtask,
                            int numParallelSubtasks)
                     throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

Method Detail

close

public void close()

Specified by:

close in interface AutoCloseable

discoverTopicChanges

public Set<TopicRange> discoverTopicChanges()
                                     throws org.apache.pulsar.client.admin.PulsarAdminException,
                                            PulsarMetadataReader.ClosedException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

PulsarMetadataReader.ClosedException

createTenant

public void createTenant(String tenant)
                  throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

tenantExists

public boolean tenantExists(String tenant)
                     throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

listNamespaces

public List<String> listNamespaces()
                            throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

namespaceExists

public boolean namespaceExists(String ns)
                        throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

createNamespace

public void createNamespace(String ns)
                     throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

createNamespace

public void createNamespace(String ns,
                            boolean retain)
                     throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

deleteNamespace

public void deleteNamespace(String ns)
                     throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

getTopics

public List<String> getTopics(String ns)
                       throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

topicExists

public boolean topicExists(String topicName)
                    throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

deleteTopic

public void deleteTopic(String topicName)
                 throws org.apache.pulsar.client.admin.PulsarAdminException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

createTopic

public void createTopic(String topicName,
                        int partitionNum)
                 throws org.apache.pulsar.client.admin.PulsarAdminException,
                        IncompatibleSchemaException

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

IncompatibleSchemaException

uploadSchema

public void uploadSchema(String topicName,
                         org.apache.pulsar.common.schema.SchemaInfo schemaInfo)
                  throws IncompatibleSchemaException

Throws:

IncompatibleSchemaException

deleteSchema

public void deleteSchema(String topicName)

setupCursor

public void setupCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset,
                        boolean failOnDataLoss)

setupCursor

public void setupCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)

commitOffsetToCursor

public void commitOffsetToCursor(Map<TopicRange,org.apache.pulsar.client.api.MessageId> offset)

removeCursor

public void removeCursor(Set<TopicRange> topics)
                  throws PulsarMetadataReader.ClosedException

Throws:

PulsarMetadataReader.ClosedException

getPositionFromSubscription

public org.apache.pulsar.client.api.MessageId getPositionFromSubscription(TopicRange topic,
                                                                          org.apache.pulsar.client.api.MessageId defaultPosition)

getPulsarSchema

public org.apache.pulsar.common.schema.SchemaInfo getPulsarSchema(List<String> topics)
                                                           throws IncompatibleSchemaException

Throws:

IncompatibleSchemaException

getPulsarSchema

public org.apache.pulsar.common.schema.SchemaInfo getPulsarSchema(String topic)

getRange

public SerializableRange getRange()

getTopicPartitionRanges

public Set<TopicRange> getTopicPartitionRanges()
                                        throws org.apache.pulsar.client.admin.PulsarAdminException

Get all TopicRange that should be consumed by the subTask.

Returns:

set of topic ranges this subTask should consume

Throws:

org.apache.pulsar.client.admin.PulsarAdminException

getTopicPartitions

public Set<String> getTopicPartitions()
                               throws org.apache.pulsar.client.admin.PulsarAdminException

Get topic partitions all. If the topic does not exist, it is created automatically

Returns:

allTopicPartitions

Throws:

org.apache.pulsar.client.admin.PulsarAdminException - pulsarAdminException

getLastMessageId

public org.apache.pulsar.client.api.MessageId getLastMessageId(String topic)

checkCursorAvailable

public boolean checkCursorAvailable(String topic,
                                    org.apache.pulsar.client.impl.MessageIdImpl startMessageId)

resetCursor

public void resetCursor(TopicRange topicRange,
                        org.apache.pulsar.client.api.MessageId messageId)