org.apache.flink.streaming.connectors.kafka.internals

Class KafkaPartitionDiscoverer

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer

org.apache.flink.streaming.connectors.kafka.internals.KafkaPartitionDiscoverer

@Internal
public class KafkaPartitionDiscoverer
extends AbstractPartitionDiscoverer

A partition discoverer that can be used to discover topics and partitions metadata from Kafka brokers via the Kafka high-level consumer API.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer

AbstractPartitionDiscoverer.ClosedException, AbstractPartitionDiscoverer.WakeupException

Constructor Summary

Constructors

Constructor and Description
KafkaPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks, Properties kafkaProperties)

Method Summary

Modifier and Type	Method and Description
protected void	closeConnections() Close all established connections.
protected List<KafkaTopicPartition>	getAllPartitionsForTopics(List<String> topics) Fetch the list of all partitions for a specific topics list from Kafka.
protected List<String>	getAllTopics() Fetch the list of all topics from Kafka.
protected void	initializeConnections() Establish the required connections in order to fetch topics and partitions metadata.
protected void	wakeupConnections() Attempt to eagerly wakeup from blocking calls to Kafka in AbstractPartitionDiscoverer.getAllTopics() and AbstractPartitionDiscoverer.getAllPartitionsForTopics(List).

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer

close, discoverPartitions, open, setAndCheckDiscoveredPartition, wakeup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaPartitionDiscoverer

public KafkaPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor,
                                int indexOfThisSubtask,
                                int numParallelSubtasks,
                                Properties kafkaProperties)

Method Detail

initializeConnections

protected void initializeConnections()

Description copied from class: AbstractPartitionDiscoverer

Establish the required connections in order to fetch topics and partitions metadata.

Specified by:

initializeConnections in class AbstractPartitionDiscoverer

getAllTopics

protected List<String> getAllTopics()
                             throws AbstractPartitionDiscoverer.WakeupException

Description copied from class: AbstractPartitionDiscoverer

Fetch the list of all topics from Kafka.

Specified by:

getAllTopics in class AbstractPartitionDiscoverer

Throws:

AbstractPartitionDiscoverer.WakeupException

getAllPartitionsForTopics

protected List<KafkaTopicPartition> getAllPartitionsForTopics(List<String> topics)
                                                       throws AbstractPartitionDiscoverer.WakeupException,
                                                              RuntimeException

Description copied from class: AbstractPartitionDiscoverer

Fetch the list of all partitions for a specific topics list from Kafka.

Specified by:

getAllPartitionsForTopics in class AbstractPartitionDiscoverer

Throws:

AbstractPartitionDiscoverer.WakeupException

RuntimeException

wakeupConnections

protected void wakeupConnections()

Description copied from class: AbstractPartitionDiscoverer

Attempt to eagerly wakeup from blocking calls to Kafka in AbstractPartitionDiscoverer.getAllTopics() and AbstractPartitionDiscoverer.getAllPartitionsForTopics(List).

If the invocation indeed results in interrupting an actual blocking Kafka call, the implementations of AbstractPartitionDiscoverer.getAllTopics() and AbstractPartitionDiscoverer.getAllPartitionsForTopics(List) are responsible of throwing a AbstractPartitionDiscoverer.WakeupException.

Specified by:

wakeupConnections in class AbstractPartitionDiscoverer

closeConnections

protected void closeConnections()
                         throws Exception

Description copied from class: AbstractPartitionDiscoverer

Close all established connections.

Specified by:

closeConnections in class AbstractPartitionDiscoverer

Throws:

Exception