org.apache.flink.connector.kafka.source.reader.fetcher

Class KafkaSourceFetcherManager<T>

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>,KafkaPartitionSplit>

org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager<T>

public class KafkaSourceFetcherManager<T>
extends org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>,KafkaPartitionSplit>

The SplitFetcherManager for Kafka source. This class is needed to help commit the offsets to Kafka using the KafkaConsumer inside the KafkaPartitionSplitReader.

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

fetchers

Constructor Summary

Constructors

Constructor and Description

KafkaSourceFetcherManager(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>>> elementsQueue, java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>,KafkaPartitionSplit>> splitReaderSupplier, java.util.function.Consumer<Collection<String>> splitFinishedHook) Creates a new SplitFetcherManager with a single I/O threads.

Method Summary

Modifier and Type	Method and Description
void	commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager

addSplits, getRunningFetcher

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

checkErrors, close, createSplitFetcher, getNumAliveFetchers, maybeShutdownFinishedFetchers, startFetcher

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaSourceFetcherManager

public KafkaSourceFetcherManager(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>>> elementsQueue,
                                 java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.flink.api.java.tuple.Tuple3<T,Long,Long>,KafkaPartitionSplit>> splitReaderSupplier,
                                 java.util.function.Consumer<Collection<String>> splitFinishedHook)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

splitFinishedHook - Hook for handling finished splits in split fetchers.

Method Detail

commitOffsets

public void commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit,
                          org.apache.kafka.clients.consumer.OffsetCommitCallback callback)