org.apache.flink.streaming.connectors.kafka.internal

Class Kafka09Fetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

@Internal
public class Kafka09Fetcher<T>
extends org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

A fetcher that fetches data from Kafka brokers via the Kafka 0.9 consumer API.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

sourceContext, unassignedPartitionsQueue

Constructor Summary

Constructors

Constructor and Description

Kafka09Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, org.apache.flink.metrics.MetricGroup subtaskMetricGroup, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
void	cancel()
protected KafkaConsumerCallBridge	createCallBridge()
org.apache.kafka.common.TopicPartition	createKafkaPartitionHandle(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition partition)
protected void	doCommitInternalOffsetsToKafka(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> offsets, org.apache.flink.streaming.connectors.kafka.internals.KafkaCommitCallback commitCallback)
protected void	emitRecord(T record, org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition, long offset, org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
protected String	getFetcherName() Gets the name of this fetcher, for thread naming and logging purposes.
void	runFetchLoop()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecord, emitRecordWithTimestamp, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Kafka09Fetcher

public Kafka09Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                      Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                      org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                      org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                      org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                      long autoWatermarkInterval,
                      ClassLoader userCodeClassLoader,
                      String taskNameWithSubtasks,
                      org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                      Properties kafkaProperties,
                      long pollTimeout,
                      org.apache.flink.metrics.MetricGroup subtaskMetricGroup,
                      org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                      boolean useMetrics)
               throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Specified by:

runFetchLoop in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

emitRecord

protected void emitRecord(T record,
                          org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition,
                          long offset,
                          org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
                   throws Exception

Throws:

Exception

getFetcherName

protected String getFetcherName()

Gets the name of this fetcher, for thread naming and logging purposes.

createCallBridge

protected KafkaConsumerCallBridge createCallBridge()

createKafkaPartitionHandle

public org.apache.kafka.common.TopicPartition createKafkaPartitionHandle(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition partition)

Specified by:

createKafkaPartitionHandle in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

doCommitInternalOffsetsToKafka

protected void doCommitInternalOffsetsToKafka(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> offsets,
                                              @Nonnull
                                              org.apache.flink.streaming.connectors.kafka.internals.KafkaCommitCallback commitCallback)
                                       throws Exception

Specified by:

doCommitInternalOffsetsToKafka in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception