org.apache.flink.streaming.connectors.kafka.internals

Class Kafka08Fetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

org.apache.flink.streaming.connectors.kafka.internals.Kafka08Fetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

@Internal
public class Kafka08Fetcher<T>
extends org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

A fetcher that fetches data from Kafka brokers via the Kafka 0.8 low-level consumer API. The fetcher also handles the explicit communication with ZooKeeper to fetch initial offsets and to write offsets to ZooKeeper.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

sourceContext, unassignedPartitionsQueue

Constructor Summary

Constructors

Constructor and Description

Kafka08Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> seedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long autoCommitInterval, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
void	cancel()
protected kafka.common.TopicAndPartition	createKafkaPartitionHandle(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition partition)
protected void	doCommitInternalOffsetsToKafka(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> offsets, org.apache.flink.streaming.connectors.kafka.internals.KafkaCommitCallback commitCallback)
void	runFetchLoop()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecord, emitRecordWithTimestamp, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Kafka08Fetcher

public Kafka08Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                      Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> seedPartitionsWithInitialOffsets,
                      org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                      org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                      org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext,
                      org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                      Properties kafkaProperties,
                      long autoCommitInterval,
                      org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                      boolean useMetrics)
               throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Specified by:

runFetchLoop in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

createKafkaPartitionHandle

protected kafka.common.TopicAndPartition createKafkaPartitionHandle(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition partition)

Specified by:

createKafkaPartitionHandle in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

doCommitInternalOffsetsToKafka

protected void doCommitInternalOffsetsToKafka(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> offsets,
                                              @Nonnull
                                              org.apache.flink.streaming.connectors.kafka.internals.KafkaCommitCallback commitCallback)
                                       throws Exception

Specified by:

doCommitInternalOffsetsToKafka in class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,kafka.common.TopicAndPartition>

Throws:

Exception