org.apache.flink.streaming.connectors.kafka.internal

Class Kafka010Fetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

public class Kafka010Fetcher<T>
extends org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

A fetcher that fetches data from Kafka brokers via the Kafka 0.10 consumer API.

This fetcher re-uses basically all functionality of the 0.9 fetcher. It only additionally takes the KafkaRecord-attached timestamp and attaches it to the Flink records.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, NO_TIMESTAMPS_WATERMARKS, PERIODIC_WATERMARKS, PUNCTUATED_WATERMARKS, sourceContext, timestampWatermarkMode, useMetrics

Constructor Summary

Constructors

Constructor and Description

Kafka010Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, org.apache.flink.metrics.MetricGroup metricGroup, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, boolean useMetrics)

Method Summary

Modifier and Type	Method and Description
protected KafkaConsumerCallBridge010	createCallBridge() This method needs to be overridden because Kafka broke binary compatibility between 0.9 and 0.10, changing binary signatures
protected void	emitRecord(T record, org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition, long offset, org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
protected String	getFetcherName()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher

cancel, commitInternalOffsetsToKafka, createKafkaPartitionHandle, runFetchLoop

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addOffsetStateGauge, emitRecord, emitRecordWithTimestamp, emitRecordWithTimestampAndPeriodicWatermark, emitRecordWithTimestampAndPunctuatedWatermark, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Kafka010Fetcher

public Kafka010Fetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                       Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                       org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                       org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                       org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                       long autoWatermarkInterval,
                       ClassLoader userCodeClassLoader,
                       String taskNameWithSubtasks,
                       org.apache.flink.metrics.MetricGroup metricGroup,
                       org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                       Properties kafkaProperties,
                       long pollTimeout,
                       boolean useMetrics)
                throws Exception

Throws:

Exception

Method Detail

emitRecord

protected void emitRecord(T record,
                          org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartitionState<org.apache.kafka.common.TopicPartition> partition,
                          long offset,
                          org.apache.kafka.clients.consumer.ConsumerRecord<?,?> consumerRecord)
                   throws Exception

Overrides:

emitRecord in class org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

Throws:

Exception

createCallBridge

protected KafkaConsumerCallBridge010 createCallBridge()

This method needs to be overridden because Kafka broke binary compatibility between 0.9 and 0.10, changing binary signatures

Overrides:

createCallBridge in class org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>

getFetcherName

protected String getFetcherName()

Overrides:

getFetcherName in class org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher<T>