org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaConsumer010<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedRestoring<HashMap<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long>>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>

public class FlinkKafkaConsumer010<T>
extends FlinkKafkaConsumer09<T>

The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from Apache Kafka 0.10.x. The consumer can run in multiple parallel instances, each of which will pull data from one or more Kafka partitions.

The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost during a failure, and that the computation processes elements "exactly once". (Note: These guarantees naturally assume that Kafka itself does not loose any data.)

Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets committed to Kafka / ZooKeeper are only to bring the outside view of progress in sync with Flink's view of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer has consumed a topic.

Please refer to Kafka's documentation for the available configuration properties: http://kafka.apache.org/documentation.html#newconsumerconfigs

NOTE: The implementation currently accesses partition metadata when the consumer is constructed. That means that the client that submits the program needs to be able to reach the Kafka brokers or ZooKeeper.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09

DEFAULT_POLL_TIMEOUT, KEY_POLL_TIMEOUT, pollTimeout, properties

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

deserializer, KEY_DISABLE_METRICS, MAX_NUM_PENDING_CHECKPOINTS, subscribedPartitions

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaConsumer010(List<String> topics, org.apache.flink.streaming.util.serialization.DeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics to the consumer.
FlinkKafkaConsumer010(List<String> topics, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics and a key/value deserialization schema.
FlinkKafkaConsumer010(String topic, org.apache.flink.streaming.util.serialization.DeserializationSchema<T> valueDeserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x
FlinkKafkaConsumer010(String topic, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing a KeyedDeserializationSchema for reading key/value pairs, offsets, and topic names from Kafka.

Method Summary

Methods

Modifier and Type

Method and Description

protected org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,?>

createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, List<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition> thisSubtaskPartitions, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic, org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext)

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09

getKafkaPartitions

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

assignPartitions, assignTimestampsAndWatermarks, assignTimestampsAndWatermarks, cancel, close, getProducedType, initializeState, logPartitionInfo, notifyCheckpointComplete, open, restoreState, run, setSubscribedPartitions, snapshotState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(String topic,
                     org.apache.flink.streaming.util.serialization.DeserializationSchema<T> valueDeserializer,
                     Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x

Parameters:

topic - The name of the topic that should be consumed.

valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(String topic,
                     org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                     Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing a KeyedDeserializationSchema for reading key/value pairs, offsets, and topic names from Kafka.

Parameters:

topic - The name of the topic that should be consumed.

deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(List<String> topics,
                     org.apache.flink.streaming.util.serialization.DeserializationSchema<T> deserializer,
                     Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics to the consumer.

Parameters:

topics - The Kafka topics to read from.

deserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties that are used to configure both the fetcher and the offset handler.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(List<String> topics,
                     org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                     Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics and a key/value deserialization schema.

Parameters:

topics - The Kafka topics to read from.

deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties that are used to configure both the fetcher and the offset handler.

Method Detail

createFetcher

protected org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,?> createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                                                                                       List<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition> thisSubtaskPartitions,
                                                                                       org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,
                                                                                       org.apache.flink.util.SerializedValue<org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,
                                                                                       org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext)
                                                                                            throws Exception

Overrides:

createFetcher in class FlinkKafkaConsumer09<T>

Throws:

Exception