org.apache.flink.streaming.connectors.kafka

类 FlinkKafkaConsumer010<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010<T>

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>

@PublicEvolving
public class FlinkKafkaConsumer010<T>
extends org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from Apache Kafka 0.10.x. The consumer can run in multiple parallel instances, each of which will pull data from one or more Kafka partitions.

The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost during a failure, and that the computation processes elements "exactly once". (Note: These guarantees naturally assume that Kafka itself does not loose any data.)

Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets committed to Kafka / ZooKeeper are only to bring the outside view of progress in sync with Flink's view of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer has consumed a topic.

Please refer to Kafka's documentation for the available configuration properties: http://kafka.apache.org/documentation.html#newconsumerconfigs

另请参阅:

序列化表格

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

字段概要

字段

限定符和类型	字段和说明
static long	DEFAULT_POLL_TIMEOUT From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available.
static String	KEY_POLL_TIMEOUT Configuration key to change the polling timeout
protected long	pollTimeout From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available.
protected Properties	properties User-supplied properties for Kafka

从类继承的字段 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

deserializer, KEY_DISABLE_METRICS, KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS, MAX_NUM_PENDING_CHECKPOINTS, PARTITION_DISCOVERY_DISABLED

构造器概要

构造器

构造器和说明
FlinkKafkaConsumer010(List<String> topics, org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics to the consumer.
FlinkKafkaConsumer010(List<String> topics, org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing multiple topics and a key/value deserialization schema.
FlinkKafkaConsumer010(Pattern subscriptionPattern, org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x.
FlinkKafkaConsumer010(Pattern subscriptionPattern, org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x.
FlinkKafkaConsumer010(String topic, org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x.
FlinkKafkaConsumer010(String topic, org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer, Properties props) Creates a new Kafka streaming source consumer for Kafka 0.10.x This constructor allows passing a KafkaDeserializationSchema for reading key/value pairs, offsets, and topic names from Kafka.

方法概要

限定符和类型	方法和说明
protected org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,?>	createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, org.apache.flink.streaming.connectors.kafka.config.OffsetCommitMode offsetCommitMode, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)
protected org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer	createPartitionDiscoverer(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks)
protected Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long>	fetchOffsetsWithTimestamp(Collection<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition> partitions, long timestamp)
protected boolean	getIsAutoCommitEnabled()
org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter	getRateLimiter()
void	setRateLimiter(org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter kafkaRateLimiter) Set a rate limiter to ratelimit bytes read from Kafka.

从类继承的方法 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase

adjustAutoCommitConfig, assignTimestampsAndWatermarks, assignTimestampsAndWatermarks, assignTimestampsAndWatermarks, cancel, close, disableFilterRestoredPartitionsWithSubscribedTopics, getEnableCommitOnCheckpoints, getProducedType, initializeState, notifyCheckpointAborted, notifyCheckpointComplete, open, run, setCommitOffsetsOnCheckpoints, setStartFromEarliest, setStartFromGroupOffsets, setStartFromLatest, setStartFromSpecificOffsets, setStartFromTimestamp, snapshotState

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

KEY_POLL_TIMEOUT

public static final String KEY_POLL_TIMEOUT

Configuration key to change the polling timeout. *

另请参阅:

常量字段值

DEFAULT_POLL_TIMEOUT

public static final long DEFAULT_POLL_TIMEOUT

From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available. If 0, returns immediately with any records that are available now.

另请参阅:

常量字段值

properties

protected final Properties properties

User-supplied properties for Kafka. *

pollTimeout

protected final long pollTimeout

From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available. If 0, returns immediately with any records that are available now

构造器详细资料

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(String topic,
                             org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer,
                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x.

参数:

topic - The name of the topic that should be consumed.

valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(String topic,
                             org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer,
                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x

This constructor allows passing a KafkaDeserializationSchema for reading key/value pairs, offsets, and topic names from Kafka.

参数:

topic - The name of the topic that should be consumed.

deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(List<String> topics,
                             org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializer,
                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x

This constructor allows passing multiple topics to the consumer.

参数:

topics - The Kafka topics to read from.

deserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties that are used to configure both the fetcher and the offset handler.

FlinkKafkaConsumer010

public FlinkKafkaConsumer010(List<String> topics,
                             org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer,
                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x

This constructor allows passing multiple topics and a key/value deserialization schema.

参数:

topics - The Kafka topics to read from.

deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties that are used to configure both the fetcher and the offset handler.

FlinkKafkaConsumer010

@PublicEvolving
public FlinkKafkaConsumer010(Pattern subscriptionPattern,
                                             org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer,
                                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x. Use this constructor to subscribe to multiple topics based on a regular expression pattern.

If partition discovery is enabled (by setting a non-negative value for FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS in the properties), topics with names matching the pattern will also be subscribed to as they are created on the fly.

参数:

subscriptionPattern - The regular expression for a pattern of topic names to subscribe to.

valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

FlinkKafkaConsumer010

@PublicEvolving
public FlinkKafkaConsumer010(Pattern subscriptionPattern,
                                             org.apache.flink.streaming.connectors.kafka.KafkaDeserializationSchema<T> deserializer,
                                             Properties props)

Creates a new Kafka streaming source consumer for Kafka 0.10.x. Use this constructor to subscribe to multiple topics based on a regular expression pattern.

If partition discovery is enabled (by setting a non-negative value for FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS in the properties), topics with names matching the pattern will also be subscribed to as they are created on the fly.

This constructor allows passing a KafkaDeserializationSchema for reading key/value pairs, offsets, and topic names from Kafka.

参数:

subscriptionPattern - The regular expression for a pattern of topic names to subscribe to.

deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.

props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.

方法详细资料

createFetcher

protected org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,?> createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                                                                                                   Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                                                                                                   org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                                                                                                   org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext,
                                                                                                   org.apache.flink.streaming.connectors.kafka.config.OffsetCommitMode offsetCommitMode,
                                                                                                   org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                                                                                                   boolean useMetrics)
                                                                                            throws Exception

指定者:

createFetcher 在类中 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

抛出:

Exception

createPartitionDiscoverer

protected org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer createPartitionDiscoverer(org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicsDescriptor topicsDescriptor,
                                                                                                                      int indexOfThisSubtask,
                                                                                                                      int numParallelSubtasks)

指定者:

createPartitionDiscoverer 在类中 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

getIsAutoCommitEnabled

protected boolean getIsAutoCommitEnabled()

指定者:

getIsAutoCommitEnabled 在类中 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

fetchOffsetsWithTimestamp

protected Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> fetchOffsetsWithTimestamp(Collection<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition> partitions,
                                                                                                                        long timestamp)

指定者:

fetchOffsetsWithTimestamp 在类中 org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

setRateLimiter

public void setRateLimiter(org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter kafkaRateLimiter)

Set a rate limiter to ratelimit bytes read from Kafka.

参数:

kafkaRateLimiter -

getRateLimiter

public org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter getRateLimiter()