org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaConsumerBase<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.runtime.state.CheckpointListener, Checkpointed<HashMap<KafkaTopicPartition,Long>>, CheckpointedAsynchronously<HashMap<KafkaTopicPartition,Long>>, ParallelSourceFunction<T>, SourceFunction<T>

public abstract class FlinkKafkaConsumerBase<T>
extends RichParallelSourceFunction<T>
implements org.apache.flink.runtime.state.CheckpointListener, CheckpointedAsynchronously<HashMap<KafkaTopicPartition,Long>>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected KeyedDeserializationSchema<T>	deserializer The schema to convert between Kafka#s byte messages, and Flink's objects
static int	MAX_NUM_PENDING_CHECKPOINTS The maximum number of pending non-committed checkpoints to track, to avoid memory leaks
protected HashMap<KafkaTopicPartition,Long>	offsetsState The offsets of the last returned elements
protected org.apache.commons.collections.map.LinkedMap	pendingCheckpoints Data for pending but uncommitted checkpoints
protected HashMap<KafkaTopicPartition,Long>	restoreToOffset The offsets to restore to, if the consumer restores state from a checkpoint
protected boolean	running Flag indicating whether the consumer is still running

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaConsumerBase(KeyedDeserializationSchema<T> deserializer, Properties props) Creates a new Flink Kafka Consumer, using the given type of fetcher and offset handler.

Method Summary

Methods

Modifier and Type	Method and Description
protected static <T> List<T>	assignPartitions(List<T> partitions, int numConsumers, int consumerIndex)
protected abstract void	commitOffsets(HashMap<KafkaTopicPartition,Long> checkpointOffsets)
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
static void	logPartitionInfo(List<KafkaTopicPartition> partitionInfos) Method to log partition information.
void	notifyCheckpointComplete(long checkpointId)
void	restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets)
HashMap<KafkaTopicPartition,Long>	snapshotState(long checkpointId, long checkpointTimestamp)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

cancel, run

Field Detail

MAX_NUM_PENDING_CHECKPOINTS

public static final int MAX_NUM_PENDING_CHECKPOINTS

The maximum number of pending non-committed checkpoints to track, to avoid memory leaks

See Also:

Constant Field Values

deserializer

protected final KeyedDeserializationSchema<T> deserializer

The schema to convert between Kafka#s byte messages, and Flink's objects

pendingCheckpoints

protected final org.apache.commons.collections.map.LinkedMap pendingCheckpoints

Data for pending but uncommitted checkpoints

offsetsState

protected transient HashMap<KafkaTopicPartition,Long> offsetsState

The offsets of the last returned elements

restoreToOffset

protected transient HashMap<KafkaTopicPartition,Long> restoreToOffset

The offsets to restore to, if the consumer restores state from a checkpoint

running

protected volatile boolean running

Flag indicating whether the consumer is still running

Constructor Detail

FlinkKafkaConsumerBase

public FlinkKafkaConsumerBase(KeyedDeserializationSchema<T> deserializer,
                      Properties props)

Creates a new Flink Kafka Consumer, using the given type of fetcher and offset handler.

To determine which kink of fetcher and offset handler to use, please refer to the docs at the beginnign of this class.

Parameters:

deserializer - The deserializer to turn raw byte messages into Java/Scala objects.

props - The properties that are used to configure both the fetcher and the offset handler.

Method Detail

snapshotState

public HashMap<KafkaTopicPartition,Long> snapshotState(long checkpointId,
                                              long checkpointTimestamp)
                                                throws Exception

Specified by:

snapshotState in interface Checkpointed<HashMap<KafkaTopicPartition,Long>>

Throws:

Exception

restoreState

public void restoreState(HashMap<KafkaTopicPartition,Long> restoredOffsets)

Specified by:

restoreState in interface Checkpointed<HashMap<KafkaTopicPartition,Long>>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener

Throws:

Exception

commitOffsets

protected abstract void commitOffsets(HashMap<KafkaTopicPartition,Long> checkpointOffsets)
                               throws Exception

Throws:

Exception

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

assignPartitions

protected static <T> List<T> assignPartitions(List<T> partitions,
                           int numConsumers,
                           int consumerIndex)

logPartitionInfo

public static void logPartitionInfo(List<KafkaTopicPartition> partitionInfos)

Method to log partition information.

Parameters:

partitionInfos - List of subscribed partitions