org.apache.flink.streaming.connectors.kafka.internals

Class LegacyFetcher

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher

All Implemented Interfaces:

Fetcher

public class LegacyFetcher
extends Object
implements Fetcher

This fetcher uses Kafka's low-level API to pull data from a specific set of topics and partitions.

This code is in parts based on the tutorial code for the low-level Kafka consumer.

Constructor Summary

Constructors

Constructor and Description
LegacyFetcher(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> initialPartitionsToRead, Properties props, String taskName, ClassLoader userCodeClassloader) Create a LegacyFetcher instance.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the fetcher.
static List<String>	getTopics(List<org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher.FetchPartition> partitionsList) Returns a unique list of topics from the topic partition list
<T> void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer, HashMap<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> lastOffsets) Starts fetch data from Kafka and emitting it into the stream.
void	stopWithError(Throwable error) Reports an error from a fetch thread.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LegacyFetcher

public LegacyFetcher(Map<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> initialPartitionsToRead,
                     Properties props,
                     String taskName,
                     ClassLoader userCodeClassloader)

Create a LegacyFetcher instance.

Parameters:

initialPartitionsToRead - Map of partitions to read. The offset passed is the last-fetched-offset (not the next-offset-to-fetch).

props - kafka properties

taskName - name of the parent task

userCodeClassloader - classloader for loading user code

Method Detail

close

public void close()

Description copied from interface: Fetcher

Closes the fetcher. This will stop any operation in the Fetcher.run(SourceFunction.SourceContext, KeyedDeserializationSchema, HashMap) method and eventually close underlying connections and release all resources.

Specified by:

close in interface Fetcher

run

public <T> void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                    org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> deserializer,
                    HashMap<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> lastOffsets)
             throws Exception

Description copied from interface: Fetcher

Starts fetch data from Kafka and emitting it into the stream.

To provide exactly once guarantees, the fetcher needs emit a record and update the update of the last consumed offset in one atomic operation:

 
 while (running) {
     T next = ...
     long offset = ...
     int partition = ...
     synchronized (sourceContext.getCheckpointLock()) {
         sourceContext.collect(next);
         lastOffsets[partition] = offset;
     }
 }
 

Specified by:

run in interface Fetcher

Type Parameters:

T - The type of elements produced by the fetcher and emitted to the source context.

Parameters:

sourceContext - The source context to emit elements to.

deserializer - The deserializer to decode the raw values with.

lastOffsets - The map into which to store the offsets for which elements are emitted (operator state)

Throws:

Exception

stopWithError

public void stopWithError(Throwable error)

Reports an error from a fetch thread. This will cause the main thread to see this error, abort, and cancel all other fetch threads.

Specified by:

stopWithError in interface Fetcher

Parameters:

error - The error to report.

getTopics

public static List<String> getTopics(List<org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher.FetchPartition> partitionsList)

Returns a unique list of topics from the topic partition list

Parameters:

partitionsList - A lost of FetchPartitions's

Returns:

A unique list of topics from the input map