org.apache.flink.streaming.connectors.kafka.internals

Interface Fetcher

All Known Implementing Classes:

LegacyFetcher

public interface Fetcher

A fetcher pulls data from Kafka, from a fix set of partitions. The fetcher supports "seeking" inside the partitions, i.e., moving to a different offset.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the fetcher.
<T> void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> valueDeserializer, HashMap<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> lastOffsets) Starts fetch data from Kafka and emitting it into the stream.
void	stopWithError(Throwable t) Exit run loop with given error and release all resources.

Method Detail

close

void close()
    throws IOException

Closes the fetcher. This will stop any operation in the run(SourceFunction.SourceContext, KeyedDeserializationSchema, HashMap) method and eventually close underlying connections and release all resources.

Throws:

IOException

run

<T> void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
             org.apache.flink.streaming.util.serialization.KeyedDeserializationSchema<T> valueDeserializer,
             HashMap<org.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition,Long> lastOffsets)
      throws Exception

Starts fetch data from Kafka and emitting it into the stream.

To provide exactly once guarantees, the fetcher needs emit a record and update the update of the last consumed offset in one atomic operation:

 
 while (running) {
     T next = ...
     long offset = ...
     int partition = ...
     synchronized (sourceContext.getCheckpointLock()) {
         sourceContext.collect(next);
         lastOffsets[partition] = offset;
     }
 }
 

Type Parameters:

T - The type of elements produced by the fetcher and emitted to the source context.

Parameters:

sourceContext - The source context to emit elements to.

valueDeserializer - The deserializer to decode the raw values with.

lastOffsets - The map into which to store the offsets for which elements are emitted (operator state)

Throws:

Exception

stopWithError

void stopWithError(Throwable t)

Exit run loop with given error and release all resources.

Parameters:

t - Error cause