org.apache.beam.sdk.io.kafka

Class KafkaIO

java.lang.Object

org.apache.beam.sdk.io.kafka.KafkaIO

public class KafkaIO
extends Object

An unbounded source and a sink for Kafka topics. Kafka version 0.9 and above are supported.

Reading from Kafka topics

KafkaIO source returns unbounded collection of Kafka records as PCollection<KafkaRecord<K, V>>. A KafkaRecord includes basic metadata like topic-partition and offset, along with key and value associated with a Kafka record.

Although most applications consumer single topic, the source can be configured to consume multiple topics or even a specific set of TopicPartitions.

To configure a Kafka source, you must specify at the minimum Kafka bootstrapServers and one or more topics to consume. The following example illustrates various options for configuring the source :

  pipeline
    .apply(KafkaIO.read()
       .withBootstrapServers("broker_1:9092,broker_2:9092")
       .withTopics(ImmutableList.of("topic_a", "topic_b"))
       // above two are required configuration. returns PCollection<KafkaRecord<byte[], byte[]>

       // rest of the settings are optional :

       // set a Coder for Key and Value (note the change to return type)
       .withKeyCoder(BigEndianLongCoder.of()) // PCollection<KafkaRecord<Long, byte[]>
       .withValueCoder(StringUtf8Coder.of())  // PCollection<KafkaRecord<Long, String>

       // you can further customize KafkaConsumer used to read the records by adding more
       // settings for ConsumerConfig. e.g :
       .updateConsumerProperties(ImmutableMap.of("receive.buffer.bytes", 1024 * 1024))

       // custom function for calculating record timestamp (default is processing time)
       .withTimestampFn(new MyTypestampFunction())

       // custom function for watermark (default is record timestamp)
       .withWatermarkFn(new MyWatermarkFunction())

       // finally, if you don't need Kafka metadata, you can drop it
       .withoutMetadata() // PCollection<KV<Long, String>>
    )
    .apply(Values.<String>create()) // PCollection<String>
     ...
 

Partition Assignment and Checkpointing

The Kafka partitions are evenly distributed among splits (workers). Dataflow checkpointing is fully supported and each split can resume from previous checkpoint. See UnboundedKafkaSource#generateInitialSplits(int, PipelineOptions) for more details on splits and checkpoint support.

When the pipeline starts for the first time without any checkpoint, the source starts consuming from the latest offsets. You can override this behavior to consume from the beginning by setting appropriate appropriate properties in ConsumerConfig, through KafkaIO.Read.updateConsumerProperties(Map).

Writing to Kafka

KafkaIO sink supports writing key-value pairs to a Kafka topic. Users can also write just the values. To configure a Kafka sink, you must specify at the minimum Kafka bootstrapServers and the topic to write to. The following example illustrates various options for configuring the sink:

  pipeline
    .apply(...) // returns PCollection<KV<Long, String>>
    .apply(KafkaIO.write()
       .withBootstrapServers("broker_1:9092,broker_2:9092")
       .withTopic("results")

       // set Coder for Key and Value
       .withKeyCoder(BigEndianLongCoder.of())
       .withValueCoder(StringUtf8Coder.of())

       // you can further customize KafkaProducer used to write the records by adding more
       // settings for ProducerConfig. e.g, to enable compression :
       .updateProducerProperties(ImmutableMap.of("compression.type", "gzip"))
    );
 

Often you might want to write just values without any keys to Kafka. Use values() to write records with default empty(null) key:

  PCollection<String> strings = ...;
  strings.apply(KafkaIO.write()
      .withBootstrapServers("broker_1:9092,broker_2:9092")
      .withTopic("results")
      .withValueCoder(StringUtf8Coder.of()) // just need coder for value
      .values() // writes values to Kafka with default key
    );
 

Advanced Kafka Configuration

KafakIO allows setting most of the properties in ConsumerConfig for source or in ProducerConfig for sink. E.g. if you would like to enable offset auto commit (for external monitoring or other purposes), you can set "group.id", "enable.auto.commit", etc.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaIO.CoderBasedKafkaSerializer<T> Implements Kafka's Serializer with a Coder.
static class	KafkaIO.Read<K,V> A PTransform to read from Kafka topics.
static class	KafkaIO.TypedRead<K,V> A PTransform to read from Kafka topics.
static class	KafkaIO.TypedWithoutMetadata<K,V> A PTransform to read from Kafka topics.
static class	KafkaIO.TypedWrite<K,V> A PTransform to write to a Kafka topic.
static class	KafkaIO.Write<K,V> A PTransform to write to a Kafka topic.

Method Summary

Modifier and Type	Method and Description
static KafkaIO.Read<byte[],byte[]>	read() Creates an uninitialized KafkaIO.Read PTransform.
static KafkaIO.Write<byte[],byte[]>	write() Creates an uninitialized KafkaIO.Write PTransform.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

read

public static KafkaIO.Read<byte[],byte[]> read()

Creates an uninitialized KafkaIO.Read PTransform. Before use, basic Kafka configuration should set with KafkaIO.Read.withBootstrapServers(String) and KafkaIO.Read.withTopics(List). Other optional settings include key and value coders, custom timestamp and watermark functions.

write

public static KafkaIO.Write<byte[],byte[]> write()

Creates an uninitialized KafkaIO.Write PTransform. Before use, Kafka configuration should be set with KafkaIO.Write.withBootstrapServers(String) and KafkaIO.Write.withTopic(java.lang.String) along with Coders for (optional) key and values.