org.apache.beam.sdk.io.kafka

Class KafkaIO.Read<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KafkaRecord<K,V>>>

org.apache.beam.sdk.io.kafka.KafkaIO.TypedRead<K,V>

org.apache.beam.sdk.io.kafka.KafkaIO.Read<K,V>

All Implemented Interfaces:

Serializable, HasDisplayData

Enclosing class:

KafkaIO

public static class KafkaIO.Read<K,V>
extends KafkaIO.TypedRead<K,V>

A PTransform to read from Kafka topics. See KafkaIO for more information on usage and configuration.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.io.kafka.KafkaIO.TypedRead

consumerConfig, consumerFactoryFn, keyCoder, maxNumRecords, maxReadTime, timestampFn, topicPartitions, topics, valueCoder, watermarkFn

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Method Summary

Modifier and Type	Method and Description
KafkaIO.Read<K,V>	updateConsumerProperties(Map<String,Object> configUpdates) Update consumer configuration with new properties.
KafkaIO.Read<K,V>	withBootstrapServers(String bootstrapServers) Returns a new KafkaIO.Read with Kafka consumer pointing to bootstrapServers.
KafkaIO.Read<K,V>	withConsumerFactoryFn(SerializableFunction<Map<String,Object>,org.apache.kafka.clients.consumer.Consumer<byte[],byte[]>> consumerFactoryFn) A factory to create Kafka Consumer from consumer configuration.
<KeyT> KafkaIO.Read<KeyT,V>	withKeyCoder(Coder<KeyT> keyCoder) Returns a new KafkaIO.Read with Coder for key bytes.
KafkaIO.Read<K,V>	withMaxNumRecords(long maxNumRecords) Similar to Read.Unbounded.withMaxNumRecords(long).
KafkaIO.Read<K,V>	withMaxReadTime(org.joda.time.Duration maxReadTime) Similar to Read.Unbounded.withMaxReadTime(Duration).
KafkaIO.Read<K,V>	withTopicPartitions(List<org.apache.kafka.common.TopicPartition> topicPartitions) Returns a new KafkaIO.Read that reads from the partitions.
KafkaIO.Read<K,V>	withTopics(List<String> topics) Returns a new KafkaIO.Read that reads from the topics.
<ValueT> KafkaIO.Read<K,ValueT>	withValueCoder(Coder<ValueT> valueCoder) Returns a new KafkaIO.Read with Coder for value bytes.

Methods inherited from class org.apache.beam.sdk.io.kafka.KafkaIO.TypedRead

apply, withoutMetadata, withTimestampFn, withTimestampFn2, withWatermarkFn, withWatermarkFn2

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

withBootstrapServers

public KafkaIO.Read<K,V> withBootstrapServers(String bootstrapServers)

Returns a new KafkaIO.Read with Kafka consumer pointing to bootstrapServers.

withTopics

public KafkaIO.Read<K,V> withTopics(List<String> topics)

Returns a new KafkaIO.Read that reads from the topics. All the partitions are from each of the topics is read. See UnboundedKafkaSource#generateInitialSplits(int, PipelineOptions) for description of how the partitions are distributed among the splits.

withTopicPartitions

public KafkaIO.Read<K,V> withTopicPartitions(List<org.apache.kafka.common.TopicPartition> topicPartitions)

Returns a new KafkaIO.Read that reads from the partitions. This allows reading only a subset of partitions for one or more topics when (if ever) needed. See UnboundedKafkaSource#generateInitialSplits(int, PipelineOptions) for description of how the partitions are distributed among the splits.

withKeyCoder

public <KeyT> KafkaIO.Read<KeyT,V> withKeyCoder(Coder<KeyT> keyCoder)

Returns a new KafkaIO.Read with Coder for key bytes.

withValueCoder

public <ValueT> KafkaIO.Read<K,ValueT> withValueCoder(Coder<ValueT> valueCoder)

Returns a new KafkaIO.Read with Coder for value bytes.

withConsumerFactoryFn

public KafkaIO.Read<K,V> withConsumerFactoryFn(SerializableFunction<Map<String,Object>,org.apache.kafka.clients.consumer.Consumer<byte[],byte[]>> consumerFactoryFn)

A factory to create Kafka Consumer from consumer configuration. This is useful for supporting another version of Kafka consumer. Default is KafkaConsumer.

updateConsumerProperties

public KafkaIO.Read<K,V> updateConsumerProperties(Map<String,Object> configUpdates)

Update consumer configuration with new properties.

withMaxNumRecords

public KafkaIO.Read<K,V> withMaxNumRecords(long maxNumRecords)

Similar to Read.Unbounded.withMaxNumRecords(long). Mainly used for tests and demo applications.

withMaxReadTime

public KafkaIO.Read<K,V> withMaxReadTime(org.joda.time.Duration maxReadTime)

Similar to Read.Unbounded.withMaxReadTime(Duration). Mainly used for tests and demo applications.