org.apache.beam.sdk.io.kafka

Class KafkaIO.TypedRead<K,V>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<KafkaRecord<K,V>>>

org.apache.beam.sdk.io.kafka.KafkaIO.TypedRead<K,V>

All Implemented Interfaces:

Serializable, HasDisplayData

Direct Known Subclasses:

KafkaIO.Read

Enclosing class:

KafkaIO

public static class KafkaIO.TypedRead<K,V>
extends PTransform<PBegin,PCollection<KafkaRecord<K,V>>>

A PTransform to read from Kafka topics. See KafkaIO for more information on usage and configuration.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,Object>	consumerConfig
protected SerializableFunction<Map<String,Object>,org.apache.kafka.clients.consumer.Consumer<byte[],byte[]>>	consumerFactoryFn
protected Coder<K>	keyCoder
protected long	maxNumRecords
protected org.joda.time.Duration	maxReadTime
protected SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant>	timestampFn
protected List<org.apache.kafka.common.TopicPartition>	topicPartitions
protected List<String>	topics
protected Coder<V>	valueCoder
protected SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant>	watermarkFn

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Method Summary

Modifier and Type	Method and Description
PCollection<KafkaRecord<K,V>>	apply(PBegin input)
PTransform<PBegin,PCollection<KV<K,V>>>	withoutMetadata() Returns a PTransform for PCollection of KV, dropping Kafka metatdata.
KafkaIO.TypedRead<K,V>	withTimestampFn(SerializableFunction<KV<K,V>,org.joda.time.Instant> timestampFn) A function to assign a timestamp to a record.
KafkaIO.TypedRead<K,V>	withTimestampFn2(SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFn) A function to assign a timestamp to a record.
KafkaIO.TypedRead<K,V>	withWatermarkFn(SerializableFunction<KV<K,V>,org.joda.time.Instant> watermarkFn) A function to calculate watermark after a record.
KafkaIO.TypedRead<K,V>	withWatermarkFn2(SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> watermarkFn) A function to calculate watermark after a record.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

topics

protected final List<String> topics

topicPartitions

protected final List<org.apache.kafka.common.TopicPartition> topicPartitions

keyCoder

protected final Coder<K> keyCoder

valueCoder

protected final Coder<V> valueCoder

timestampFn

@Nullable
protected final SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFn

watermarkFn

@Nullable
protected final SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> watermarkFn

consumerFactoryFn

protected final SerializableFunction<Map<String,Object>,org.apache.kafka.clients.consumer.Consumer<byte[],byte[]>> consumerFactoryFn

consumerConfig

protected final Map<String,Object> consumerConfig

maxNumRecords

protected final long maxNumRecords

maxReadTime

protected final org.joda.time.Duration maxReadTime

Method Detail

withTimestampFn2

public KafkaIO.TypedRead<K,V> withTimestampFn2(SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFn)

A function to assign a timestamp to a record. Default is processing timestamp.

withWatermarkFn2

public KafkaIO.TypedRead<K,V> withWatermarkFn2(SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> watermarkFn)

A function to calculate watermark after a record. Default is last record timestamp

See Also:

withTimestampFn(SerializableFunction)

withTimestampFn

public KafkaIO.TypedRead<K,V> withTimestampFn(SerializableFunction<KV<K,V>,org.joda.time.Instant> timestampFn)

A function to assign a timestamp to a record. Default is processing timestamp.

withWatermarkFn

public KafkaIO.TypedRead<K,V> withWatermarkFn(SerializableFunction<KV<K,V>,org.joda.time.Instant> watermarkFn)

A function to calculate watermark after a record. Default is last record timestamp

See Also:

withTimestampFn(SerializableFunction)

withoutMetadata

public PTransform<PBegin,PCollection<KV<K,V>>> withoutMetadata()

Returns a PTransform for PCollection of KV, dropping Kafka metatdata.

apply

public PCollection<KafkaRecord<K,V>> apply(PBegin input)

Overrides:

apply in class PTransform<PBegin,PCollection<KafkaRecord<K,V>>>