org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,KinesisIO.Write.Result>

org.apache.beam.sdk.io.aws2.kinesis.KinesisIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Write<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,KinesisIO.Write.Result>

Implementation of KinesisIO.write().

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KinesisIO.Write.Result Result of KinesisIO.write().

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
KinesisIO.Write.Result	expand(org.apache.beam.sdk.values.PCollection<T> input)
KinesisIO.Write<T>	withBatchMaxBytes(int bytes) Max.
KinesisIO.Write<T>	withBatchMaxRecords(int records) Max.
KinesisIO.Write<T>	withClientConfiguration(ClientConfiguration config) Configuration of Kinesis client.
KinesisIO.Write<T>	withConcurrentRequests(int concurrentRequests) Max number of concurrent batch write requests per bundle.
KinesisIO.Write<T>	withPartitioner(KinesisPartitioner<T> partitioner) Specify how to partition records among all stream shards (required).
KinesisIO.Write<T>	withRecordAggregation(java.util.function.Consumer<KinesisIO.RecordAggregation.Builder> aggregation) Enable record aggregation that is compatible with the KPL / KCL.
KinesisIO.Write<T>	withRecordAggregation(KinesisIO.RecordAggregation aggregation) Enable record aggregation that is compatible with the KPL / KCL.
KinesisIO.Write<T>	withRecordAggregationDisabled() Disable KPL / KCL like record aggregation.
KinesisIO.Write<T>	withSerializer(org.apache.beam.sdk.transforms.SerializableFunction<T,byte[]> serializer) Specify how to serialize records to bytes on the stream (required).
KinesisIO.Write<T>	withStreamName(java.lang.String streamName) Kinesis stream name which will be used for writing (required).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withStreamName

public KinesisIO.Write<T> withStreamName(java.lang.String streamName)

Kinesis stream name which will be used for writing (required).

withBatchMaxRecords

public KinesisIO.Write<T> withBatchMaxRecords(int records)

Max. number of records to send per batch write request.

withBatchMaxBytes

public KinesisIO.Write<T> withBatchMaxBytes(int bytes)

Max. number of bytes to send per batch write request.

Single records that exceed this limit are sent individually. Though, be careful to not violate the AWS API limit of 1MB per request.

This includes both partition keys and data.

withConcurrentRequests

public KinesisIO.Write<T> withConcurrentRequests(int concurrentRequests)

Max number of concurrent batch write requests per bundle.

Note: Concurrency settings above the default have caused a bug in the AWS SDK v2. Therefore, this configuration is currently not exposed to users.

withRecordAggregation

public KinesisIO.Write<T> withRecordAggregation(KinesisIO.RecordAggregation aggregation)

Enable record aggregation that is compatible with the KPL / KCL.

https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

withRecordAggregation

public KinesisIO.Write<T> withRecordAggregation(java.util.function.Consumer<KinesisIO.RecordAggregation.Builder> aggregation)

Enable record aggregation that is compatible with the KPL / KCL.

https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

withRecordAggregationDisabled

public KinesisIO.Write<T> withRecordAggregationDisabled()

Disable KPL / KCL like record aggregation.

withPartitioner

public KinesisIO.Write<T> withPartitioner(KinesisPartitioner<T> partitioner)

Specify how to partition records among all stream shards (required).

The partitioner is critical to distribute new records among all stream shards.

withSerializer

public KinesisIO.Write<T> withSerializer(org.apache.beam.sdk.transforms.SerializableFunction<T,byte[]> serializer)

Specify how to serialize records to bytes on the stream (required).

withClientConfiguration

public KinesisIO.Write<T> withClientConfiguration(ClientConfiguration config)

Configuration of Kinesis client.

expand

public KinesisIO.Write.Result expand(org.apache.beam.sdk.values.PCollection<T> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,KinesisIO.Write.Result>