Package org.apache.beam.sdk.io.aws2.kinesis

Class KinesisIO.Write<T>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    KinesisIO
    public abstract static class KinesisIO.Write<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,​KinesisIO.Write.Result>
    Implementation of KinesisIO.write().
    See Also:
    Serialized Form

    • Constructor Detail

      • Write

        public Write()

    • Method Detail

      • withStreamName

        public KinesisIO.Write<T> withStreamName​(java.lang.String streamName)
        Kinesis stream name which will be used for writing (required).

      • withBatchMaxRecords

        public KinesisIO.Write<T> withBatchMaxRecords​(int records)
        Max. number of records to send per batch write request.

      • withBatchMaxBytes

        public KinesisIO.Write<T> withBatchMaxBytes​(int bytes)
        Max. number of bytes to send per batch write request.

        Single records that exceed this limit are sent individually. Though, be careful to not violate the AWS API limit of 1MB per request.

        This includes both partition keys and data.

      • withConcurrentRequests

        public KinesisIO.Write<T> withConcurrentRequests​(int concurrentRequests)
        Max number of concurrent batch write requests per bundle.

        Note: Concurrency settings above the default have caused a bug in the AWS SDK v2. Therefore, this configuration is currently not exposed to users.

      • withRecordAggregation

        public KinesisIO.Write<T> withRecordAggregation​(KinesisIO.RecordAggregation aggregation)
        Enable record aggregation that is compatible with the KPL / KCL.

        https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

      • withRecordAggregation

        public KinesisIO.Write<T> withRecordAggregation​(java.util.function.Consumer<KinesisIO.RecordAggregation.Builder> aggregation)
        Enable record aggregation that is compatible with the KPL / KCL.

        https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation

      • withRecordAggregationDisabled

        public KinesisIO.Write<T> withRecordAggregationDisabled()
        Disable KPL / KCL like record aggregation.

      • withPartitioner

        public KinesisIO.Write<T> withPartitioner​(KinesisPartitioner<T> partitioner)
        Specify how to partition records among all stream shards (required).

        The partitioner is critical to distribute new records among all stream shards.

      • withSerializer

        public KinesisIO.Write<T> withSerializer​(org.apache.beam.sdk.transforms.SerializableFunction<T,​byte[]> serializer)
        Specify how to serialize records to bytes on the stream (required).

      • expand

        public KinesisIO.Write.Result expand​(org.apache.beam.sdk.values.PCollection<T> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,​KinesisIO.Write.Result>