org.apache.beam.sdk.io.kinesis

Class KinesisIO.Write

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<byte[]>,org.apache.beam.sdk.values.PDone>

org.apache.beam.sdk.io.kinesis.KinesisIO.Write

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

KinesisIO

public abstract static class KinesisIO.Write
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<byte[]>,org.apache.beam.sdk.values.PDone>

Implementation of KinesisIO.write().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PDone	expand(org.apache.beam.sdk.values.PCollection<byte[]> input)
KinesisIO.Write	withAWSClientsProvider(AWSClientsProvider awsClientsProvider) Allows to specify custom AWSClientsProvider.
KinesisIO.Write	withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider, com.amazonaws.regions.Regions region) Specify AWSCredentialsProvider and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider, com.amazonaws.regions.Regions region, java.lang.String serviceEndpoint) Specify AWSCredentialsProvider and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider, com.amazonaws.regions.Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, com.amazonaws.regions.Regions region) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, com.amazonaws.regions.Regions region, java.lang.String serviceEndpoint) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withAWSClientsProvider(java.lang.String awsAccessKey, java.lang.String awsSecretKey, com.amazonaws.regions.Regions region, java.lang.String serviceEndpoint, boolean verifyCertificate) Specify credential details and region to be used to write to Kinesis.
KinesisIO.Write	withPartitioner(KinesisPartitioner partitioner) Allows to specify custom implementation of KinesisPartitioner.
KinesisIO.Write	withPartitionKey(java.lang.String partitionKey) Specify default partition key.
KinesisIO.Write	withProducerProperties(java.util.Properties properties) Specify the configuration properties for Kinesis Producer Library (KPL).
KinesisIO.Write	withStreamName(java.lang.String streamName) Specify Kinesis stream name which will be used for writing, this name is required.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withStreamName

public KinesisIO.Write withStreamName(java.lang.String streamName)

Specify Kinesis stream name which will be used for writing, this name is required.

withPartitionKey

public KinesisIO.Write withPartitionKey(java.lang.String partitionKey)

Specify default partition key.

In case if you need to define more complicated logic for key partitioning then you can create your own implementation of KinesisPartitioner and specify it by withPartitioner(KinesisPartitioner)

Using one of the methods withPartitioner(KinesisPartitioner) or withPartitionKey(String) is required but not both in the same time.

withPartitioner

public KinesisIO.Write withPartitioner(KinesisPartitioner partitioner)

Allows to specify custom implementation of KinesisPartitioner.

This method should be used to balance a distribution of new written records among all stream shards.

Using one of the methods withPartitioner(KinesisPartitioner) or withPartitionKey(String) is required but not both in the same time.

withProducerProperties

public KinesisIO.Write withProducerProperties(java.util.Properties properties)

Specify the configuration properties for Kinesis Producer Library (KPL).

Example of creating new KPL configuration:

Properties properties = new Properties(); properties.setProperty("CollectionMaxCount", "1000"); properties.setProperty("ConnectTimeout", "10000");

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(AWSClientsProvider awsClientsProvider)

Allows to specify custom AWSClientsProvider. AWSClientsProvider creates new IKinesisProducer which is later used for writing to Kinesis.

This method should be used if withAWSClientsProvider(AWSCredentialsProvider, Regions) does not suit well.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider,
                                              com.amazonaws.regions.Regions region)

Specify AWSCredentialsProvider and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              com.amazonaws.regions.Regions region)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider,
                                              com.amazonaws.regions.Regions region,
                                              java.lang.String serviceEndpoint)

Specify AWSCredentialsProvider and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              com.amazonaws.regions.Regions region,
                                              java.lang.String serviceEndpoint)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(com.amazonaws.auth.AWSCredentialsProvider awsCredentialsProvider,
                                              com.amazonaws.regions.Regions region,
                                              java.lang.String serviceEndpoint,
                                              boolean verifyCertificate)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

withAWSClientsProvider

public KinesisIO.Write withAWSClientsProvider(java.lang.String awsAccessKey,
                                              java.lang.String awsSecretKey,
                                              com.amazonaws.regions.Regions region,
                                              java.lang.String serviceEndpoint,
                                              boolean verifyCertificate)

Specify credential details and region to be used to write to Kinesis. If you need more sophisticated credential protocol, then you should look at withAWSClientsProvider(AWSClientsProvider).

The serviceEndpoint sets an alternative service host. This is useful to execute the tests with Kinesis service emulator.

The verifyCertificate disables or enables certificate verification. Never set it to false in production.

expand

public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<byte[]> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<byte[]>,org.apache.beam.sdk.values.PDone>