org.apache.flink.connector.firehose.sink

Class KinesisFirehoseSinkBuilder<InputT>

java.lang.Object

org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder<InputT,software.amazon.awssdk.services.firehose.model.Record,KinesisFirehoseSinkBuilder<InputT>>

org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkBuilder<InputT>

Type Parameters:

InputT - type of elements that should be persisted in the destination

@PublicEvolving
public class KinesisFirehoseSinkBuilder<InputT>
extends org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder<InputT,software.amazon.awssdk.services.firehose.model.Record,KinesisFirehoseSinkBuilder<InputT>>

Builder to construct KinesisFirehoseSink.

The following example shows the minimum setup to create a KinesisFirehoseSink that writes String values to a Kinesis Data Firehose delivery stream named delivery-stream-name.

 Properties sinkProperties = new Properties();
 sinkProperties.put(AWSConfigConstants.AWS_REGION, "eu-west-1");

 KinesisFirehoseSink<String> kdfSink =
         KinesisFirehoseSink.<String>builder()
                 .setElementConverter(elementConverter)
                 .setDeliveryStreamName("delivery-stream-name")
                 .setMaxBatchSize(20)
                 .setFirehoseClientProperties(sinkProperties)
                 .setSerializationSchema(new SimpleStringSchema())
                 .build();
 

If the following parameters are not set in this builder, the following defaults will be used:

• maxBatchSize will be 500
• maxInFlightRequests will be 50
• maxBufferedRequests will be 10000
• maxBatchSizeInBytes will be 4 MB i.e. 4 * 1024 * 1024
• maxTimeInBufferMS will be 5000ms
• maxRecordSizeInBytes will be 1000 KB i.e. 1000 * 1024
• failOnError will be false

Method Summary

Modifier and Type	Method and Description
KinesisFirehoseSink<InputT>	build()
KinesisFirehoseSinkBuilder<InputT>	setDeliveryStreamName(String deliveryStreamName) Sets the name of the KDF delivery stream that the sink will connect to.
KinesisFirehoseSinkBuilder<InputT>	setFailOnError(boolean failOnError) If writing to Kinesis Data Firehose results in a partial or full failure being returned, the job will fail immediately with a KinesisFirehoseException if failOnError is set.
KinesisFirehoseSinkBuilder<InputT>	setFirehoseClientProperties(Properties firehoseClientProperties) A set of properties used by the sink to create the firehose client.
KinesisFirehoseSinkBuilder<InputT>	setSerializationSchema(org.apache.flink.api.common.serialization.SerializationSchema<InputT> serializationSchema) Allows the user to specify a serialization schema to serialize each record to persist to Firehose.

Methods inherited from class org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder

getMaxBatchSize, getMaxBatchSizeInBytes, getMaxBufferedRequests, getMaxInFlightRequests, getMaxRecordSizeInBytes, getMaxTimeInBufferMS, setMaxBatchSize, setMaxBatchSizeInBytes, setMaxBufferedRequests, setMaxInFlightRequests, setMaxRecordSizeInBytes, setMaxTimeInBufferMS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

setDeliveryStreamName

public KinesisFirehoseSinkBuilder<InputT> setDeliveryStreamName(String deliveryStreamName)

Sets the name of the KDF delivery stream that the sink will connect to. There is no default for this parameter, therefore, this must be provided at sink creation time otherwise the build will fail.

Parameters:

deliveryStreamName - the name of the delivery stream

Returns:

KinesisFirehoseSinkBuilder itself

setSerializationSchema

public KinesisFirehoseSinkBuilder<InputT> setSerializationSchema(org.apache.flink.api.common.serialization.SerializationSchema<InputT> serializationSchema)

Allows the user to specify a serialization schema to serialize each record to persist to Firehose.

Parameters:

serializationSchema - serialization schema to use

Returns:

KinesisFirehoseSinkBuilder itself

setFailOnError

public KinesisFirehoseSinkBuilder<InputT> setFailOnError(boolean failOnError)

If writing to Kinesis Data Firehose results in a partial or full failure being returned, the job will fail immediately with a KinesisFirehoseException if failOnError is set.

Parameters:

failOnError - whether to fail on error

Returns:

KinesisFirehoseSinkBuilder itself

setFirehoseClientProperties

public KinesisFirehoseSinkBuilder<InputT> setFirehoseClientProperties(Properties firehoseClientProperties)

A set of properties used by the sink to create the firehose client. This may be used to set the aws region, credentials etc. See the docs for usage and syntax.

Parameters:

firehoseClientProperties - Firehose client properties

Returns:

KinesisFirehoseSinkBuilder itself

build

public KinesisFirehoseSink<InputT> build()

Specified by:

build in class org.apache.flink.connector.base.sink.AsyncSinkBaseBuilder<InputT,software.amazon.awssdk.services.firehose.model.Record,KinesisFirehoseSinkBuilder<InputT>>