org.apache.flink.connector.pulsar.sink

Class PulsarSink<IN>

java.lang.Object

org.apache.flink.connector.pulsar.sink.PulsarSink<IN>

Type Parameters:

IN - The input type of the sink.

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.sink2.Sink<IN>, org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,PulsarCommittable>

@PublicEvolving
public class PulsarSink<IN>
extends Object
implements org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,PulsarCommittable>

The Sink implementation of Pulsar. Please use a PulsarSinkBuilder to construct a PulsarSink. The following example shows how to create a PulsarSink receiving records of String type.

 PulsarSink<String> sink = PulsarSink.builder()
      .setServiceUrl(operator().serviceUrl())
      .setAdminUrl(operator().adminUrl())
      .setTopic(topic)
      .setSerializationSchema(PulsarSerializationSchema.pulsarSchema(Schema.STRING))
      .build();
 

The sink supports all delivery guarantees described by DeliveryGuarantee.

• DeliveryGuarantee.NONE does not provide any guarantees: messages may be lost in case of issues on the Pulsar broker and messages may be duplicated in case of a Flink failure.
• DeliveryGuarantee.AT_LEAST_ONCE the sink will wait for all outstanding records in the Pulsar buffers to be acknowledged by the Pulsar producer on a checkpoint. No messages will be lost in case of any issue with the Pulsar brokers but messages may be duplicated when Flink restarts.
• DeliveryGuarantee.EXACTLY_ONCE: In this mode the PulsarSink will write all messages in a Pulsar transaction that will be committed to Pulsar on a checkpoint. Thus, no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly. Additionally, it is highly recommended to tweak Pulsar transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Pulsar expires an uncommitted transaction.

See PulsarSinkBuilder for more details.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink

org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter<InputT,CommT>

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.Sink

org.apache.flink.api.connector.sink2.Sink.InitContext

Method Summary

Modifier and Type	Method and Description
static <IN> PulsarSinkBuilder<IN>	builder() Create a PulsarSinkBuilder to construct a new PulsarSink.
org.apache.flink.api.connector.sink2.Committer<PulsarCommittable>	createCommitter()
org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,PulsarCommittable>	createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext initContext)
org.apache.flink.core.io.SimpleVersionedSerializer<PulsarCommittable>	getCommittableSerializer()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static <IN> PulsarSinkBuilder<IN> builder()

Create a PulsarSinkBuilder to construct a new PulsarSink.

Type Parameters:

IN - Type of incoming records.

Returns:

A Pulsar sink builder.

createWriter

@Internal
public org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,PulsarCommittable> createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext initContext)

Specified by:

createWriter in interface org.apache.flink.api.connector.sink2.Sink<IN>

Specified by:

createWriter in interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,PulsarCommittable>

createCommitter

@Internal
public org.apache.flink.api.connector.sink2.Committer<PulsarCommittable> createCommitter()

Specified by:

createCommitter in interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,PulsarCommittable>

getCommittableSerializer

@Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<PulsarCommittable> getCommittableSerializer()

Specified by:

getCommittableSerializer in interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,PulsarCommittable>