org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaProducerBase<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase<IN>

Type Parameters:

IN - Type of the messages to write into Kafka.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, SinkFunction<IN>

public abstract class FlinkKafkaProducerBase<IN>
extends RichSinkFunction<IN>

Flink Sink to produce data into a Kafka topic. Please note that this producer does not have any reliability guarantees.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Exception	asyncException Errors encountered in the async producer are stored here
protected org.apache.kafka.clients.producer.Callback	callback The callback than handles error propagation or logging callbacks
static String	KEY_DISABLE_METRICS Configuration key for disabling the metrics reporting
protected boolean	logFailuresOnly Flag indicating whether to accept failures (and log them), or to fail on failures
protected KafkaPartitioner<IN>	partitioner User-provided partitioner for assigning an object to a Kafka partition.
protected int[]	partitions Array with the partition ids of the given topicId The size of this array is the number of partitions
protected org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]>	producer KafkaProducer instance
protected Properties	producerConfig User defined properties for the Producer
protected KeyedSerializationSchema<IN>	schema (Serializable) SerializationSchema for turning objects used with Flink into byte[] for Kafka.
protected String	topicId The name of the topic this producer is writing data to

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaProducerBase(String topicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, KafkaPartitioner<IN> customPartitioner) The main constructor for creating a FlinkKafkaProducer.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkErroneous()
void	close()
static Properties	getPropertiesFromBrokerList(String brokerList)
void	invoke(IN next) Called when new data arrives to the sink, and forwards it to Kafka.
void	open(org.apache.flink.configuration.Configuration configuration) Initializes the connection to Kafka.
void	setLogFailuresOnly(boolean logFailuresOnly) Defines whether the producer should fail on errors, or only log them.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEY_DISABLE_METRICS

public static final String KEY_DISABLE_METRICS

Configuration key for disabling the metrics reporting

See Also:

Constant Field Values

partitions

protected final int[] partitions

Array with the partition ids of the given topicId The size of this array is the number of partitions

producerConfig

protected final Properties producerConfig

User defined properties for the Producer

topicId

protected final String topicId

The name of the topic this producer is writing data to

schema

protected final KeyedSerializationSchema<IN> schema

(Serializable) SerializationSchema for turning objects used with Flink into byte[] for Kafka.

partitioner

protected final KafkaPartitioner<IN> partitioner

User-provided partitioner for assigning an object to a Kafka partition.

logFailuresOnly

protected boolean logFailuresOnly

Flag indicating whether to accept failures (and log them), or to fail on failures

producer

protected transient org.apache.kafka.clients.producer.KafkaProducer<byte[],byte[]> producer

KafkaProducer instance

callback

protected transient org.apache.kafka.clients.producer.Callback callback

The callback than handles error propagation or logging callbacks

asyncException

protected transient volatile Exception asyncException

Errors encountered in the async producer are stored here

Constructor Detail

FlinkKafkaProducerBase

public FlinkKafkaProducerBase(String topicId,
                      KeyedSerializationSchema<IN> serializationSchema,
                      Properties producerConfig,
                      KafkaPartitioner<IN> customPartitioner)

The main constructor for creating a FlinkKafkaProducer.

Parameters:

topicId - The topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. Passing null will use Kafka's partitioner

Method Detail

setLogFailuresOnly

public void setLogFailuresOnly(boolean logFailuresOnly)

Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery).

Parameters:

logFailuresOnly - The flag to indicate logging-only on exceptions.

open

public void open(org.apache.flink.configuration.Configuration configuration)

Initializes the connection to Kafka.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

invoke

public void invoke(IN next)
            throws Exception

Called when new data arrives to the sink, and forwards it to Kafka.

Specified by:

invoke in interface SinkFunction<IN>

Specified by:

invoke in class RichSinkFunction<IN>

Parameters:

next - The incoming data

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

checkErroneous

protected void checkErroneous()
                       throws Exception

Throws:

Exception

getPropertiesFromBrokerList

public static Properties getPropertiesFromBrokerList(String brokerList)