org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaProducer010<T>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<Object,SinkFunction<IN>>

org.apache.flink.streaming.api.operators.StreamSink<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer010<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, SinkFunction<T>, CheckpointedRestoringOperator, KeyContext, OneInputStreamOperator<T,Object>, OutputTypeConfigurable<Object>, StreamCheckpointedOperator, StreamOperator<Object>

public class FlinkKafkaProducer010<T>
extends StreamSink<T>
implements SinkFunction<T>, org.apache.flink.api.common.functions.RichFunction

Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.10.x Implementation note: This producer is a hybrid between a regular regular sink function (a) and a custom operator (b). For (a), the class implements the SinkFunction and RichFunction interfaces. For (b), it extends the StreamTask class. Details about approach (a): Pre Kafka 0.10 producers only follow approach (a), allowing users to use the producer using the DataStream.addSink() method. Since the APIs exposed in that variant do not allow accessing the the timestamp attached to the record the Kafka 0.10 producer has a second invocation option, approach (b). Details about approach (b): Kafka 0.10 supports writing the timestamp attached to a record to Kafka. When adding the FlinkKafkaProducer010 using the FlinkKafkaProducer010.writeToKafkaWithTimestamps() method, the Kafka producer can access the internal record timestamp of the record and write it to Kafka. All methods and constructors in this class are marked with the approach they are needed for.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T> Configuration object returned by the writeToKafkaWithTimestamps() call.

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

AbstractStreamOperator.CountingOutput, AbstractStreamOperator.LatencyGauge

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

userFunction

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyGauge, LOG, metrics, output

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaProducer010(String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer010(String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner) Create Kafka producer This constructor does not allow writing timestamps to Kafka, it follow approach (a) (see above)
FlinkKafkaProducer010(String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner) Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.
FlinkKafkaProducer010(String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer010(String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer010(String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner) Deprecated. This is a deprecated since it does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.
FlinkKafkaProducer010(String brokerList, String topicId, KeyedSerializationSchema<T> serializationSchema) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer010(String brokerList, String topicId, SerializationSchema<T> serializationSchema) Creates a FlinkKafkaProducer for a given topic.

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.flink.api.common.functions.IterationRuntimeContext	getIterationRuntimeContext() This method is used for approach (a) (see above)
void	invoke(T value) Invoke method for using the Sink as DataStream.addSink() sink.
void	open(org.apache.flink.configuration.Configuration parameters) This method is used for approach (a) (see above)
void	processElement(StreamRecord<T> element) Process method for using the sink with timestamp support.
void	setFlushOnCheckpoint(boolean flush) If set to true, the Flink producer will wait for all outstanding messages in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint.
void	setLogFailuresOnly(boolean logFailuresOnly) Defines whether the producer should fail on errors, or only log them.
void	setRuntimeContext(org.apache.flink.api.common.functions.RuntimeContext t) This method is used for approach (a) (see above)
static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T>	writeToKafkaWithTimestamps(DataStream<T> inStream, String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T>	writeToKafkaWithTimestamps(DataStream<T> inStream, String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner) Creates a FlinkKafkaProducer for a given topic.
static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T>	writeToKafkaWithTimestamps(DataStream<T> inStream, String topicId, KeyedSerializationSchema<T> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner) Deprecated. This is a deprecated since it does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.
static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T>	writeToKafkaWithTimestamps(DataStream<T> inStream, String topicId, SerializationSchema<T> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.

Methods inherited from class org.apache.flink.streaming.api.operators.StreamSink

reportOrForwardLatencyMarker

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

close, dispose, getUserFunction, getUserFunctionParameters, initializeState, notifyOfCompletedCheckpoint, open, restoreState, setOutputType, setup, snapshotState, snapshotState

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getUserCodeClassloader, initializeState, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, snapshotLegacyOperatorState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

close, getRuntimeContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker, processWatermark

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, dispose, getChainingStrategy, getMetricGroup, initializeState, notifyOfCompletedCheckpoint, open, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setup, snapshotLegacyOperatorState, snapshotState

Constructor Detail

FlinkKafkaProducer010

public FlinkKafkaProducer010(String brokerList,
                     String topicId,
                     SerializationSchema<T> serializationSchema)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined (keyless) serialization schema.

FlinkKafkaProducer010

public FlinkKafkaProducer010(String topicId,
                     SerializationSchema<T> serializationSchema,
                     Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to the topic.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined (keyless) serialization schema.

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer010

public FlinkKafkaProducer010(String topicId,
                     SerializationSchema<T> serializationSchema,
                     Properties producerConfig,
                     org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner)

Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to the topic.

Parameters:

topicId - The topic to write data to

serializationSchema - A (keyless) serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions (when passing null, we'll use Kafka's partitioner)

FlinkKafkaProducer010

public FlinkKafkaProducer010(String brokerList,
                     String topicId,
                     KeyedSerializationSchema<T> serializationSchema)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

FlinkKafkaProducer010

public FlinkKafkaProducer010(String topicId,
                     KeyedSerializationSchema<T> serializationSchema,
                     Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer010

public FlinkKafkaProducer010(String topicId,
                     KeyedSerializationSchema<T> serializationSchema,
                     Properties producerConfig,
                     org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner)

Create Kafka producer This constructor does not allow writing timestamps to Kafka, it follow approach (a) (see above)

FlinkKafkaProducer010

@Deprecated
public FlinkKafkaProducer010(String topicId,
                                SerializationSchema<T> serializationSchema,
                                Properties producerConfig,
                                org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner)

Deprecated. This is a deprecated since it does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to the topic.

Parameters:

topicId - The topic to write data to

serializationSchema - A (keyless) serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions (when passing null, we'll use Kafka's partitioner)

FlinkKafkaProducer010

@Deprecated
public FlinkKafkaProducer010(String topicId,
                                KeyedSerializationSchema<T> serializationSchema,
                                Properties producerConfig,
                                org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner)

Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Create Kafka producer This constructor does not allow writing timestamps to Kafka, it follow approach (a) (see above)

Method Detail

writeToKafkaWithTimestamps

public static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T> writeToKafkaWithTimestamps(DataStream<T> inStream,
                                                                                         String topicId,
                                                                                         KeyedSerializationSchema<T> serializationSchema,
                                                                                         Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic. This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)

Parameters:

inStream - The stream to write to Kafka

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

producerConfig - Properties with the producer configuration.

writeToKafkaWithTimestamps

public static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T> writeToKafkaWithTimestamps(DataStream<T> inStream,
                                                                                         String topicId,
                                                                                         SerializationSchema<T> serializationSchema,
                                                                                         Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to the topic. This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)

Parameters:

inStream - The stream to write to Kafka

topicId - ID of the Kafka topic.

serializationSchema - User defined (keyless) serialization schema.

producerConfig - Properties with the producer configuration.

writeToKafkaWithTimestamps

public static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T> writeToKafkaWithTimestamps(DataStream<T> inStream,
                                                                                         String topicId,
                                                                                         KeyedSerializationSchema<T> serializationSchema,
                                                                                         Properties producerConfig,
                                                                                         org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T> customPartitioner)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic. This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)

Parameters:

inStream - The stream to write to Kafka

topicId - The name of the target topic

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions.

writeToKafkaWithTimestamps

@Deprecated
public static <T> FlinkKafkaProducer010.FlinkKafkaProducer010Configuration<T> writeToKafkaWithTimestamps(DataStream<T> inStream,
                                                                                                    String topicId,
                                                                                                    KeyedSerializationSchema<T> serializationSchema,
                                                                                                    Properties producerConfig,
                                                                                                    org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<T> customPartitioner)

Deprecated. This is a deprecated since it does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer010(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic. This constructor allows writing timestamps to Kafka, it follow approach (b) (see above)

Parameters:

inStream - The stream to write to Kafka

topicId - The name of the target topic

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions.

setLogFailuresOnly

public void setLogFailuresOnly(boolean logFailuresOnly)

Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery). Method is only accessible for approach (a) (see above)

Parameters:

logFailuresOnly - The flag to indicate logging-only on exceptions.

setFlushOnCheckpoint

public void setFlushOnCheckpoint(boolean flush)

If set to true, the Flink producer will wait for all outstanding messages in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. This way, the producer can guarantee that messages in the Kafka buffers are part of the checkpoint. Method is only accessible for approach (a) (see above)

Parameters:

flush - Flag indicating the flushing mode (true = flush on checkpoint)

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

This method is used for approach (a) (see above)

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Throws:

Exception

getIterationRuntimeContext

public org.apache.flink.api.common.functions.IterationRuntimeContext getIterationRuntimeContext()

This method is used for approach (a) (see above)

Specified by:

getIterationRuntimeContext in interface org.apache.flink.api.common.functions.RichFunction

setRuntimeContext

public void setRuntimeContext(org.apache.flink.api.common.functions.RuntimeContext t)

This method is used for approach (a) (see above)

Specified by:

setRuntimeContext in interface org.apache.flink.api.common.functions.RichFunction

invoke

public void invoke(T value)
            throws Exception

Invoke method for using the Sink as DataStream.addSink() sink. This method is used for approach (a) (see above)

Specified by:

invoke in interface SinkFunction<T>

Parameters:

value - The input record.

Throws:

Exception

processElement

public void processElement(StreamRecord<T> element)
                    throws Exception

Process method for using the sink with timestamp support. This method is used for approach (b) (see above)

Specified by:

processElement in interface OneInputStreamOperator<T,Object>

Overrides:

processElement in class StreamSink<T>

Throws:

Exception