org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaProducer09<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase<IN>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer09<IN>

Type Parameters:

IN - Type of the messages to write into Kafka.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

@PublicEvolving
public class FlinkKafkaProducer09<IN>
extends org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase<IN>

Flink Sink to produce data into a Kafka topic. This producer is compatible with Kafka 0.9.

Please note that this producer does not have any reliability guarantees.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context<T>

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase

asyncException, callback, defaultTopicId, flinkKafkaPartitioner, flushOnCheckpoint, KEY_DISABLE_METRICS, logFailuresOnly, pendingRecords, pendingRecordsLock, producer, producerConfig, schema, topicPartitionsMap

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaProducer09(String topicId, org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer09(String topicId, org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<IN> customPartitioner) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer09(String topicId, org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<IN> customPartitioner) Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer09(String, KeyedSerializationSchema, Properties, FlinkKafkaPartitioner) instead.
FlinkKafkaProducer09(String topicId, org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer09(String topicId, org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<IN> customPartitioner) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer09(String topicId, org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<IN> customPartitioner) Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer09(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.
FlinkKafkaProducer09(String brokerList, String topicId, org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer09(String brokerList, String topicId, org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema) Creates a FlinkKafkaProducer for a given topic.

Method Summary

Modifier and Type	Method and Description
protected void	flush()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase

checkErroneous, close, getKafkaProducer, getPartitionsByTopic, getPropertiesFromBrokerList, initializeState, invoke, numPendingRecords, open, setFlushOnCheckpoint, setLogFailuresOnly, snapshotState

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

invoke

Constructor Detail

FlinkKafkaProducer09

public FlinkKafkaProducer09(String brokerList,
                            String topicId,
                            org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer09(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined key-less serialization schema.

FlinkKafkaProducer09

public FlinkKafkaProducer09(String topicId,
                            org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema,
                            Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer09(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined key-less serialization schema.

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer09

public FlinkKafkaProducer09(String topicId,
                            org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema,
                            Properties producerConfig,
                            @Nullable
                            org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<IN> customPartitioner)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a key-less SerializationSchema and possibly a custom FlinkKafkaPartitioner.

Since a key-less SerializationSchema is used, all records sent to Kafka will not have an attached key. Therefore, if a partitioner is also not provided, records will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

topicId - The topic to write data to

serializationSchema - A key-less serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If set to null, records will be distributed to Kafka partitions in a round-robin fashion.

FlinkKafkaProducer09

public FlinkKafkaProducer09(String brokerList,
                            String topicId,
                            org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer09(String, KeyedSerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

FlinkKafkaProducer09

public FlinkKafkaProducer09(String topicId,
                            org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema,
                            Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Using this constructor, the default FlinkFixedPartitioner will be used as the partitioner. This default partitioner maps each sink subtask to a single Kafka partition (i.e. all records received by a sink subtask will end up in the same Kafka partition).

To use a custom partitioner, please use FlinkKafkaProducer09(String, KeyedSerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema supporting key/value messages

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer09

public FlinkKafkaProducer09(String topicId,
                            org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema,
                            Properties producerConfig,
                            @Nullable
                            org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<IN> customPartitioner)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic. It accepts a keyed KeyedSerializationSchema and possibly a custom FlinkKafkaPartitioner.

If a partitioner is not provided, written records will be partitioned by the attached key of each record (as determined by KeyedSerializationSchema.serializeKey(Object)). If written records do not have a key (i.e., KeyedSerializationSchema.serializeKey(Object) returns null), they will be distributed to Kafka partitions in a round-robin fashion.

Parameters:

topicId - The topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions. If set to null, records will be partitioned by the key of each record (determined by KeyedSerializationSchema.serializeKey(Object)). If the keys are null, then records will be distributed to Kafka partitions in a round-robin fashion.

FlinkKafkaProducer09

@Deprecated
public FlinkKafkaProducer09(String topicId,
                                        org.apache.flink.api.common.serialization.SerializationSchema<IN> serializationSchema,
                                        Properties producerConfig,
                                        org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<IN> customPartitioner)

Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer09(String, SerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Creates a FlinkKafkaProducer for a given topic. the sink produces a DataStream to the topic.

Parameters:

topicId - The topic to write data to

serializationSchema - A (keyless) serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions (when passing null, we'll use Kafka's partitioner)

FlinkKafkaProducer09

@Deprecated
public FlinkKafkaProducer09(String topicId,
                                        org.apache.flink.streaming.util.serialization.KeyedSerializationSchema<IN> serializationSchema,
                                        Properties producerConfig,
                                        org.apache.flink.streaming.connectors.kafka.partitioner.KafkaPartitioner<IN> customPartitioner)

Deprecated. This is a deprecated constructor that does not correctly handle partitioning when producing to multiple topics. Use FlinkKafkaProducer09(String, KeyedSerializationSchema, Properties, FlinkKafkaPartitioner) instead.

Creates a FlinkKafkaProducer for a given topic. The sink produces a DataStream to the topic.

Parameters:

topicId - The topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assigning messages to Kafka partitions.

Method Detail

flush

protected void flush()

Specified by:

flush in class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase<IN>