cn.leancloud.kafka.consumer

Class LcKafkaConsumerBuilder<K,V>

java.lang.Object

cn.leancloud.kafka.consumer.LcKafkaConsumerBuilder<K,V>

Type Parameters:

K - the type of key for records consumed from Kafka

V - the type of value for records consumed from Kafka

public final class LcKafkaConsumerBuilder<K,V>
extends Object

A builder used to create a LcKafkaConsumer which uses a KafkaConsumer to consume records from Kafka broker.

Method Summary

Modifier and Type	Method and Description
<K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1>	buildAsync() Build a consumer in which the polling thread always does a async commit after all the polled records has been handled.
<K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1>	buildAuto() Build a consumer which commits offset automatically at fixed interval.
<K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1>	buildPartialAsync() Build a consumer in which the polling thread does a async commits whenever there's any handled consumer records.
<K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1>	buildPartialSync() Build a consumer in which the polling thread does a sync commits whenever there's any handled consumer records.
<K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1>	buildSync() Build a consumer in which the polling thread always does a sync commit after all the polled records has been handled.
LcKafkaConsumerBuilder<K,V>	forceSeekTo(ConsumerSeekDestination destination) After subscribed to any topic, force seek offsets for every assigned partitions to beginning or end no matter where the recorded offsets for the group of this consumer are.
LcKafkaConsumerBuilder<K,V>	gracefulShutdownTimeout(Duration gracefulShutdownTimeout) Sets the amount of time to wait after calling LcKafkaConsumer.close() for consumed records to handle before actually shutting down.
LcKafkaConsumerBuilder<K,V>	gracefulShutdownTimeoutMillis(long gracefulShutdownTimeoutMillis) Sets the amount of time to wait after calling LcKafkaConsumer.close() for consumed records to handle before actually shutting down.
LcKafkaConsumerBuilder<K,V>	handleRecordTimeout(Duration handleRecordTimeout) The maximum time spent in handling a single ConsumerRecord by calling ConsumerRecordHandler.handleRecord(ConsumerRecord).
LcKafkaConsumerBuilder<K,V>	handleRecordTimeoutMillis(long handleRecordTimeoutMillis) The maximum time spent in handling a single ConsumerRecord by calling ConsumerRecordHandler.handleRecord(ConsumerRecord).
LcKafkaConsumerBuilder<K,V>	maxAttemptsForEachSyncCommit(int maxAttemptsForEachSyncCommit) Sets the maximum attempt times for a synchronous commit by calling KafkaConsumer.commitSync().
LcKafkaConsumerBuilder<K,V>	maxPendingAsyncCommits(int maxPendingAsyncCommits) When using async consumer to commit offset asynchronously, this argument can force consumer to do a synchronous commit after there's already this (maxPendingAsyncCommits) many async commits on the fly without response from broker.
static <K,V> LcKafkaConsumerBuilder<K,V>	newBuilder(Map<String,Object> kafkaConfigs, ConsumerRecordHandler<K,V> consumerRecordHandler) Create a LcKafkaConsumerBuilder used to build LcKafkaConsumer.
static <K,V> LcKafkaConsumerBuilder<K,V>	newBuilder(Map<String,Object> kafkaConfigs, ConsumerRecordHandler<K,V> consumerRecordHandler, org.apache.kafka.common.serialization.Deserializer<K> keyDeserializer, org.apache.kafka.common.serialization.Deserializer<V> valueDeserializer) Create a LcKafkaConsumerBuilder used to build LcKafkaConsumer.
LcKafkaConsumerBuilder<K,V>	pollTimeout(Duration pollTimeout) The pollTimeout is the maximum time spent waiting in polling data from kafka broker if data is not available in the buffer.
LcKafkaConsumerBuilder<K,V>	pollTimeoutMillis(long pollTimeoutMillis) The pollTimeout is the maximum time spent waiting in polling data from kafka broker if data is not available in the buffer.
LcKafkaConsumerBuilder<K,V>	recommitInterval(Duration recommitInterval) The interval to commit all partitions and it's completed offsets to broker on a non-automatic commit consumer.
LcKafkaConsumerBuilder<K,V>	recommitIntervalInMillis(long recommitIntervalInMillis) The interval to commit all partitions and it's completed offsets to broker on a non-automatic commit consumer.
LcKafkaConsumerBuilder<K,V>	syncCommitRetryInterval(Duration syncCommitRetryInterval) Sets the amount of time to wait before retry a failed synchronous commit on calling KafkaConsumer.commitSync().
LcKafkaConsumerBuilder<K,V>	syncCommitRetryIntervalMillis(long syncCommitRetryIntervalMillis) Sets the amount of time to wait before retry a failed synchronous commit on calling KafkaConsumer.commitSync().
LcKafkaConsumerBuilder<K,V>	workerPool(ExecutorService workerPool, boolean shutdownOnStop) The thread pool used by consumer to handle the consumed records from Kafka broker.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

newBuilder

public static <K,V> LcKafkaConsumerBuilder<K,V> newBuilder(Map<String,Object> kafkaConfigs,
                                                           ConsumerRecordHandler<K,V> consumerRecordHandler)

Create a LcKafkaConsumerBuilder used to build LcKafkaConsumer.

Parameters:

kafkaConfigs - the kafka configs for KafkaConsumer. Please refer this document for valid configurations.

consumerRecordHandler - a ConsumerRecordHandler to handle the consumed record from kafka

Returns:

a new LcKafkaConsumerBuilder

Throws:

NullPointerException - when kafkaConfigs or consumerRecordHandler is null

newBuilder

public static <K,V> LcKafkaConsumerBuilder<K,V> newBuilder(Map<String,Object> kafkaConfigs,
                                                           ConsumerRecordHandler<K,V> consumerRecordHandler,
                                                           org.apache.kafka.common.serialization.Deserializer<K> keyDeserializer,
                                                           org.apache.kafka.common.serialization.Deserializer<V> valueDeserializer)

Create a LcKafkaConsumerBuilder used to build LcKafkaConsumer.

Parameters:

kafkaConfigs - the kafka configs for KafkaConsumer. Please refer this document for valid configurations.

consumerRecordHandler - a ConsumerRecordHandler to handle the consumed record from kafka

keyDeserializer - the deserializer for key that implements Deserializer

valueDeserializer - the deserializer for value that implements Deserializer

Returns:

a new LcKafkaConsumerBuilder

Throws:

NullPointerException - when any of the input argument is null

pollTimeoutMillis

public LcKafkaConsumerBuilder<K,V> pollTimeoutMillis(long pollTimeoutMillis)

The pollTimeout is the maximum time spent waiting in polling data from kafka broker if data is not available in the buffer.

If 0, poll operation will return immediately with any records that are available currently in the buffer, else returns empty.

Must not be negative. And the default pollTimeoutMillis is 100.

Parameters:

pollTimeoutMillis - the poll timeout in milliseconds

Returns:

this

Throws:

IllegalArgumentException - if pollTimeoutMillis is a negative value

pollTimeout

public LcKafkaConsumerBuilder<K,V> pollTimeout(Duration pollTimeout)

The pollTimeout is the maximum time spent waiting in polling data from kafka broker if data is not available in the buffer.

If 0, poll operation will return immediately with any records that are available currently in the buffer, else returns empty.

The default pollTimeout is 100 millis seconds.

Parameters:

pollTimeout - the poll timeout duration

Returns:

this

Throws:

NullPointerException - if pollTimeout is null

IllegalArgumentException - if pollTimeout is a negative duration

handleRecordTimeoutMillis

public LcKafkaConsumerBuilder<K,V> handleRecordTimeoutMillis(long handleRecordTimeoutMillis)

The maximum time spent in handling a single ConsumerRecord by calling ConsumerRecordHandler.handleRecord(ConsumerRecord). If the handling time for any ConsumerRecord exceeds this limit, a TimeoutException will be thrown which then will drag the LcKafkaConsumer to shutdown. This mechanism is to prevent the "livelock" situation where it seems the LcKafkaConsumer is OK, continuing on sending heartbeat and calling Consumer.poll(long), but no progress is being made.

The default handleRecordTimeoutMillis is zero which means no limit on handling a ConsumerRecord.

Parameters:

handleRecordTimeoutMillis - the handle record timeout in millis seconds.

Returns:

this

Throws:

IllegalArgumentException - if handleRecordTimeoutMillis is a negative value

handleRecordTimeout

public LcKafkaConsumerBuilder<K,V> handleRecordTimeout(Duration handleRecordTimeout)

The maximum time spent in handling a single ConsumerRecord by calling ConsumerRecordHandler.handleRecord(ConsumerRecord). If the handling time for any ConsumerRecord exceeds this limit, a TimeoutException will be thrown which then will drag the LcKafkaConsumer to shutdown. This mechanism is to prevent the "livelock" situation where it seems the LcKafkaConsumer is OK, continuing on sending heartbeat and calling Consumer.poll(long), but no progress is being made.

The default handleRecordTimeoutMillis is zero which means no limit on handling a ConsumerRecord.

Parameters:

handleRecordTimeout - the handle record timeout duration.

Returns:

this

Throws:

NullPointerException - if handleRecordTimeout is null

IllegalArgumentException - if handleRecordTimeout is a negative duration

gracefulShutdownTimeoutMillis

public LcKafkaConsumerBuilder<K,V> gracefulShutdownTimeoutMillis(long gracefulShutdownTimeoutMillis)

Sets the amount of time to wait after calling LcKafkaConsumer.close() for consumed records to handle before actually shutting down.

The default gracefulShutdownTimeoutMillis is 10_000.

Parameters:

gracefulShutdownTimeoutMillis - the graceful shutdown timeout in milliseconds

Returns:

this

Throws:

IllegalArgumentException - if gracefulShutdownTimeoutMillis is a negative value

gracefulShutdownTimeout

public LcKafkaConsumerBuilder<K,V> gracefulShutdownTimeout(Duration gracefulShutdownTimeout)

Sets the amount of time to wait after calling LcKafkaConsumer.close() for consumed records to handle before actually shutting down.

The default gracefulShutdownTimeout is 10 seconds.

Parameters:

gracefulShutdownTimeout - the graceful shutdown timeout duration

Returns:

this

Throws:

NullPointerException - if gracefulShutdownTimeout is null

IllegalArgumentException - if gracefulShutdownTimeout is a negative duration

syncCommitRetryIntervalMillis

public LcKafkaConsumerBuilder<K,V> syncCommitRetryIntervalMillis(long syncCommitRetryIntervalMillis)

Sets the amount of time to wait before retry a failed synchronous commit on calling KafkaConsumer.commitSync(). or KafkaConsumer.commitSync(Map). Every synchronous commit may fail but most of times they are caused by RetriableException and we can retry commit on this kind of exception safely. This configuration set the interval between each retry.

For those failures of asynchronous commit by calling KafkaConsumer.commitAsync() or KafkaConsumer.commitAsync(OffsetCommitCallback), we retry them by a synchronous commit automatically when we found any of them. So we only need configurations for synchronous commits.

The default syncCommitRetryIntervalMillis is 1000.

Parameters:

syncCommitRetryIntervalMillis - the retry interval in milliseconds

Returns:

this

Throws:

IllegalArgumentException - if syncCommitRetryIntervalMillis is a negative value

syncCommitRetryInterval

public LcKafkaConsumerBuilder<K,V> syncCommitRetryInterval(Duration syncCommitRetryInterval)

Sets the amount of time to wait before retry a failed synchronous commit on calling KafkaConsumer.commitSync(). or KafkaConsumer.commitSync(Map). Every synchronous commit may fail but most of times they are caused by RetriableException and we can retry commit on this kind of exception safely. This configuration set the interval between each retry.

For those failures of asynchronous commit by calling KafkaConsumer.commitAsync() or KafkaConsumer.commitAsync(OffsetCommitCallback), we retry them by a synchronous commit automatically when we found any of them. So we only need configurations for synchronous commits.

The default syncCommitRetryInterval is 1 second.

Parameters:

syncCommitRetryInterval - the retry interval

Returns:

this

Throws:

NullPointerException - if syncCommitRetryInterval is null

IllegalArgumentException - if syncCommitRetryInterval is a negative duration

maxAttemptsForEachSyncCommit

public LcKafkaConsumerBuilder<K,V> maxAttemptsForEachSyncCommit(int maxAttemptsForEachSyncCommit)

Sets the maximum attempt times for a synchronous commit by calling KafkaConsumer.commitSync(). or KafkaConsumer.commitSync(Map). Every synchronous commit may fail but most of times they are caused by RetriableException and we can retry commit on this kind of exception safely. This configuration cap the maximum retry times. If attempts reach to maxAttemptsForEachSyncCommit, the cached RetriableException will be rethrown by then it will cause the Kafka Consumer to stop and quit.

For those failures of asynchronous commit by calling KafkaConsumer.commitAsync() or KafkaConsumer.commitAsync(OffsetCommitCallback), we retry them by a synchronous commit automatically when we found any of them. So we only need configurations for synchronous commits.

Please note that maxAttemptsForEachSyncCommit multiplies syncCommitRetryInterval should far lower than max.poll.interval.ms, otherwise Kafka Consumer may encounter session timeout or polling timeout due to not calling KafkaConsumer.poll(long) for too long.

The default maxAttemptsForEachSyncCommit is 3.

Parameters:

maxAttemptsForEachSyncCommit - maximum attempt times for a synchronous commit

Returns:

this

Throws:

IllegalArgumentException - if maxAttemptsForEachSyncCommit is zero or a negative value

maxPendingAsyncCommits

public LcKafkaConsumerBuilder<K,V> maxPendingAsyncCommits(int maxPendingAsyncCommits)

When using async consumer to commit offset asynchronously, this argument can force consumer to do a synchronous commit after there's already this (maxPendingAsyncCommits) many async commits on the fly without response from broker.

The default maxPendingAsyncCommits is 10.

Parameters:

maxPendingAsyncCommits - do a synchronous commit when pending async commits beyond this limit

Returns:

this

Throws:

IllegalArgumentException - if maxPendingAsyncCommits is a non-positive value

recommitIntervalInMillis

public LcKafkaConsumerBuilder<K,V> recommitIntervalInMillis(long recommitIntervalInMillis)

The interval to commit all partitions and it's completed offsets to broker on a non-automatic commit consumer.

This configuration is only valid and is required on a non-automatic commit consumer build with buildSync(), buildAsync(), buildPartialSync() or buildPartialAsync(). For these kind of consumers, usually they only commit offsets of a partition when there was records consumed from that partition and all these consumed records was handled successfully. But we must periodically commit those subscribed partitions who have had records but no new records for a long time too. Otherwise, after commit offset retention timeout, Kafka broker may forget where the current commit offset of these partition for the consumer are. Then, when the consumer crashed and recovered, if the consumer set auto.offset.reset configuration to earliest, it may consume a already consumed record again. So please make sure that recommitIntervalInMillis is within the limit set by offsets.retention.minutes on Kafka broker or even within 1/3 of that limit to tolerate some commit failures on async commit consumer.

The default recommitInterval is 1 hour.

Parameters:

recommitIntervalInMillis - the interval in millis seconds to do a recommit

Returns:

this

Throws:

IllegalArgumentException - if recommitIntervalInMillis is a non-positive value

recommitInterval

public LcKafkaConsumerBuilder<K,V> recommitInterval(Duration recommitInterval)

The interval to commit all partitions and it's completed offsets to broker on a non-automatic commit consumer.

This configuration is only valid and is required on a non-automatic commit consumer build with buildSync(), buildAsync(), buildPartialSync() or buildPartialAsync(). For these kind of consumers, usually they only commit offsets of a partition when there was records consumed from that partition and all these consumed records was handled successfully. But we must periodically commit those subscribed partitions who have had records but no new records for a long time too. Otherwise, after commit offset retention timeout, Kafka broker may forget where the current commit offset of these partition for the consumer are. Then, when the consumer crashed and recovered, if the consumer set auto.offset.reset configuration to earliest, it may consume a already consumed record again. So please make sure that recommitInterval is within the limit set by offsets.retention.minutes on Kafka broker or even within 1/3 of that limit to tolerate some commit failures on async commit consumer..

The default recommitInterval is 1 hour.

Parameters:

recommitInterval - the interval to do a recommit

Returns:

this

Throws:

NullPointerException - if recommitInterval is null

IllegalArgumentException - if recommitInterval is a non-positive duration

workerPool

public LcKafkaConsumerBuilder<K,V> workerPool(ExecutorService workerPool,
                                              boolean shutdownOnStop)

The thread pool used by consumer to handle the consumed records from Kafka broker. If no worker pool is provided, the created LcKafkaConsumer will use ImmediateExecutorService to handle records in the records polling thread instead.

When a worker pool is provided, after each poll, the polling thread will take one thread from this worker pool for each polled ConsumerRecord to handle the record. Please tune the max.poll.records in kafka configs to limit the number of records polled at each time do not exceed the max size of the provided worker thread pool. Otherwise, a RejectedExecutionException will thrown when the polling thread submitting too much tasks to the pool. Then this exception will lead the only polling thread to exit.

If you are using partial sync/async commit consumer by building LcKafkaConsumer with buildPartialSync() or buildPartialAsync(), without a worker pool, they degrade to sync/async commit consumer as built with buildSync() or buildAsync().

If no worker pool provided, you also need to tune max.poll.interval.ms in kafka configs, to ensure the polling thread can at least poll once within max.poll.interval.ms during handling consumed messages to prevent itself from session timeout or polling timeout.

Parameters:

workerPool - a thread pool to handle consumed records

shutdownOnStop - true to shutdown the input worker pool when this consumer closed

Returns:

this

Throws:

NullPointerException - if workerPool is null

forceSeekTo

public LcKafkaConsumerBuilder<K,V> forceSeekTo(ConsumerSeekDestination destination)

After subscribed to any topic, force seek offsets for every assigned partitions to beginning or end no matter where the recorded offsets for the group of this consumer are. The difference between this configuration with "auto.offset.reset" in kafka configurations is that "auto.offset.reset" only reset offset to beginning or end when there's no recorded offsets or the previous recorded offsets was removed, while forceSeekTo() will reset offsets even when there are offsets records.

Please note that the reset offset operation will take place only once when a partition is assigned on the first time. If a partition was revoked then assigned again, the offset of this partition will not reset again.

Parameters:

destination - where to seek to

Returns:

this

buildAuto

public <K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1> buildAuto()

Build a consumer which commits offset automatically at fixed interval. It is both OK for with or without a worker thread pool. But without a worker pool, please tune the max.poll.interval.ms in Kafka configs as mentioned in workerPool(ExecutorService, boolean).

This kind of consumer requires the following kafka configs must be set, otherwise IllegalArgumentException will be thrown:

1. max.poll.interval.ms
2. max.poll.records
3. auto.offset.reset
4. auto.commit.interval.ms

Though all of these configs have default values in kafka, we still require every user to set them specifically. Because these configs is vital for using this consumer safely.

If you set enable.auto.commit to false, this consumer will set it to true by itself.

Returns:

this

Throws:

IllegalArgumentException - if any required kafka consumer configurations is invalid or not provided

buildSync

public <K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1> buildSync()

Build a consumer in which the polling thread always does a sync commit after all the polled records has been handled. Because it only commits after all the polled records handled, so the longer the records handling process， the longer the interval between each commits, the bigger of the possibility to repeatedly consume a same record when the consumer crash.

This kind of consumer ensures to do a sync commit to commit all the finished records at that time when the consumer is shutdown or any partition was revoked. It requires the following kafka configs must be set, otherwise an IllegalArgumentException will be thrown:

1. max.poll.records
2. auto.offset.reset

Though all of these configs have default values in kafka, we still require every user to set them specifically. Because these configs is vital for using this consumer safely.

If you set enable.auto.commit to true, this consumer will set it to false by itself.

Returns:

this

Throws:

IllegalArgumentException - if any required kafka consumer configurations is invalid or not provided

buildPartialSync

public <K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1> buildPartialSync()

Build a consumer in which the polling thread does a sync commits whenever there's any handled consumer records. It commits often, so after a consumer crash, comparatively little records may be handled more than once. But also due to commit often, the overhead causing by committing is relatively high.

This kind of consumer ensures to do a sync commit to commit all the finished records at that time when the consumer is shutdown or any partition was revoked. It requires the following kafka configs must be set, otherwise an IllegalArgumentException will be thrown:

1. max.poll.records
2. auto.offset.reset

Though all of these configs have default values in kafka, we still require every user to set them specifically. Because these configs is vital for using this consumer safely.

If you set enable.auto.commit to true, this consumer will set it to false by itself.

Returns:

this

Throws:

IllegalArgumentException - if any required kafka consumer configurations is invalid or not provided

buildAsync

public <K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1> buildAsync()

Build a consumer in which the polling thread always does a async commit after all the polled records has been handled. Because it only commits after all the polled records handled, so the longer the records handling process， the longer the interval between each commits, the bigger of the possibility to repeatedly consume a same record when the consumer crash.

If any async commit is failed or the number of pending async commits is beyond the limit set by maxPendingAsyncCommits(int), this consumer will do a sync commit to commit all the records which have been handled.

This kind of consumer ensures to do a sync commit to commit all the finished records at that time when the consumer is shutdown or any partition was revoked. It requires the following kafka configs must be set, otherwise an IllegalArgumentException will be thrown:

1. max.poll.records
2. auto.offset.reset

Though all of these configs have default values in kafka, we still require every user to set them specifically. Because these configs is vital for using this consumer safely.

If you set enable.auto.commit to true, this consumer will set it to false by itself.

Returns:

this

Throws:

IllegalArgumentException - if any required kafka consumer configurations is invalid or not provided

buildPartialAsync

public <K1 extends K,V1 extends V> LcKafkaConsumer<K1,V1> buildPartialAsync()

Build a consumer in which the polling thread does a async commits whenever there's any handled consumer records. It commits often, so after a consumer crash, comparatively little records may be handled more than once. It use async commit to mitigate the overhead causing by high committing times.

If any async commit is failed or the number of pending async commits is beyond the limit set by maxPendingAsyncCommits(int), this consumer will do a sync commit to commit all the records which have been handled.

This kind of consumer ensures to do a sync commit to commit all the finished records at that time when the consumer is shutdown or any partition was revoked. It requires the following kafka configs must be set, otherwise an IllegalArgumentException will be thrown:

1. max.poll.records
2. auto.offset.reset

Though all of these configs have default values in kafka, we still require every user to set them specifically. Because these configs is vital for using this consumer safely.

If you set enable.auto.commit to true, this consumer will set it to false by itself.

Returns:

this

Throws:

IllegalArgumentException - if any required kafka consumer configurations is invalid or not provided