Package org.apache.camel.processor.idempotent.kafka

Class KafkaIdempotentRepository

java.lang.Object
org.apache.camel.support.service.BaseService
org.apache.camel.support.service.ServiceSupport
org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository
All Implemented Interfaces:
AutoCloseable, org.apache.camel.CamelContextAware, org.apache.camel.Service, org.apache.camel.ShutdownableService, org.apache.camel.spi.HasCamelContext, org.apache.camel.spi.IdempotentRepository, org.apache.camel.StatefulService, org.apache.camel.SuspendableService
@Metadata(label="bean", description="Idempotent repository that uses Kafka to store message ids. Uses a local cache of previously seen Message IDs. The topic used must be unique per logical repository (i.e. two routes de-duplicate using different repositories, and different topics) On startup, the instance consumes the full content of the topic, rebuilding the cache to the latest state.", annotations="interfaceName=org.apache.camel.spi.IdempotentRepository") @Configurer(metadataOnly=true) @ManagedResource(description="Kafka IdempotentRepository") public class KafkaIdempotentRepository extends org.apache.camel.support.service.ServiceSupport implements org.apache.camel.spi.IdempotentRepository, org.apache.camel.CamelContextAware
A Kafka topic-based implementation of IdempotentRepository. Uses a local cache of previously seen Message IDs. Mutations that come in via the (add(String)), or remove(String) method will update the local cache and broadcast the change in state on a Kafka topic to other instances. The cache is back-filled from the topic by a Kafka consumer. The topic used must be unique per logical repository (i.e. two routes de-duplicate using different repositories, and different topics). This class makes no assumptions about the number of partitions (it is designed to consume from all at the same time), or replication factor of the topic. Each repository instance that uses the topic (e.g. typically on different machines running in parallel) controls its own consumer group, so in a cluster of 10 Camel processes using the same topic each will control its own offset. On startup, the instance consumes the full content of the topic, rebuilding the cache to the latest state. To use, this repository must be placed in the Camel registry.

  • Constructor Details

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository()

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, String bootstrapServers)

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, String bootstrapServers, String groupId)

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, String bootstrapServers, int maxCacheSize, int pollDurationMs)

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig)

    • KafkaIdempotentRepository

      @Deprecated public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, String groupId)
      Deprecated.

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs)

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, String bootstrapServers, int maxCacheSize, int pollDurationMs, String groupId)

    • KafkaIdempotentRepository

      public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs, String groupId)

  • Method Details

    • getTopic

      public String getTopic()

    • setTopic

      public void setTopic(String topic)
      Sets the name of the Kafka topic used by this idempotent repository. Each functionally-separate repository should use a different topic.
      Parameters:
      topic - The topic name.

    • getBootstrapServers

      public String getBootstrapServers()

    • setBootstrapServers

      public void setBootstrapServers(String bootstrapServers)
      Sets the bootstrap.servers property on the internal Kafka producer and consumer. Use this as shorthand if not setting consumerConfig and producerConfig. If used, this component will apply sensible default configurations for the producer and consumer.

    • isStartupOnly

      public boolean isStartupOnly()

    • setStartupOnly

      public void setStartupOnly(boolean startupOnly)
      Whether to sync on startup only, or to continue syncing while Camel is running.

    • getProducerConfig

      public Properties getProducerConfig()

    • setProducerConfig

      public void setProducerConfig(Properties producerConfig)
      Sets the properties that will be used by the Kafka producer. Overrides bootstrapServers, so must define the 
       bootstrap.servers

      property itself. Prefer using bootstrapServers for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.

      Parameters:
      producerConfig - The producer configuration properties.

    • getConsumerConfig

      public Properties getConsumerConfig()

    • setConsumerConfig

      public void setConsumerConfig(Properties consumerConfig)
      Sets the properties that will be used by the Kafka consumer. Overrides bootstrapServers, so must define the 
       bootstrap.servers

      property itself. Prefer using bootstrapServers for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.

      Parameters:
      consumerConfig - The consumer configuration properties.

    • getMaxCacheSize

      public int getMaxCacheSize()

    • setMaxCacheSize

      public void setMaxCacheSize(int maxCacheSize)
      Sets the maximum size of the local key cache.
      Parameters:
      maxCacheSize - The maximum key cache size.

    • getPollDurationMs

      public int getPollDurationMs()

    • setPollDurationMs

      public void setPollDurationMs(int pollDurationMs)
      Sets the poll duration of the Kafka consumer. The local caches are updated immediately; this value will affect how far behind other peers in the cluster are, which are updating their caches from the topic, relative to the idempotent consumer instance issued the cache action message. The default value of this is DEFAULT_POLL_DURATION_MS. If setting this value explicitly, be aware that there is a tradeoff between the remote cache liveness and the volume of network traffic between this repository's consumer and the Kafka brokers. The cache warmup process also depends on there being one poll that fetches nothing - this indicates that the stream has been consumed up to the current point. If the poll duration is excessively long for the rate at which messages are sent on the topic, there exists a possibility that the cache cannot be warmed up and will operate in an inconsistent state relative to its peers until it catches up.
      Parameters:
      pollDurationMs - The poll duration in milliseconds.

    • getGroupId

      public String getGroupId()

    • setGroupId

      public void setGroupId(String groupId)
      A string that uniquely identifies the group of consumer processes to which this consumer belongs. By setting the same group id, multiple processes can indicate that they are all part of the same consumer group.

    • setCamelContext

      public void setCamelContext(org.apache.camel.CamelContext camelContext)
      Specified by:
      setCamelContext in interface org.apache.camel.CamelContextAware

    • getCamelContext

      public org.apache.camel.CamelContext getCamelContext()
      Specified by:
      getCamelContext in interface org.apache.camel.spi.HasCamelContext

    • doStart

      protected void doStart() throws Exception
      Overrides:
      doStart in class org.apache.camel.support.service.BaseService
      Throws:
      Exception

    • doStop

      protected void doStop() throws Exception
      Overrides:
      doStop in class org.apache.camel.support.service.BaseService
      Throws:
      Exception

    • add

      public boolean add(String key)
      Specified by:
      add in interface org.apache.camel.spi.IdempotentRepository

    • contains

      @ManagedOperation(description="Does the store contain the given key") public boolean contains(String key)
      Specified by:
      contains in interface org.apache.camel.spi.IdempotentRepository

    • remove

      @ManagedOperation(description="Remove the key from the store") public boolean remove(String key)
      Specified by:
      remove in interface org.apache.camel.spi.IdempotentRepository

    • confirm

      public boolean confirm(String key)
      Specified by:
      confirm in interface org.apache.camel.spi.IdempotentRepository

    • clear

      public void clear()
      Specified by:
      clear in interface org.apache.camel.spi.IdempotentRepository

    • getCacheCounter

      @ManagedOperation(description="Number of sync events received from the kafka topic") public long getCacheCounter()

    • getCacheSize

      @ManagedOperation(description="Number of elements currently in the cache") public long getCacheSize()