org.apache.druid.indexing.kafka

Class IncrementalPublishingKafkaIndexTaskRunner

java.lang.Object

org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner<Integer,Long>

org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner

All Implemented Interfaces:

org.apache.druid.segment.realtime.firehose.ChatHandler

public class IncrementalPublishingKafkaIndexTaskRunner
extends SeekableStreamIndexTaskRunner<Integer,Long>

Kafka indexing task runner supporting incremental segments publishing

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner

SeekableStreamIndexTaskRunner.Status

Field Summary

Fields inherited from class org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner

isAwaitingRetry, pauseRequested, pollRetryLock, stopRequested

Constructor Summary

Constructors

Constructor and Description
IncrementalPublishingKafkaIndexTaskRunner(KafkaIndexTask task, InputRowParser<ByteBuffer> parser, org.apache.druid.server.security.AuthorizerMapper authorizerMapper, com.google.common.base.Optional<org.apache.druid.segment.realtime.firehose.ChatHandlerProvider> chatHandlerProvider, CircularBuffer<Throwable> savedParseExceptions, RowIngestionMetersFactory rowIngestionMetersFactory)

Method Summary

Modifier and Type	Method and Description
protected SeekableStreamDataSourceMetadata<Integer,Long>	createDataSourceMetadata(SeekableStreamSequenceNumbers<Integer,Long> partitions)
protected OrderedSequenceNumber<Long>	createSequenceNumber(Long sequenceNumber)
protected SeekableStreamEndSequenceNumbers<Integer,Long>	deserializePartitionsFromMetadata(com.fasterxml.jackson.databind.ObjectMapper mapper, Object object)
protected TreeMap<Integer,Map<Integer,Long>>	getCheckPointsFromContext(TaskToolbox toolbox, String checkpointsString)
protected Long	getNextStartOffset(Long sequenceNumber)
protected List<OrderedPartitionableRecord<Integer,Long>>	getRecords(RecordSupplier<Integer,Long> recordSupplier, TaskToolbox toolbox)
com.fasterxml.jackson.core.type.TypeReference<List<SequenceMetadata<Integer,Long>>>	getSequenceMetadataTypeReference()
protected boolean	isEndOffsetExclusive()
protected boolean	isEndOfShard(Long seqNum)
protected void	possiblyResetDataSourceMetadata(TaskToolbox toolbox, RecordSupplier<Integer,Long> recordSupplier, Set<StreamPartition<Integer>> assignment)

Methods inherited from class org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner

getAppenderator, getCheckpointsHTTP, getCurrentOffsets, getCurrentOffsets, getEndOffsets, getEndOffsetsHTTP, getRowIngestionMeters, getRowStats, getSequences, getStartTime, getStatus, getStatusHTTP, getUnparseableEvents, initializeSequences, pause, pauseHTTP, resume, resumeHTTP, run, sendResetRequestAndWait, setEndOffsets, setEndOffsetsHTTP, setToolbox, stop, stopGracefully

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IncrementalPublishingKafkaIndexTaskRunner

public IncrementalPublishingKafkaIndexTaskRunner(KafkaIndexTask task,
                                                 InputRowParser<ByteBuffer> parser,
                                                 org.apache.druid.server.security.AuthorizerMapper authorizerMapper,
                                                 com.google.common.base.Optional<org.apache.druid.segment.realtime.firehose.ChatHandlerProvider> chatHandlerProvider,
                                                 CircularBuffer<Throwable> savedParseExceptions,
                                                 RowIngestionMetersFactory rowIngestionMetersFactory)

Method Detail

getNextStartOffset

protected Long getNextStartOffset(@NotNull
                                  Long sequenceNumber)

Specified by:

getNextStartOffset in class SeekableStreamIndexTaskRunner<Integer,Long>

getRecords

@Nonnull
protected List<OrderedPartitionableRecord<Integer,Long>> getRecords(RecordSupplier<Integer,Long> recordSupplier,
                                                                             TaskToolbox toolbox)
                                                                      throws Exception

Specified by:

getRecords in class SeekableStreamIndexTaskRunner<Integer,Long>

Throws:

Exception

deserializePartitionsFromMetadata

protected SeekableStreamEndSequenceNumbers<Integer,Long> deserializePartitionsFromMetadata(com.fasterxml.jackson.databind.ObjectMapper mapper,
                                                                                           Object object)

Specified by:

deserializePartitionsFromMetadata in class SeekableStreamIndexTaskRunner<Integer,Long>

createDataSourceMetadata

protected SeekableStreamDataSourceMetadata<Integer,Long> createDataSourceMetadata(SeekableStreamSequenceNumbers<Integer,Long> partitions)

Specified by:

createDataSourceMetadata in class SeekableStreamIndexTaskRunner<Integer,Long>

createSequenceNumber

protected OrderedSequenceNumber<Long> createSequenceNumber(Long sequenceNumber)

Specified by:

createSequenceNumber in class SeekableStreamIndexTaskRunner<Integer,Long>

possiblyResetDataSourceMetadata

protected void possiblyResetDataSourceMetadata(TaskToolbox toolbox,
                                               RecordSupplier<Integer,Long> recordSupplier,
                                               Set<StreamPartition<Integer>> assignment)

Specified by:

possiblyResetDataSourceMetadata in class SeekableStreamIndexTaskRunner<Integer,Long>

isEndOffsetExclusive

protected boolean isEndOffsetExclusive()

Specified by:

isEndOffsetExclusive in class SeekableStreamIndexTaskRunner<Integer,Long>

isEndOfShard

protected boolean isEndOfShard(Long seqNum)

Specified by:

isEndOfShard in class SeekableStreamIndexTaskRunner<Integer,Long>

getSequenceMetadataTypeReference

public com.fasterxml.jackson.core.type.TypeReference<List<SequenceMetadata<Integer,Long>>> getSequenceMetadataTypeReference()

Specified by:

getSequenceMetadataTypeReference in class SeekableStreamIndexTaskRunner<Integer,Long>

getCheckPointsFromContext

@Nullable
protected TreeMap<Integer,Map<Integer,Long>> getCheckPointsFromContext(TaskToolbox toolbox,
                                                                                 String checkpointsString)
                                                                          throws IOException

Specified by:

getCheckPointsFromContext in class SeekableStreamIndexTaskRunner<Integer,Long>

Throws:

IOException