Package org.apache.druid.segment.realtime.appenderator

Class BatchAppenderatorDriver

java.lang.Object

org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver

org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver

All Implemented Interfaces:

Closeable, AutoCloseable

public class BatchAppenderatorDriver
extends BaseAppenderatorDriver

This class is specifialized for batch ingestion. In batch ingestion, the segment lifecycle is like:

 APPENDING -> PUSHED_AND_DROPPED -> PUBLISHED
 

• APPENDING: Segment is available for appending.
• PUSHED_AND_DROPPED: Segment is pushed to deep storage and dropped from the local storage.
• PUBLISHED: Segment's metadata is published to metastore.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver

BaseAppenderatorDriver.SegmentsForSequence

Field Summary

Fields inherited from class org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver

appenderator, executor, segments

Constructor Summary

Constructors

Constructor	Description
BatchAppenderatorDriver(Appenderator appenderator, SegmentAllocator segmentAllocator, PublishedSegmentRetriever segmentRetriever, DataSegmentKiller dataSegmentKiller)	Creates a driver for batch ingestion.

Method Summary

Modifier and Type	Method	Description
AppenderatorDriverAddResult	add(InputRow row, String sequenceName)	Add a row.
com.google.common.util.concurrent.ListenableFuture<SegmentsAndCommitMetadata>	publishAll(Set<DataSegment> segmentsToBeOverwritten, Set<DataSegment> tombstones, TransactionalSegmentPublisher publisher, Function<Set<DataSegment>,Set<DataSegment>> outputSegmentsAnnotateFunction, SegmentSchemaMapping segmentSchemaMapping)	Publish all segments.
SegmentsAndCommitMetadata	pushAllAndClear(long pushAndClearTimeoutMs)	Push and drop all segments in the SegmentWithState.SegmentState.APPENDING state.
Object	startJob()
Object	startJob(AppenderatorDriverSegmentLockHelper lockHelper)	This method always returns null because batch ingestion doesn't support restoring tasks on failures.

Methods inherited from class org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver

append, clear, close, getSegments

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BatchAppenderatorDriver

public BatchAppenderatorDriver(Appenderator appenderator,
 SegmentAllocator segmentAllocator,
 PublishedSegmentRetriever segmentRetriever,
 DataSegmentKiller dataSegmentKiller)

Creates a driver for batch ingestion.

Method Details

startJob

@Nullable
public Object startJob()

startJob

@Nullable
public Object startJob(AppenderatorDriverSegmentLockHelper lockHelper)

This method always returns null because batch ingestion doesn't support restoring tasks on failures.

Specified by:

startJob in class BaseAppenderatorDriver

Returns:

always null

add

public AppenderatorDriverAddResult add(InputRow row,
 String sequenceName)
                                throws IOException

Add a row. Must not be called concurrently from multiple threads.

Parameters:

row - the row to add

sequenceName - sequenceName for this row's segment

Returns:

AppenderatorDriverAddResult

Throws:

IOException - if there is an I/O error while allocating or writing to a segment

pushAllAndClear

public SegmentsAndCommitMetadata pushAllAndClear(long pushAndClearTimeoutMs)
                                          throws InterruptedException,
ExecutionException,
TimeoutException

Push and drop all segments in the SegmentWithState.SegmentState.APPENDING state.

Parameters:

pushAndClearTimeoutMs - timeout for pushing and dropping segments

Returns:

SegmentsAndCommitMetadata for pushed and dropped segments

Throws:

InterruptedException

ExecutionException

TimeoutException

publishAll

public com.google.common.util.concurrent.ListenableFuture<SegmentsAndCommitMetadata> publishAll(@Nullable
 Set<DataSegment> segmentsToBeOverwritten,
 @Nullable
 Set<DataSegment> tombstones,
 TransactionalSegmentPublisher publisher,
 Function<Set<DataSegment>,Set<DataSegment>> outputSegmentsAnnotateFunction,
 SegmentSchemaMapping segmentSchemaMapping)

Publish all segments.

Parameters:

segmentsToBeOverwritten - segments which can be overwritten by new segments published by the given publisher

publisher - segment publisher

Returns:

a ListenableFuture for the publish task