Package org.apache.druid.metadata

Class IndexerSQLMetadataStorageCoordinator

java.lang.Object

org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator

All Implemented Interfaces:

IndexerMetadataStorageCoordinator

public class IndexerSQLMetadataStorageCoordinator
extends Object
implements IndexerMetadataStorageCoordinator

Constructor Summary

Constructors

Constructor	Description
IndexerSQLMetadataStorageCoordinator(SegmentMetadataTransactionFactory transactionFactory, com.fasterxml.jackson.databind.ObjectMapper jsonMapper, MetadataStorageTablesConfig dbTables, SQLMetadataConnector connector, SegmentSchemaManager segmentSchemaManager, CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)

Method Summary

Modifier and Type	Method	Description
SegmentIdWithShardSpec	allocatePendingSegment(String dataSource, org.joda.time.Interval interval, boolean skipSegmentLineageCheck, SegmentCreateRequest createRequest)	Allocate a new pending segment in the pending segments table.
Map<SegmentCreateRequest,SegmentIdWithShardSpec>	allocatePendingSegments(String dataSource, org.joda.time.Interval allocateInterval, boolean skipSegmentLineageCheck, List<SegmentCreateRequest> requests, boolean reduceMetadataIO)	Allocates pending segments for the given requests in the pending segments table.
SegmentPublishResult	commitAppendSegments(Set<DataSegment> appendSegments, Map<DataSegment,ReplaceTaskLock> appendSegmentToReplaceLock, String taskAllocatorId, SegmentSchemaMapping segmentSchemaMapping)	Commits segments and corresponding schema created by an APPEND task.
SegmentPublishResult	commitAppendSegmentsAndMetadata(Set<DataSegment> appendSegments, Map<DataSegment,ReplaceTaskLock> appendSegmentToReplaceLock, String supervisorId, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata, String taskAllocatorId, SegmentSchemaMapping segmentSchemaMapping)	Commits segments created by an APPEND task.
SegmentPublishResult	commitMetadataOnly(String supervisorId, String dataSource, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata)	Similar to IndexerMetadataStorageCoordinator.commitSegments(java.util.Set<org.apache.druid.timeline.DataSegment>, org.apache.druid.segment.SegmentSchemaMapping), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made.
SegmentPublishResult	commitReplaceSegments(Set<DataSegment> replaceSegments, Set<ReplaceTaskLock> locksHeldByReplaceTask, SegmentSchemaMapping segmentSchemaMapping)	Commits segments and corresponding schema created by a REPLACE task.
Set<DataSegment>	commitSegments(Set<DataSegment> segments, SegmentSchemaMapping segmentSchemaMapping)	Attempts to insert a set of segments and corresponding schema to the metadata storage.
SegmentPublishResult	commitSegmentsAndMetadata(Set<DataSegment> segments, String supervisorId, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata, SegmentSchemaMapping segmentSchemaMapping)	Attempts to insert a set of segments and corresponding schema to the metadata storage.
boolean	deleteDataSourceMetadata(String supervisorId)	Removes entry for supervisorId from the dataSource metadata table.
int	deletePendingSegments(String dataSource)	Delete all pending segments belonging to the given data source from the pending segments table.
int	deletePendingSegmentsCreatedInInterval(String dataSource, org.joda.time.Interval deleteInterval)	Delete pending segments created in the given interval belonging to the given data source from the pending segments table.
int	deletePendingSegmentsForTaskAllocatorId(String datasource, String taskAllocatorId)	Delete pending segment for a give task group after all the tasks belonging to it have completed.
int	deleteSegments(Set<DataSegment> segments)	Deletes unused segments from the metadata store.
int	deleteUpgradeSegmentsForTask(String taskId)	Delete entries from the upgrade segments table after the corresponding replace task has ended
List<PendingSegmentRecord>	getPendingSegments(String datasource, org.joda.time.Interval interval)	Fetches all the pending segments of the datasource that overlap with a given interval.
SegmentTimeline	getSegmentTimelineForAllocation(String dataSource, org.joda.time.Interval interval, boolean reduceMetadataIO)	Return a segment timeline of all used segments including overshadowed ones for a given datasource and interval if skipSegmentPayloadFetchForAllocation is set to true, do not fetch all the segment payloads for allocation Instead fetch all the ids and numCorePartitions using exactly one segment per version per interval return a dummy DataSegment for each id that holds only the SegmentId and a NumberedShardSpec with numCorePartitions
List<org.joda.time.Interval>	getUnusedSegmentIntervals(String dataSource, org.joda.time.DateTime minStartTime, org.joda.time.DateTime maxEndTime, int limit, org.joda.time.DateTime maxUsedStatusLastUpdatedTime)	Returns a list of up to limit unused segment intervals for the specified datasource.
boolean	insertDataSourceMetadata(String supervisorId, DataSourceMetadata metadata)	Insert DataSourceMetadata entry for supervisorId.
protected Set<DataSegment>	insertSegments(SegmentMetadataTransaction transaction, Set<DataSegment> segments, SegmentSchemaMapping segmentSchemaMapping)
List<DataSegmentPlus>	iterateAllUnusedSegmentsForDatasource(String datasource, org.joda.time.Interval interval, Integer limit, String lastSegmentId, SortOrder sortOrder)	Returns a list of unused segments and their associated metadata for a given datasource over an optional interval.
int	markAllNonOvershadowedSegmentsAsUsed(String dataSource)	Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as used).
int	markAllSegmentsAsUnused(String dataSource)	Marks all the segments in given datasource as unused.
int	markNonOvershadowedSegmentsAsUsed(String dataSource, Set<SegmentId> segmentIds)	Marks the given segment IDs as "used" only if there are not already overshadowed by other used segments.
int	markNonOvershadowedSegmentsAsUsed(String dataSource, org.joda.time.Interval interval, List<String> versions)	Marks non-overshadowed unused segments for the given interval and optional list of versions as used.
boolean	markSegmentAsUnused(SegmentId segmentId)	Marks the segment as unused.
boolean	markSegmentAsUsed(SegmentId segmentId)	Returns true if the state of the segment entry is changed in the database as the result of this call (that is, the segment was marked as used), false otherwise.
int	markSegmentsAsUnused(String dataSource, Set<SegmentId> segmentIds)	Marks the given segments as unused.
int	markSegmentsWithinIntervalAsUnused(String dataSource, org.joda.time.Interval interval, List<String> versions)	Marks segments that are fully contained in the given interval as unused.
int	removeDataSourceMetadataOlderThan(long timestamp, @NotNull Set<String> excludeSupervisorIds)	Remove supervisors' datasource metadata created before the given timestamp and not in given excludeSupervisorIds set.
boolean	resetDataSourceMetadata(String supervisorId, DataSourceMetadata dataSourceMetadata)	Resets DataSourceMetadata entry for supervisorId to the one supplied.
Set<String>	retrieveAllDatasourceNames()
Set<DataSegment>	retrieveAllUsedSegments(String dataSource, Segments visibility)	Retrieves all published used segments for the given data source.
DataSourceMetadata	retrieveDataSourceMetadata(String supervisorId)	Read dataSource metadata for the given supervisorId.
DataSegment	retrieveSegmentForId(SegmentId segmentId)	Retrieve the segment for a given id from the metadata store.
Set<DataSegment>	retrieveSegmentsById(String dataSource, Set<String> segmentIds)	Retrieves segments for the given IDs, regardless of their visibility (visible, overshadowed or unused).
List<org.joda.time.Interval>	retrieveUnusedSegmentIntervals(String dataSource, int limit)	Retrieves intervals of the specified datasource that contain any unused segments.
List<DataSegment>	retrieveUnusedSegmentsForInterval(String dataSource, org.joda.time.Interval interval, List<String> versions, Integer limit, org.joda.time.DateTime maxUsedStatusLastUpdatedTime)	Retrieve all published segments which include ONLY data within the given interval and are marked as unused from the metadata store.
List<DataSegment>	retrieveUnusedSegmentsWithExactInterval(String dataSource, org.joda.time.Interval interval, org.joda.time.DateTime maxUpdatedTime, int limit)	Retrieves unused segments from the metadata store that match the given interval exactly.
Map<String,String>	retrieveUpgradedFromSegmentIds(String dataSource, Set<String> segmentIds)	Map from a segment ID to the segment ID from which it was upgraded There should be no entry in the map for an original non-upgraded segment
Map<String,Set<String>>	retrieveUpgradedToSegmentIds(String dataSource, Set<String> segmentIds)	Map from a segment ID to a set containing 1) all segment IDs that were upgraded from it AND are still present in the metadata store 2) the segment ID itself if and only if it is still present in the metadata store
DataSegment	retrieveUsedSegmentForId(SegmentId segmentId)
List<Pair<DataSegment,String>>	retrieveUsedSegmentsAndCreatedDates(String dataSource, List<org.joda.time.Interval> intervals)	Retrieve all published segments which are marked as used and the created_date of these segments belonging to the given data source and list of intervals from the metadata store.
Set<DataSegment>	retrieveUsedSegmentsForIntervals(String dataSource, List<org.joda.time.Interval> intervals, Segments visibility)	Retrieves all published segments that have partial or complete overlap with the given intervals and are marked as used.
void	start()
protected SegmentPublishResult	updateDataSourceMetadataInTransaction(SegmentMetadataTransaction transaction, String supervisorId, String dataSource, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata)	Compare-and-swap DataSourceMetadata for a datasource in a transaction.
void	updateSegmentMetadata(Set<DataSegment> segments)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator

retrieveUnusedSegmentsForInterval, retrieveUsedSegmentsForInterval

Constructor Details

IndexerSQLMetadataStorageCoordinator

@Inject
public IndexerSQLMetadataStorageCoordinator(SegmentMetadataTransactionFactory transactionFactory,
 com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
 MetadataStorageTablesConfig dbTables,
 SQLMetadataConnector connector,
 SegmentSchemaManager segmentSchemaManager,
 CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)

Method Details

start

public void start()

retrieveAllDatasourceNames

public Set<String> retrieveAllDatasourceNames()

Specified by:

retrieveAllDatasourceNames in interface IndexerMetadataStorageCoordinator

Returns:

Set of all datasource names for which there are used or unused segments present in the metadata store.

retrieveUnusedSegmentIntervals

public List<org.joda.time.Interval> retrieveUnusedSegmentIntervals(String dataSource,
 int limit)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieves intervals of the specified datasource that contain any unused segments. There is no guarantee on the order of intervals in the list or on whether the limited list contains the earliest or latest intervals of the datasource.

Specified by:

retrieveUnusedSegmentIntervals in interface IndexerMetadataStorageCoordinator

Returns:

Unsorted list of unused segment intervals containing upto limit entries.

retrieveUsedSegmentsForIntervals

public Set<DataSegment> retrieveUsedSegmentsForIntervals(String dataSource,
 List<org.joda.time.Interval> intervals,
 Segments visibility)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieves all published segments that have partial or complete overlap with the given intervals and are marked as used.

Specified by:

retrieveUsedSegmentsForIntervals in interface IndexerMetadataStorageCoordinator

retrieveAllUsedSegments

public Set<DataSegment> retrieveAllUsedSegments(String dataSource,
 Segments visibility)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieves all published used segments for the given data source.

Specified by:

retrieveAllUsedSegments in interface IndexerMetadataStorageCoordinator

See Also:

• IndexerMetadataStorageCoordinator.retrieveUsedSegmentsForInterval(String, Interval, Segments)

retrieveUsedSegmentsAndCreatedDates

public List<Pair<DataSegment,String>> retrieveUsedSegmentsAndCreatedDates(String dataSource,
 List<org.joda.time.Interval> intervals)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieve all published segments which are marked as used and the created_date of these segments belonging to the given data source and list of intervals from the metadata store.

Unlike other similar methods in this interface, this method doesn't accept a Segments "visibility" parameter. The returned collection may include overshadowed segments and their created_dates, as if Segments.INCLUDING_OVERSHADOWED was passed. It's the responsibility of the caller to filter out overshadowed ones if needed.

Specified by:

retrieveUsedSegmentsAndCreatedDates in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - The data source to query

intervals - The list of interval to query

Returns:

The DataSegments and the related created_date of segments

retrieveUnusedSegmentsForInterval

public List<DataSegment> retrieveUnusedSegmentsForInterval(String dataSource,
 org.joda.time.Interval interval,
 @Nullable
 List<String> versions,
 @Nullable
 Integer limit,
 @Nullable
 org.joda.time.DateTime maxUsedStatusLastUpdatedTime)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieve all published segments which include ONLY data within the given interval and are marked as unused from the metadata store.

Specified by:

retrieveUnusedSegmentsForInterval in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - The data source the segments belong to

interval - Filter the data segments to ones that include data in this interval exclusively.

versions - An optional list of segment versions to retrieve in the given interval. If unspecified, all versions of unused segments in the interval must be retrieved. If an empty list is passed, no segments are retrieved.

limit - The maximum number of unused segments to retreive. If null, no limit is applied.

maxUsedStatusLastUpdatedTime - The maximum used_status_last_updated time. Any unused segment in interval with used_status_last_updated no later than this time will be included in the kill task. Segments without used_status_last_updated time (due to an upgrade from legacy Druid) will have maxUsedStatusLastUpdatedTime ignored

Returns:

DataSegments which include ONLY data within the requested interval and are marked as unused. Segments NOT returned here may include data in the interval

retrieveUnusedSegmentsWithExactInterval

public List<DataSegment> retrieveUnusedSegmentsWithExactInterval(String dataSource,
 org.joda.time.Interval interval,
 org.joda.time.DateTime maxUpdatedTime,
 int limit)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieves unused segments from the metadata store that match the given interval exactly. There is no guarantee on the order of segments in the list or on whether the limited list contains the highest or lowest segment IDs in the interval.

Specified by:

retrieveUnusedSegmentsWithExactInterval in interface IndexerMetadataStorageCoordinator

interval - Returned segments must exactly match this interval.

maxUpdatedTime - Returned segments must have a used_status_last_updated which is either null or earlier than this value.

limit - Maximum number of segments to return.

Returns:

Unsorted list of unused segments that match the given parameters.

retrieveSegmentsById

public Set<DataSegment> retrieveSegmentsById(String dataSource,
 Set<String> segmentIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieves segments for the given IDs, regardless of their visibility (visible, overshadowed or unused).

Specified by:

retrieveSegmentsById in interface IndexerMetadataStorageCoordinator

markSegmentsWithinIntervalAsUnused

public int markSegmentsWithinIntervalAsUnused(String dataSource,
 org.joda.time.Interval interval,
 @Nullable
 List<String> versions)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks segments that are fully contained in the given interval as unused.

Specified by:

markSegmentsWithinIntervalAsUnused in interface IndexerMetadataStorageCoordinator

versions - Optional list of segment versions eligible for update. If this list is passed as null, all segment versions are eligible for updated. If passed as empty, no segment is updated.

Returns:

Number of segments updated

markSegmentAsUnused

public boolean markSegmentAsUnused(SegmentId segmentId)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks the segment as unused.

Specified by:

markSegmentAsUnused in interface IndexerMetadataStorageCoordinator

Returns:

true if the segment was updated, false otherwise

markSegmentsAsUnused

public int markSegmentsAsUnused(String dataSource,
 Set<SegmentId> segmentIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks the given segments as unused.

Specified by:

markSegmentsAsUnused in interface IndexerMetadataStorageCoordinator

Returns:

Number of segments updated

markAllSegmentsAsUnused

public int markAllSegmentsAsUnused(String dataSource)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks all the segments in given datasource as unused.

Specified by:

markAllSegmentsAsUnused in interface IndexerMetadataStorageCoordinator

Returns:

Number of updated segments

markSegmentAsUsed

public boolean markSegmentAsUsed(SegmentId segmentId)

Description copied from interface: IndexerMetadataStorageCoordinator

Returns true if the state of the segment entry is changed in the database as the result of this call (that is, the segment was marked as used), false otherwise. If the call results in a database error, an exception is relayed to the caller.

Specified by:

markSegmentAsUsed in interface IndexerMetadataStorageCoordinator

markNonOvershadowedSegmentsAsUsed

public int markNonOvershadowedSegmentsAsUsed(String dataSource,
 Set<SegmentId> segmentIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks the given segment IDs as "used" only if there are not already overshadowed by other used segments. Qualifying segment IDs that are already marked as "used" are not updated.

Specified by:

markNonOvershadowedSegmentsAsUsed in interface IndexerMetadataStorageCoordinator

Returns:

Number of segments updated

markNonOvershadowedSegmentsAsUsed

public int markNonOvershadowedSegmentsAsUsed(String dataSource,
 org.joda.time.Interval interval,
 @Nullable
 List<String> versions)

Description copied from interface: IndexerMetadataStorageCoordinator

Marks non-overshadowed unused segments for the given interval and optional list of versions as used. If versions are not specified, all versions of non-overshadowed unused segments in the interval will be marked as used. If an empty list of versions is passed, no segments are marked as used.

Specified by:

markNonOvershadowedSegmentsAsUsed in interface IndexerMetadataStorageCoordinator

Returns:

Number of segments updated in the metadata store

markAllNonOvershadowedSegmentsAsUsed

public int markAllNonOvershadowedSegmentsAsUsed(String dataSource)

Description copied from interface: IndexerMetadataStorageCoordinator

Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as used). If the call results in a database error, an exception is relayed to the caller.

Specified by:

markAllNonOvershadowedSegmentsAsUsed in interface IndexerMetadataStorageCoordinator

Returns:

Number of segments updated in the metadata store

iterateAllUnusedSegmentsForDatasource

public List<DataSegmentPlus> iterateAllUnusedSegmentsForDatasource(String datasource,
 @Nullable
 org.joda.time.Interval interval,
 @Nullable
 Integer limit,
 @Nullable
 String lastSegmentId,
 @Nullable
 SortOrder sortOrder)

Description copied from interface: IndexerMetadataStorageCoordinator

Returns a list of unused segments and their associated metadata for a given datasource over an optional interval. The order in which segments are iterated is from earliest start-time, with ties being broken with earliest end-time first. Note: the iteration may not be as trivially cheap as for example, iteration over an ArrayList. Try (to some reasonable extent) to organize the code so that it iterates the returned iterable only once rather than several times.

Specified by:

iterateAllUnusedSegmentsForDatasource in interface IndexerMetadataStorageCoordinator

Parameters:

datasource - the name of the datasource.

interval - an optional interval to search over. If none is specified, Intervals.ETERNITY

limit - an optional maximum number of results to return. If none is specified, the results are not limited.

lastSegmentId - an optional last segment id from which to search for results. All segments returned are > this segment lexigraphically if sortOrder is null or SortOrder.ASC, or < this segment lexigraphically if sortOrder is SortOrder.DESC. If none is specified, no such filter is used.

sortOrder - an optional order with which to return the matching segments by id, start time, end time. If none is specified, the order of the results is not guarenteed.

getUnusedSegmentIntervals

public List<org.joda.time.Interval> getUnusedSegmentIntervals(String dataSource,
 @Nullable
 org.joda.time.DateTime minStartTime,
 org.joda.time.DateTime maxEndTime,
 int limit,
 org.joda.time.DateTime maxUsedStatusLastUpdatedTime)

Description copied from interface: IndexerMetadataStorageCoordinator

Returns a list of up to limit unused segment intervals for the specified datasource. Segments are filtered based on the following criteria:

The start time of the segment must be no earlier than the specified minStartTime (if not null).

The end time of the segment must be no later than the specified maxEndTime.

The used_status_last_updated time of the segment must be no later than maxUsedStatusLastUpdatedTime. Segments that have no used_status_last_updated time (due to an upgrade from legacy Druid) will have maxUsedStatusLastUpdatedTime ignored.

Specified by:

getUnusedSegmentIntervals in interface IndexerMetadataStorageCoordinator

Returns:

list of intervals ordered by segment start time and then by end time. Note that the list may contain duplicate intervals.

commitSegments

public Set<DataSegment> commitSegments(Set<DataSegment> segments,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)

Description copied from interface: IndexerMetadataStorageCoordinator

Attempts to insert a set of segments and corresponding schema to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

Specified by:

commitSegments in interface IndexerMetadataStorageCoordinator

Parameters:

segments - set of segments to add

segmentSchemaMapping - segment schema information to add

Returns:

set of segments actually added

commitSegmentsAndMetadata

public SegmentPublishResult commitSegmentsAndMetadata(Set<DataSegment> segments,
 @Nullable
 String supervisorId,
 @Nullable
 DataSourceMetadata startMetadata,
 @Nullable
 DataSourceMetadata endMetadata,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)

Description copied from interface: IndexerMetadataStorageCoordinator

Attempts to insert a set of segments and corresponding schema to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

If startMetadata and endMetadata are set, this insertion will be atomic with a compare-and-swap on dataSource commit metadata.

If segmentsToDrop is not null and not empty, this insertion will be atomic with a insert-and-drop on inserting and dropping .

Specified by:

commitSegmentsAndMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

segments - set of segments to add, must all be from the same dataSource

supervisorId - supervisorID which is committing the segments. Cannot be null if startMetadata and endMetadata are both non-null.

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata). If null, this insert will not involve a metadata transaction

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata). If null, this insert will not involve a metadata transaction

segmentSchemaMapping - segment schema information to persist.

Returns:

segment publish result indicating transaction success or failure, and set of segments actually published. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

commitReplaceSegments

public SegmentPublishResult commitReplaceSegments(Set<DataSegment> replaceSegments,
 Set<ReplaceTaskLock> locksHeldByReplaceTask,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)

Description copied from interface: IndexerMetadataStorageCoordinator

Commits segments and corresponding schema created by a REPLACE task. This method also handles the segment upgrade scenarios that may result from concurrent append and replace.

• If an APPEND task committed a segment to an interval locked by this task, the append segment is upgraded to the version of the corresponding lock. This is done with the help of entries created in the upgrade_segments table in IndexerMetadataStorageCoordinator.commitAppendSegments(java.util.Set<org.apache.druid.timeline.DataSegment>, java.util.Map<org.apache.druid.timeline.DataSegment, org.apache.druid.metadata.ReplaceTaskLock>, java.lang.String, org.apache.druid.segment.SegmentSchemaMapping)

Specified by:

commitReplaceSegments in interface IndexerMetadataStorageCoordinator

Parameters:

replaceSegments - All segments created by a REPLACE task that must be committed in a single transaction.

locksHeldByReplaceTask - All active non-revoked REPLACE locks held by the task

segmentSchemaMapping - Segment schema to add.

commitAppendSegments

public SegmentPublishResult commitAppendSegments(Set<DataSegment> appendSegments,
 Map<DataSegment,ReplaceTaskLock> appendSegmentToReplaceLock,
 String taskAllocatorId,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)

Description copied from interface: IndexerMetadataStorageCoordinator

Commits segments and corresponding schema created by an APPEND task. This method also handles segment upgrade scenarios that may result from concurrent append and replace.

• If a REPLACE task committed a segment that overlaps with any of the appendSegments while this APPEND task was in progress, the appendSegments are upgraded to the version of the replace segment.
• If an appendSegment is covered by a currently active REPLACE lock, then an entry is created for it in the upgrade_segments table, so that when the REPLACE task finishes, it can upgrade the appendSegment as required.

Specified by:

commitAppendSegments in interface IndexerMetadataStorageCoordinator

Parameters:

appendSegments - All segments created by an APPEND task that must be committed in a single transaction.

appendSegmentToReplaceLock - Map from append segment to the currently active REPLACE lock (if any) covering it

taskAllocatorId - allocator id of the task committing the segments to be appended

segmentSchemaMapping - schema of append segments

commitAppendSegmentsAndMetadata

public SegmentPublishResult commitAppendSegmentsAndMetadata(Set<DataSegment> appendSegments,
 Map<DataSegment,ReplaceTaskLock> appendSegmentToReplaceLock,
 String supervisorId,
 DataSourceMetadata startMetadata,
 DataSourceMetadata endMetadata,
 String taskAllocatorId,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)

Description copied from interface: IndexerMetadataStorageCoordinator

Commits segments created by an APPEND task. This method also handles segment upgrade scenarios that may result from concurrent append and replace. Also commits start and end DataSourceMetadata.

Specified by:

commitAppendSegmentsAndMetadata in interface IndexerMetadataStorageCoordinator

See Also:

• IndexerMetadataStorageCoordinator.commitAppendSegments(java.util.Set<org.apache.druid.timeline.DataSegment>, java.util.Map<org.apache.druid.timeline.DataSegment, org.apache.druid.metadata.ReplaceTaskLock>, java.lang.String, org.apache.druid.segment.SegmentSchemaMapping)
• IndexerMetadataStorageCoordinator.commitSegmentsAndMetadata(java.util.Set<org.apache.druid.timeline.DataSegment>, java.lang.String, org.apache.druid.indexing.overlord.DataSourceMetadata, org.apache.druid.indexing.overlord.DataSourceMetadata, org.apache.druid.segment.SegmentSchemaMapping)

commitMetadataOnly

public SegmentPublishResult commitMetadataOnly(String supervisorId,
 String dataSource,
 DataSourceMetadata startMetadata,
 DataSourceMetadata endMetadata)

Description copied from interface: IndexerMetadataStorageCoordinator

Similar to IndexerMetadataStorageCoordinator.commitSegments(java.util.Set<org.apache.druid.timeline.DataSegment>, org.apache.druid.segment.SegmentSchemaMapping), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made.

The metadata should undergo the same validation checks as performed by IndexerMetadataStorageCoordinator.commitSegments(java.util.Set<org.apache.druid.timeline.DataSegment>, org.apache.druid.segment.SegmentSchemaMapping).

Specified by:

commitMetadataOnly in interface IndexerMetadataStorageCoordinator

Parameters:

supervisorId - the supervisorId

dataSource - the dataSource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata).

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata).

Returns:

segment publish result indicating transaction success or failure. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

allocatePendingSegments

public Map<SegmentCreateRequest,SegmentIdWithShardSpec> allocatePendingSegments(String dataSource,
 org.joda.time.Interval allocateInterval,
 boolean skipSegmentLineageCheck,
 List<SegmentCreateRequest> requests,
 boolean reduceMetadataIO)

Description copied from interface: IndexerMetadataStorageCoordinator

Allocates pending segments for the given requests in the pending segments table. The segment id allocated for a request will not be given out again unless a request is made with the same SegmentCreateRequest.

Specified by:

allocatePendingSegments in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - dataSource for which to allocate a segment

allocateInterval - interval for which to allocate a segment

skipSegmentLineageCheck - if true, perform lineage validation using previousSegmentId for this sequence. Should be set to false if replica tasks would index events in same order

requests - Requests for which to allocate segments. All the requests must share the same partition space.

reduceMetadataIO - If true, try to use the segment ids instead of fetching every segment payload from the metadata store

Returns:

Map from request to allocated segment id. The map does not contain entries for failed requests.

allocatePendingSegment

@Nullable
public SegmentIdWithShardSpec allocatePendingSegment(String dataSource,
 org.joda.time.Interval interval,
 boolean skipSegmentLineageCheck,
 SegmentCreateRequest createRequest)

Description copied from interface: IndexerMetadataStorageCoordinator

Allocate a new pending segment in the pending segments table. This segment identifier will never be given out again, unless another call is made with the same dataSource, sequenceName, and previousSegmentId.

The sequenceName and previousSegmentId parameters are meant to make it easy for two independent ingestion tasks to produce the same series of segments.

Note that a segment sequence may include segments with a variety of different intervals and versions.

Specified by:

allocatePendingSegment in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - dataSource for which to allocate a segment

interval - interval for which to allocate a segment

skipSegmentLineageCheck - if true, perform lineage validation using previousSegmentId for this sequence. Should be set to false if replica tasks would index events in same order

Returns:

the pending segment identifier, or null if it was impossible to allocate a new segment

getSegmentTimelineForAllocation

public SegmentTimeline getSegmentTimelineForAllocation(String dataSource,
 org.joda.time.Interval interval,
 boolean reduceMetadataIO)

Description copied from interface: IndexerMetadataStorageCoordinator

Return a segment timeline of all used segments including overshadowed ones for a given datasource and interval if skipSegmentPayloadFetchForAllocation is set to true, do not fetch all the segment payloads for allocation Instead fetch all the ids and numCorePartitions using exactly one segment per version per interval return a dummy DataSegment for each id that holds only the SegmentId and a NumberedShardSpec with numCorePartitions

Specified by:

getSegmentTimelineForAllocation in interface IndexerMetadataStorageCoordinator

deletePendingSegmentsCreatedInInterval

public int deletePendingSegmentsCreatedInInterval(String dataSource,
 org.joda.time.Interval deleteInterval)

Description copied from interface: IndexerMetadataStorageCoordinator

Delete pending segments created in the given interval belonging to the given data source from the pending segments table. The created_date field of the pending segments table is checked to find segments to be deleted.

Note that the semantic of the interval (for `created_date`s) is different from the semantic of the interval parameters in some other methods in this class, such as IndexerMetadataStorageCoordinator.retrieveUsedSegmentsForInterval(java.lang.String, org.joda.time.Interval, org.apache.druid.indexing.overlord.Segments) (where the interval is about the time column value in rows belonging to the segment).

Specified by:

deletePendingSegmentsCreatedInInterval in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - dataSource

deleteInterval - interval to check the created_date of pendingSegments

Returns:

number of deleted pending segments

deletePendingSegments

public int deletePendingSegments(String dataSource)

Description copied from interface: IndexerMetadataStorageCoordinator

Delete all pending segments belonging to the given data source from the pending segments table.

Specified by:

deletePendingSegments in interface IndexerMetadataStorageCoordinator

Returns:

number of deleted pending segments

See Also:

• similar to this method but also accepts interval for segments' `created_date`s

insertSegments

protected Set<DataSegment> insertSegments(SegmentMetadataTransaction transaction,
 Set<DataSegment> segments,
 @Nullable
 SegmentSchemaMapping segmentSchemaMapping)
                                   throws Exception

Throws:

Exception

retrieveDataSourceMetadata

@Nullable
public DataSourceMetadata retrieveDataSourceMetadata(String supervisorId)

Read dataSource metadata for the given supervisorId. Returns null if there is no metadata.

Specified by:

retrieveDataSourceMetadata in interface IndexerMetadataStorageCoordinator

updateDataSourceMetadataInTransaction

protected SegmentPublishResult updateDataSourceMetadataInTransaction(SegmentMetadataTransaction transaction,
 String supervisorId,
 String dataSource,
 DataSourceMetadata startMetadata,
 DataSourceMetadata endMetadata)
                                                              throws IOException

Compare-and-swap DataSourceMetadata for a datasource in a transaction. This method updates the metadata for the given datasource only if the currently persisted entry matches the startMetadata. If the current entry in the DB matches the endMetadata, this method returns immediately with success.

Parameters:

supervisorId - The supervisor ID. Used as the PK for the corresponding metadata entry in the DB.

dataSource - The dataSource. Currently used only for logging purposes.

startMetadata - Current entry in the DB must match this value.

endMetadata - The updated entry will be equal to the current entry plus this value.

Returns:

Successful SegmentPublishResult if the metadata in the DB was successfully updated, or if the current entry already matches endMetadata. Otherwise, returns a failed SegmentPublishResult (retryable or not).

Throws:

IOException - if an error occurred while serializing or deserializing.

NullPointerException - if any of the arguments is null.

deleteDataSourceMetadata

public boolean deleteDataSourceMetadata(String supervisorId)

Description copied from interface: IndexerMetadataStorageCoordinator

Removes entry for supervisorId from the dataSource metadata table.

Specified by:

deleteDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

supervisorId - identifier

Returns:

true if the entry was deleted, false otherwise

resetDataSourceMetadata

public boolean resetDataSourceMetadata(String supervisorId,
 DataSourceMetadata dataSourceMetadata)
                                throws IOException

Description copied from interface: IndexerMetadataStorageCoordinator

Resets DataSourceMetadata entry for supervisorId to the one supplied.

Specified by:

resetDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

supervisorId - identifier

dataSourceMetadata - value to set

Returns:

true if the entry was reset, false otherwise

Throws:

IOException

updateSegmentMetadata

public void updateSegmentMetadata(Set<DataSegment> segments)

Specified by:

updateSegmentMetadata in interface IndexerMetadataStorageCoordinator

deleteSegments

public int deleteSegments(Set<DataSegment> segments)

Description copied from interface: IndexerMetadataStorageCoordinator

Deletes unused segments from the metadata store.

Specified by:

deleteSegments in interface IndexerMetadataStorageCoordinator

Returns:

Number of segments actually deleted.

insertDataSourceMetadata

public boolean insertDataSourceMetadata(String supervisorId,
 DataSourceMetadata metadata)

Description copied from interface: IndexerMetadataStorageCoordinator

Insert DataSourceMetadata entry for supervisorId.

Specified by:

insertDataSourceMetadata in interface IndexerMetadataStorageCoordinator

Parameters:

supervisorId - identifier

metadata - value to set

Returns:

true if the entry was inserted, false otherwise

removeDataSourceMetadataOlderThan

public int removeDataSourceMetadataOlderThan(long timestamp,
 @NotNull
 @NotNull Set<String> excludeSupervisorIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Remove supervisors' datasource metadata created before the given timestamp and not in given excludeSupervisorIds set.

Specified by:

removeDataSourceMetadataOlderThan in interface IndexerMetadataStorageCoordinator

Parameters:

timestamp - timestamp in milliseconds

excludeSupervisorIds - set of supervisor ids to exclude from removal

Returns:

number of datasource metadata removed

retrieveSegmentForId

public DataSegment retrieveSegmentForId(SegmentId segmentId)

Description copied from interface: IndexerMetadataStorageCoordinator

Retrieve the segment for a given id from the metadata store. Return null if no such segment exists
The retrieval also considers the set of unused segments in the metadata store. Unused segments could be deleted by a kill task at any time and might lead to unexpected behaviour. This option exists mainly to provide a consistent view of the metadata, for example, in calls from MSQ controller and worker and would generally not be required.

Specified by:

retrieveSegmentForId in interface IndexerMetadataStorageCoordinator

retrieveUsedSegmentForId

public DataSegment retrieveUsedSegmentForId(SegmentId segmentId)

Specified by:

retrieveUsedSegmentForId in interface IndexerMetadataStorageCoordinator

deletePendingSegmentsForTaskAllocatorId

public int deletePendingSegmentsForTaskAllocatorId(String datasource,
 String taskAllocatorId)

Description copied from interface: IndexerMetadataStorageCoordinator

Delete pending segment for a give task group after all the tasks belonging to it have completed.

Specified by:

deletePendingSegmentsForTaskAllocatorId in interface IndexerMetadataStorageCoordinator

Parameters:

datasource - datasource of the task

taskAllocatorId - task id / task group / replica group for an appending task

Returns:

number of pending segments deleted from the metadata store

getPendingSegments

public List<PendingSegmentRecord> getPendingSegments(String datasource,
 org.joda.time.Interval interval)

Description copied from interface: IndexerMetadataStorageCoordinator

Fetches all the pending segments of the datasource that overlap with a given interval.

Specified by:

getPendingSegments in interface IndexerMetadataStorageCoordinator

Parameters:

datasource - datasource to be queried

interval - interval with which segments overlap

Returns:

List of pending segment records

deleteUpgradeSegmentsForTask

public int deleteUpgradeSegmentsForTask(String taskId)

Description copied from interface: IndexerMetadataStorageCoordinator

Delete entries from the upgrade segments table after the corresponding replace task has ended

Specified by:

deleteUpgradeSegmentsForTask in interface IndexerMetadataStorageCoordinator

Parameters:

taskId - - id of the task with replace locks

Returns:

number of deleted entries from the metadata store

retrieveUpgradedFromSegmentIds

public Map<String,String> retrieveUpgradedFromSegmentIds(String dataSource,
 Set<String> segmentIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Map from a segment ID to the segment ID from which it was upgraded There should be no entry in the map for an original non-upgraded segment

Specified by:

retrieveUpgradedFromSegmentIds in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - data source

segmentIds - ids of segments

retrieveUpgradedToSegmentIds

public Map<String,Set<String>> retrieveUpgradedToSegmentIds(String dataSource,
 Set<String> segmentIds)

Description copied from interface: IndexerMetadataStorageCoordinator

Map from a segment ID to a set containing 1) all segment IDs that were upgraded from it AND are still present in the metadata store 2) the segment ID itself if and only if it is still present in the metadata store

Specified by:

retrieveUpgradedToSegmentIds in interface IndexerMetadataStorageCoordinator

Parameters:

dataSource - data source

segmentIds - ids of the first segments which had the corresponding load spec