Package org.apache.druid.segment.metadata

Class CoordinatorSegmentMetadataCache

java.lang.Object

org.apache.druid.segment.metadata.AbstractSegmentMetadataCache<DataSourceInformation>

org.apache.druid.segment.metadata.CoordinatorSegmentMetadataCache

public class CoordinatorSegmentMetadataCache
extends AbstractSegmentMetadataCache<DataSourceInformation>

Coordinator-side cache of segment metadata that combines segments to build datasources. The cache provides metadata about a datasource, see DataSourceInformation.

Major differences from the other implementation BrokerSegmentMetadataCache are,

The refresh is executed only on the leader Coordinator node.

Realtime segment schema refresh. Schema update for realtime segment is pushed periodically. The schema is merged with any existing schema for the segment and the cache is updated. Corresponding datasource is marked for refresh.

The refresh mechanism is significantly different from the other implementation,

• Metadata query is executed only for those non-realtime segments for which the schema is not cached.
• Datasources marked for refresh are then rebuilt.

It is important to note that the datasource schema returned in getDatasource(java.lang.String) & getDataSourceInformationMap() also includes columns from cold segments. Cold segments are processed in a separate thread and datasource schema from cold segments is separately stored.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

AbstractSegmentMetadataCache.ColumnTypeMergePolicy, AbstractSegmentMetadataCache.FirstTypeMergePolicy, AbstractSegmentMetadataCache.LeastRestrictiveTypeMergePolicy

Field Summary

Fields inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

cacheExec, callbackExec, dataSourcesNeedingRebuild, isServerViewInitialized, lock, mutableSegments, ROW_SIGNATURE_INTERNER, SEGMENT_ORDER, segmentMetadataInfo, segmentsNeedingRefresh, tables

Constructor Summary

Constructors

Constructor

Description

CoordinatorSegmentMetadataCache(QueryLifecycleFactory queryLifecycleFactory, CoordinatorServerView serverView, SegmentMetadataCacheConfig config, Escalator escalator, InternalQueryConfig internalQueryConfig, ServiceEmitter emitter, SegmentSchemaCache segmentSchemaCache, SegmentSchemaBackFillQueue segmentSchemaBackfillQueue, SegmentsMetadataManager segmentsMetadataManager, com.google.common.base.Supplier<SegmentsMetadataManagerConfig> segmentsMetadataManagerConfigSupplier)

Method Summary

Modifier and Type	Method	Description
RowSignature	buildDataSourceRowSignature(String dataSource)
protected boolean	fetchAggregatorsInSegmentMetadataQuery()
AvailableSegmentMetadata	getAvailableSegmentMetadata(String datasource, SegmentId segmentId)	Get metadata for the specified segment, which includes information like RowSignature, realtime & numRows.
DataSourceInformation	getDatasource(String name)	Fetch schema for the given datasource.
Map<String,DataSourceInformation>	getDataSourceInformationMap()
Iterator<AvailableSegmentMetadata>	iterateSegmentMetadata()	Get metadata for all the cached segments, which includes information like RowSignature, realtime & numRows etc.
void	onLeaderStart()
void	onLeaderStop()
void	refresh(Set<SegmentId> segmentsToRefresh, Set<String> dataSourcesToRebuild)	Executes SegmentMetadataQuery to fetch schema information for each segment in the refresh list.
protected void	refreshColdSegmentSchemas()	Recomputes the cold schema of all datasources and updates in coldSchemaTable.
void	refreshWaitCondition()	This method ensures that the refresh goes through only when schemaCache is initialized.
protected void	removeSegmentAction(SegmentId segmentId)	This method should be overridden by child classes to execute any action on segment removal.
void	start()	Lifecycle start method.
void	stop()	Lifecycle stop method.
protected void	unmarkSegmentAsMutable(SegmentId segmentId)
protected boolean	updateSegmentMetadata(SegmentId segmentId, SegmentAnalysis analysis)	Updates metadata of a segment using the results of a metadata query.
void	updateSegmentReplicationStatus(SegmentReplicationStatus segmentReplicationStatus)

Methods inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

addSegment, awaitInitialization, cacheExecLoop, doInLock, emitMetric, emitMetric, getDatasourceNames, getDataSourcesNeedingRebuild, getMutableSegments, getSegmentMetadataSnapshot, getSegmentsNeedingRefresh, getTotalSegments, markDataSourceAsNeedRebuild, markSegmentAsNeedRefresh, refreshSegments, removeSegment, removeServerSegment, runSegmentMetadataQuery, setAvailableSegmentMetadata, shouldRefresh

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CoordinatorSegmentMetadataCache

@Inject
public CoordinatorSegmentMetadataCache(QueryLifecycleFactory queryLifecycleFactory,
 CoordinatorServerView serverView,
 SegmentMetadataCacheConfig config,
 Escalator escalator,
 InternalQueryConfig internalQueryConfig,
 ServiceEmitter emitter,
 SegmentSchemaCache segmentSchemaCache,
 SegmentSchemaBackFillQueue segmentSchemaBackfillQueue,
 SegmentsMetadataManager segmentsMetadataManager,
 com.google.common.base.Supplier<SegmentsMetadataManagerConfig> segmentsMetadataManagerConfigSupplier)

Method Details

start

public void start()

Description copied from class: AbstractSegmentMetadataCache

Lifecycle start method.

Specified by:

start in class AbstractSegmentMetadataCache<DataSourceInformation>

stop

public void stop()

Description copied from class: AbstractSegmentMetadataCache

Lifecycle stop method.

Specified by:

stop in class AbstractSegmentMetadataCache<DataSourceInformation>

onLeaderStart

public void onLeaderStart()

onLeaderStop

public void onLeaderStop()

refreshWaitCondition

public void refreshWaitCondition()
                          throws InterruptedException

This method ensures that the refresh goes through only when schemaCache is initialized.

Overrides:

refreshWaitCondition in class AbstractSegmentMetadataCache<DataSourceInformation>

Throws:

InterruptedException

updateSegmentReplicationStatus

public void updateSegmentReplicationStatus(SegmentReplicationStatus segmentReplicationStatus)

unmarkSegmentAsMutable

protected void unmarkSegmentAsMutable(SegmentId segmentId)

Overrides:

unmarkSegmentAsMutable in class AbstractSegmentMetadataCache<DataSourceInformation>

removeSegmentAction

protected void removeSegmentAction(SegmentId segmentId)

Description copied from class: AbstractSegmentMetadataCache

This method should be overridden by child classes to execute any action on segment removal.

Specified by:

removeSegmentAction in class AbstractSegmentMetadataCache<DataSourceInformation>

fetchAggregatorsInSegmentMetadataQuery

protected boolean fetchAggregatorsInSegmentMetadataQuery()

Overrides:

fetchAggregatorsInSegmentMetadataQuery in class AbstractSegmentMetadataCache<DataSourceInformation>

updateSegmentMetadata

protected boolean updateSegmentMetadata(SegmentId segmentId,
 SegmentAnalysis analysis)

Description copied from class: AbstractSegmentMetadataCache

Updates metadata of a segment using the results of a metadata query.

Overrides:

updateSegmentMetadata in class AbstractSegmentMetadataCache<DataSourceInformation>

Returns:

true if the segment metadata was updated successfully.

iterateSegmentMetadata

public Iterator<AvailableSegmentMetadata> iterateSegmentMetadata()

Description copied from class: AbstractSegmentMetadataCache

Get metadata for all the cached segments, which includes information like RowSignature, realtime & numRows etc. This is a lower-overhead method than AbstractSegmentMetadataCache.getSegmentMetadataSnapshot().

Overrides:

iterateSegmentMetadata in class AbstractSegmentMetadataCache<DataSourceInformation>

Returns:

iterator of metadata.

getAvailableSegmentMetadata

@Nullable
public AvailableSegmentMetadata getAvailableSegmentMetadata(String datasource,
 SegmentId segmentId)

Description copied from class: AbstractSegmentMetadataCache

Get metadata for the specified segment, which includes information like RowSignature, realtime & numRows.

Overrides:

getAvailableSegmentMetadata in class AbstractSegmentMetadataCache<DataSourceInformation>

Parameters:

datasource - segment datasource

segmentId - segment Id

Returns:

Metadata information for the given segment

getDatasource

public DataSourceInformation getDatasource(String name)

Description copied from class: AbstractSegmentMetadataCache

Fetch schema for the given datasource.

Overrides:

getDatasource in class AbstractSegmentMetadataCache<DataSourceInformation>

Parameters:

name - datasource

Returns:

schema information for the given datasource

getDataSourceInformationMap

public Map<String,DataSourceInformation> getDataSourceInformationMap()

Overrides:

getDataSourceInformationMap in class AbstractSegmentMetadataCache<DataSourceInformation>

Returns:

Map of datasource and corresponding schema information.

refresh

public void refresh(Set<SegmentId> segmentsToRefresh,
 Set<String> dataSourcesToRebuild)
             throws IOException

Executes SegmentMetadataQuery to fetch schema information for each segment in the refresh list. The schema information for individual segments is combined to construct a table schema, which is then cached.

Specified by:

refresh in class AbstractSegmentMetadataCache<DataSourceInformation>

Parameters:

segmentsToRefresh - segments for which the schema might have changed

dataSourcesToRebuild - datasources for which the schema might have changed

Throws:

IOException - when querying segment from data nodes and tasks

refreshColdSegmentSchemas

protected void refreshColdSegmentSchemas()

Recomputes the cold schema of all datasources and updates in coldSchemaTable. The cold schema row signature is obtained by merging the column types from the schemas of all cold segments (used segments with zero replication).

buildDataSourceRowSignature

@Nullable
public RowSignature buildDataSourceRowSignature(String dataSource)

Overrides:

buildDataSourceRowSignature in class AbstractSegmentMetadataCache<DataSourceInformation>