Package org.apache.druid.segment.loading

Class SegmentLocalCacheManager

java.lang.Object

org.apache.druid.segment.loading.SegmentLocalCacheManager

All Implemented Interfaces:

SegmentCacheManager

public class SegmentLocalCacheManager
extends Object
implements SegmentCacheManager

Constructor Summary

Constructors

Constructor	Description
SegmentLocalCacheManager(List<StorageLocation> locations, SegmentLoaderConfig config, StorageLocationSelectorStrategy strategy, IndexIO indexIO, com.fasterxml.jackson.databind.ObjectMapper mapper)

Method Summary

Modifier and Type	Method	Description
boolean	canHandleSegments()	Return whether the cache manager can handle segments or not.
void	cleanup(DataSegment segment)	Cleanup the segment files cache space used by the segment.
ReferenceCountedSegmentProvider	getBootstrapSegment(DataSegment dataSegment, SegmentLazyLoadFailCallback loadFailed)	Similar to SegmentCacheManager.getSegment(DataSegment), this method returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline during startup on data nodes.
List<DataSegment>	getCachedSegments()	Return a list of cached segments from local disk, if any.
List<StorageLocation>	getLocations()
ReferenceCountedSegmentProvider	getSegment(DataSegment dataSegment)	Returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline.
File	getSegmentFiles(DataSegment segment)	Make sure segments files in loc is intact, otherwise function like loadSegments will failed because of segment files is damaged.
ConcurrentHashMap<DataSegment,org.apache.druid.segment.loading.SegmentLocalCacheManager.ReferenceCountingLock>	getSegmentLocks()
void	loadSegmentIntoPageCache(DataSegment segment)	Asynchronously load the supplied segment into the page cache on each download after the service finishes bootstrapping.
void	loadSegmentIntoPageCacheOnBootstrap(DataSegment segment)	Similar to SegmentCacheManager.loadSegmentIntoPageCache(DataSegment), but asynchronously load the supplied segment into the page cache during service bootstrap.
boolean	release(DataSegment segment)	Reverts the effects of SegmentCacheManager.reserve(DataSegment) by releasing the location reserved for this segment.
void	removeInfoFile(DataSegment segment)	Remove the segment info file for the supplied segment from disk.
boolean	reserve(DataSegment segment)
void	shutdownBootstrap()	Shutdown any previously set up bootstrap executor to save resources.
void	storeInfoFile(DataSegment segment)	Store a segment info file for the supplied segment on disk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SegmentLocalCacheManager

@Inject
public SegmentLocalCacheManager(List<StorageLocation> locations,
 SegmentLoaderConfig config,
 @Nonnull
 StorageLocationSelectorStrategy strategy,
 IndexIO indexIO,
 com.fasterxml.jackson.databind.ObjectMapper mapper)

Method Details

canHandleSegments

public boolean canHandleSegments()

Description copied from interface: SegmentCacheManager

Return whether the cache manager can handle segments or not.

Specified by:

canHandleSegments in interface SegmentCacheManager

getCachedSegments

public List<DataSegment> getCachedSegments()
                                    throws IOException

Description copied from interface: SegmentCacheManager

Return a list of cached segments from local disk, if any. This should be called only when SegmentCacheManager.canHandleSegments() is true.

Specified by:

getCachedSegments in interface SegmentCacheManager

Throws:

IOException

storeInfoFile

public void storeInfoFile(DataSegment segment)
                   throws IOException

Description copied from interface: SegmentCacheManager

Store a segment info file for the supplied segment on disk. This operation is idempotent when called multiple times for a given segment.

Specified by:

storeInfoFile in interface SegmentCacheManager

Throws:

IOException

removeInfoFile

public void removeInfoFile(DataSegment segment)

Description copied from interface: SegmentCacheManager

Remove the segment info file for the supplied segment from disk. If the file cannot be deleted, do nothing.

Specified by:

removeInfoFile in interface SegmentCacheManager

See Also:

• SegmentCacheManager.cleanup(DataSegment)

getSegment

public ReferenceCountedSegmentProvider getSegment(DataSegment dataSegment)
                                           throws SegmentLoadingException

Description copied from interface: SegmentCacheManager

Returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline. This method can be called multiple times by the SegmentManager and implementation can either return same ReferenceCountedSegmentProvider or a different ReferenceCountedSegmentProvider. Caller should not assume any particular behavior.

Returning a ReferenceCountingSegment will let custom implementations keep track of reference count for segments that the custom implementations are creating. That way, custom implementations can know when the segment is in use or not.

Specified by:

getSegment in interface SegmentCacheManager

Parameters:

dataSegment - Segment to get on each download after service bootstrap

Throws:

SegmentLoadingException - If there is an error in loading the segment

See Also:

• SegmentCacheManager.getBootstrapSegment(DataSegment, SegmentLazyLoadFailCallback)

getBootstrapSegment

public ReferenceCountedSegmentProvider getBootstrapSegment(DataSegment dataSegment,
 SegmentLazyLoadFailCallback loadFailed)
                                                    throws SegmentLoadingException

Description copied from interface: SegmentCacheManager

Similar to SegmentCacheManager.getSegment(DataSegment), this method returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline during startup on data nodes.

Specified by:

getBootstrapSegment in interface SegmentCacheManager

Parameters:

dataSegment - Segment to retrieve during service bootstrap

loadFailed - Callback to execute when segment lazy load failed. This applies only when lazyLoadOnStart is enabled

Throws:

SegmentLoadingException - - If there is an error in loading the segment

See Also:

• SegmentCacheManager.getSegment(DataSegment)

getSegmentFiles

public File getSegmentFiles(DataSegment segment)
                     throws SegmentLoadingException

Make sure segments files in loc is intact, otherwise function like loadSegments will failed because of segment files is damaged.

Specified by:

getSegmentFiles in interface SegmentCacheManager

Parameters:

segment -

Returns:

Throws:

SegmentLoadingException

reserve

public boolean reserve(DataSegment segment)

Specified by:

reserve in interface SegmentCacheManager

release

public boolean release(DataSegment segment)

Description copied from interface: SegmentCacheManager

Reverts the effects of SegmentCacheManager.reserve(DataSegment) by releasing the location reserved for this segment. Callers that explicitly reserve the space via SegmentCacheManager.reserve(DataSegment) should use this method to release the space.

Implementation can throw error if the space is being released but there is data present. Callers are supposed to ensure that any data is removed via SegmentCacheManager.cleanup(DataSegment). Only return a boolean instead of a pointer to StorageLocation since we don't want callers to operate on StorageLocation directly outside this interface.

Specified by:

release in interface SegmentCacheManager

Parameters:

segment - - Segment to release the location for.

Returns:

- True if any location was reserved and released, false otherwise.

cleanup

public void cleanup(DataSegment segment)

Description copied from interface: SegmentCacheManager

Cleanup the segment files cache space used by the segment. It will not release the space if the space has been explicitly reserved via SegmentCacheManager.reserve(DataSegment).

Specified by:

cleanup in interface SegmentCacheManager

See Also:

• SegmentCacheManager.removeInfoFile(DataSegment)

loadSegmentIntoPageCache

public void loadSegmentIntoPageCache(DataSegment segment)

Description copied from interface: SegmentCacheManager

Asynchronously load the supplied segment into the page cache on each download after the service finishes bootstrapping. Equivalent to `cat segment_files > /dev/null` to force loading the segment index files into page cache so that later when the segment is queried, they are already in page cache and only a minor page fault needs to be triggered instead of a major page fault to make the query latency more consistent.

Specified by:

loadSegmentIntoPageCache in interface SegmentCacheManager

See Also:

• SegmentCacheManager.loadSegmentIntoPageCacheOnBootstrap(DataSegment)

loadSegmentIntoPageCacheOnBootstrap

public void loadSegmentIntoPageCacheOnBootstrap(DataSegment segment)

Description copied from interface: SegmentCacheManager

Similar to SegmentCacheManager.loadSegmentIntoPageCache(DataSegment), but asynchronously load the supplied segment into the page cache during service bootstrap.

Specified by:

loadSegmentIntoPageCacheOnBootstrap in interface SegmentCacheManager

See Also:

• SegmentCacheManager.loadSegmentIntoPageCache(DataSegment)

shutdownBootstrap

public void shutdownBootstrap()

Description copied from interface: SegmentCacheManager

Shutdown any previously set up bootstrap executor to save resources. This should be called after loading bootstrap segments into the page cache.

Specified by:

shutdownBootstrap in interface SegmentCacheManager

getSegmentLocks

public ConcurrentHashMap<DataSegment,org.apache.druid.segment.loading.SegmentLocalCacheManager.ReferenceCountingLock> getSegmentLocks()

getLocations

public List<StorageLocation> getLocations()