Package org.apache.druid.segment.loading

Interface SegmentCacheManager

All Known Implementing Classes:

SegmentLocalCacheManager

public interface SegmentCacheManager

A class to fetch segment files to local disk and manage the local cache. Implementations must be thread-safe.

Method Summary

Modifier and Type	Method	Description
boolean	canHandleSegments()	Return whether the cache manager can handle segments or not.
void	cleanup(DataSegment segment)	Cleanup the segment files cache space used by the segment.
ReferenceCountedSegmentProvider	getBootstrapSegment(DataSegment segment, SegmentLazyLoadFailCallback loadFailed)	Similar to getSegment(DataSegment), this method returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline during startup on data nodes.
List<DataSegment>	getCachedSegments()	Return a list of cached segments from local disk, if any.
ReferenceCountedSegmentProvider	getSegment(DataSegment segment)	Returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline.
File	getSegmentFiles(DataSegment segment)	This method fetches the files for the given segment if the segment is not downloaded already.
void	loadSegmentIntoPageCache(DataSegment segment)	Asynchronously load the supplied segment into the page cache on each download after the service finishes bootstrapping.
void	loadSegmentIntoPageCacheOnBootstrap(DataSegment segment)	Similar to loadSegmentIntoPageCache(DataSegment), but asynchronously load the supplied segment into the page cache during service bootstrap.
boolean	release(DataSegment segment)	Reverts the effects of reserve(DataSegment) by releasing the location reserved for this segment.
void	removeInfoFile(DataSegment segment)	Remove the segment info file for the supplied segment from disk.
boolean	reserve(DataSegment segment)
void	shutdownBootstrap()	Shutdown any previously set up bootstrap executor to save resources.
void	storeInfoFile(DataSegment segment)	Store a segment info file for the supplied segment on disk.

Method Details

canHandleSegments

boolean canHandleSegments()

Return whether the cache manager can handle segments or not.

getCachedSegments

List<DataSegment> getCachedSegments()
                             throws IOException

Return a list of cached segments from local disk, if any. This should be called only when canHandleSegments() is true.

Throws:

IOException

storeInfoFile

void storeInfoFile(DataSegment segment)
            throws IOException

Store a segment info file for the supplied segment on disk. This operation is idempotent when called multiple times for a given segment.

Throws:

IOException

removeInfoFile

void removeInfoFile(DataSegment segment)

Remove the segment info file for the supplied segment from disk. If the file cannot be deleted, do nothing.

See Also:

• cleanup(DataSegment)

getSegment

ReferenceCountedSegmentProvider getSegment(DataSegment segment)
                                    throws SegmentLoadingException

Returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline. This method can be called multiple times by the SegmentManager and implementation can either return same ReferenceCountedSegmentProvider or a different ReferenceCountedSegmentProvider. Caller should not assume any particular behavior.

Returning a ReferenceCountingSegment will let custom implementations keep track of reference count for segments that the custom implementations are creating. That way, custom implementations can know when the segment is in use or not.

Parameters:

segment - Segment to get on each download after service bootstrap

Throws:

SegmentLoadingException - If there is an error in loading the segment

See Also:

• getBootstrapSegment(DataSegment, SegmentLazyLoadFailCallback)

getBootstrapSegment

ReferenceCountedSegmentProvider getBootstrapSegment(DataSegment segment,
 SegmentLazyLoadFailCallback loadFailed)
                                             throws SegmentLoadingException

Similar to getSegment(DataSegment), this method returns a ReferenceCountedSegmentProvider that will be added by the SegmentManager to the VersionedIntervalTimeline during startup on data nodes.

Parameters:

segment - Segment to retrieve during service bootstrap

loadFailed - Callback to execute when segment lazy load failed. This applies only when lazyLoadOnStart is enabled

Throws:

SegmentLoadingException - - If there is an error in loading the segment

See Also:

• getSegment(DataSegment)

getSegmentFiles

File getSegmentFiles(DataSegment segment)
              throws SegmentLoadingException

This method fetches the files for the given segment if the segment is not downloaded already. It is not required to reserve(DataSegment) before calling this method. If caller has not reserved the space explicitly via reserve(DataSegment), the implementation should reserve space on caller's behalf. If the space has been explicitly reserved already - implementation should use only the reserved space to store segment files. - implementation should not release the location in case of download erros and leave it to the caller.

Throws:

SegmentLoadingException - if there is an error in downloading files

loadSegmentIntoPageCache

void loadSegmentIntoPageCache(DataSegment segment)

Asynchronously load the supplied segment into the page cache on each download after the service finishes bootstrapping. Equivalent to `cat segment_files > /dev/null` to force loading the segment index files into page cache so that later when the segment is queried, they are already in page cache and only a minor page fault needs to be triggered instead of a major page fault to make the query latency more consistent.

See Also:

• loadSegmentIntoPageCacheOnBootstrap(DataSegment)

loadSegmentIntoPageCacheOnBootstrap

void loadSegmentIntoPageCacheOnBootstrap(DataSegment segment)

Similar to loadSegmentIntoPageCache(DataSegment), but asynchronously load the supplied segment into the page cache during service bootstrap.

See Also:

• loadSegmentIntoPageCache(DataSegment)

shutdownBootstrap

void shutdownBootstrap()

Shutdown any previously set up bootstrap executor to save resources. This should be called after loading bootstrap segments into the page cache.

reserve

boolean reserve(DataSegment segment)

release

boolean release(DataSegment segment)

Reverts the effects of reserve(DataSegment) by releasing the location reserved for this segment. Callers that explicitly reserve the space via reserve(DataSegment) should use this method to release the space.

Implementation can throw error if the space is being released but there is data present. Callers are supposed to ensure that any data is removed via cleanup(DataSegment). Only return a boolean instead of a pointer to StorageLocation since we don't want callers to operate on StorageLocation directly outside this interface.

Parameters:

segment - - Segment to release the location for.

Returns:

- True if any location was reserved and released, false otherwise.

cleanup

void cleanup(DataSegment segment)

Cleanup the segment files cache space used by the segment. It will not release the space if the space has been explicitly reserved via reserve(DataSegment).

See Also:

• removeInfoFile(DataSegment)