org.apache.hadoop.hbase.regionserver

Interface Store

All Superinterfaces:

ConfigurationObserver, HeapSize, PropagatingConfigurationObserver, StoreConfigInformation

All Known Implementing Classes:

HMobStore, HStore

@InterfaceAudience.LimitedPrivate(value="Coprocesssor")
 @InterfaceStability.Evolving
public interface Store
extends HeapSize, StoreConfigInformation, PropagatingConfigurationObserver

Interface for objects that hold a column family in a Region. Its a memstore and a set of zero or more StoreFiles, which stretch backwards over time.

Field Summary

Fields

Modifier and Type	Field and Description
static int	NO_PRIORITY
static int	PRIORITY_USER The default priority for user-specified compaction requests.

Method Summary

Modifier and Type	Method and Description
void	addChangedReaderObserver(ChangedReadersObserver o)
boolean	areWritesEnabled()
void	cancelRequestedCompaction(CompactionContext compaction)
boolean	canSplit()
Collection<StoreFile>	close() Close all the readers We don't need to worry about subsequent requests because the Region holds a write lock that will prevent any more reads or writes.
void	closeAndArchiveCompactedFiles() Closes and archives the compacted files under this store
List<StoreFile>	compact(CompactionContext compaction, ThroughputController throughputController) Deprecated. see compact(CompactionContext, ThroughputController, User)
List<StoreFile>	compact(CompactionContext compaction, ThroughputController throughputController, User user)
org.apache.hadoop.hbase.regionserver.StoreFlushContext	createFlushContext(long cacheFlushId)
StoreFileWriter	createWriterInTmp(long maxKeyCount, Compression.Algorithm compression, boolean isCompaction, boolean includeMVCCReadpoint, boolean includesTags)
StoreFileWriter	createWriterInTmp(long maxKeyCount, Compression.Algorithm compression, boolean isCompaction, boolean includeMVCCReadpoint, boolean includesTags, boolean shouldDropBehind)
StoreFileWriter	createWriterInTmp(long maxKeyCount, Compression.Algorithm compression, boolean isCompaction, boolean includeMVCCReadpoint, boolean includesTags, boolean shouldDropBehind, TimeRangeTracker trt)
void	deleteChangedReaderObserver(ChangedReadersObserver o)
long	getAvgStoreFileAge()
CacheConfig	getCacheConfig() Used for tests.
ColumnFamilyDescriptor	getColumnFamilyDescriptor()
String	getColumnFamilyName()
long	getCompactedCellsCount()
long	getCompactedCellsSize()
Collection<StoreFile>	getCompactedFiles()
int	getCompactedFilesCount()
double	getCompactionPressure() This value can represent the degree of emergency of compaction for this store.
CompactionProgress	getCompactionProgress() getter for CompactionProgress object
int	getCompactPriority()
CellComparator	getComparator()
RegionCoprocessorHost	getCoprocessorHost()
HFileDataBlockEncoder	getDataBlockEncoder()
org.apache.hadoop.fs.FileSystem	getFileSystem()
long	getFlushableSize() Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeToFlush() instead. Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.
long	getFlushedCellsCount()
long	getFlushedCellsSize()
long	getFlushedOutputFileSize()
long	getHFilesSize()
long	getLastCompactSize()
long	getMajorCompactedCellsCount()
long	getMajorCompactedCellsSize()
long	getMaxMemstoreTS()
long	getMaxSequenceId()
long	getMaxStoreFileAge()
long	getMemStoreSize() Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeOfMemStore() instead. Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.
long	getMinStoreFileAge()
long	getNumHFiles()
long	getNumReferenceFiles()
HRegionInfo	getRegionInfo()
ScanInfo	getScanInfo()
KeyValueScanner	getScanner(Scan scan, NavigableSet<byte[]> targetCols, long readPt) Return a scanner for both the memstore and the HStore files.
default List<KeyValueScanner>	getScanners(boolean cacheBlocks, boolean isGet, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, byte[] stopRow, long readPt) Get all scanners with no filtering based on TTL (that happens further down the line).
List<KeyValueScanner>	getScanners(boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt) Get all scanners with no filtering based on TTL (that happens further down the line).
default List<KeyValueScanner>	getScanners(List<StoreFile> files, boolean cacheBlocks, boolean isGet, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, byte[] stopRow, long readPt, boolean includeMemstoreScanner) Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).
List<KeyValueScanner>	getScanners(List<StoreFile> files, boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt, boolean includeMemstoreScanner) Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).
long	getSize()
MemstoreSize	getSizeOfMemStore()
MemstoreSize	getSizeOfSnapshot()
MemstoreSize	getSizeToFlush()
long	getSmallestReadPoint()
long	getSnapshotSize() Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeOfSnapshot() instead. Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.
byte[]	getSplitPoint() Determines if Store should be split
Collection<StoreFile>	getStorefiles()
int	getStorefilesCount()
long	getStorefilesIndexSize()
long	getStorefilesSize()
long	getStoreSizeUncompressed()
TableName	getTableName()
long	getTotalStaticBloomSize() Returns the total byte size of all Bloom filter bit arrays.
long	getTotalStaticIndexSize() Returns the total size of all index blocks in the data block indexes, including the root level, intermediate levels, and the leaf level for multi-level indexes, or just the root level for single-level indexes.
boolean	hasReferences()
boolean	hasTooManyStoreFiles()
boolean	isMajorCompaction()
boolean	isPrimaryReplicaStore()
boolean	isSloppyMemstore()
boolean	needsCompaction() See if there's too much store files in this store
List<KeyValueScanner>	recreateScanners(List<KeyValueScanner> currentFileScanners, boolean cacheBlocks, boolean usePread, boolean isCompaction, ScanQueryMatcher matcher, byte[] startRow, boolean includeStartRow, byte[] stopRow, boolean includeStopRow, long readPt, boolean includeMemstoreScanner) Recreates the scanners on the current list of active store file scanners
void	refreshStoreFiles() Checks the underlying store files, and opens the files that have not been opened, and removes the store file readers for store files no longer available.
void	refreshStoreFiles(Collection<String> newFiles) Replaces the store files that the store has with the given files.
default Optional<CompactionContext>	requestCompaction()
Optional<CompactionContext>	requestCompaction(int priority, CompactionLifeCycleTracker tracker, User user)
boolean	throttleCompaction(long compactionSize)
long	timeOfOldestEdit() When was the last edit done in the memstore
void	triggerMajorCompaction()

Methods inherited from interface org.apache.hadoop.hbase.io.HeapSize

heapSize

Methods inherited from interface org.apache.hadoop.hbase.regionserver.StoreConfigInformation

getBlockingFileCount, getCompactionCheckMultiplier, getMemstoreFlushSize, getStoreFileTtl

Methods inherited from interface org.apache.hadoop.hbase.conf.PropagatingConfigurationObserver

deregisterChildren, registerChildren

Methods inherited from interface org.apache.hadoop.hbase.conf.ConfigurationObserver

onConfigurationChange

Field Detail

PRIORITY_USER

static final int PRIORITY_USER

The default priority for user-specified compaction requests. The user gets top priority unless we have blocking compactions. (Pri <= 0)

See Also:

Constant Field Values

NO_PRIORITY

static final int NO_PRIORITY

See Also:

Constant Field Values

Method Detail

getComparator

CellComparator getComparator()

getStorefiles

Collection<StoreFile> getStorefiles()

getCompactedFiles

Collection<StoreFile> getCompactedFiles()

close

Collection<StoreFile> close()
                     throws IOException

Close all the readers We don't need to worry about subsequent requests because the Region holds a write lock that will prevent any more reads or writes.

Returns:

the StoreFiles that were previously being used.

Throws:

IOException - on failure

getScanner

KeyValueScanner getScanner(Scan scan,
                           NavigableSet<byte[]> targetCols,
                           long readPt)
                    throws IOException

Return a scanner for both the memstore and the HStore files. Assumes we are not in a compaction.

Parameters:

scan - Scan to apply when scanning the stores

targetCols - columns to scan

Returns:

a scanner over the current key values

Throws:

IOException - on failure

getScanners

default List<KeyValueScanner> getScanners(boolean cacheBlocks,
                                          boolean isGet,
                                          boolean usePread,
                                          boolean isCompaction,
                                          ScanQueryMatcher matcher,
                                          byte[] startRow,
                                          byte[] stopRow,
                                          long readPt)
                                   throws IOException

Get all scanners with no filtering based on TTL (that happens further down the line).

Parameters:

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

stopRow - the stop row

readPt - the read point of the current scan

Returns:

all scanners for this store

Throws:

IOException

getScanners

List<KeyValueScanner> getScanners(boolean cacheBlocks,
                                  boolean usePread,
                                  boolean isCompaction,
                                  ScanQueryMatcher matcher,
                                  byte[] startRow,
                                  boolean includeStartRow,
                                  byte[] stopRow,
                                  boolean includeStopRow,
                                  long readPt)
                           throws IOException

Get all scanners with no filtering based on TTL (that happens further down the line).

Parameters:

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

includeStartRow - true to include start row, false if not

stopRow - the stop row

includeStopRow - true to include stop row, false if not

readPt - the read point of the current scan

Returns:

all scanners for this store

Throws:

IOException

recreateScanners

List<KeyValueScanner> recreateScanners(List<KeyValueScanner> currentFileScanners,
                                       boolean cacheBlocks,
                                       boolean usePread,
                                       boolean isCompaction,
                                       ScanQueryMatcher matcher,
                                       byte[] startRow,
                                       boolean includeStartRow,
                                       byte[] stopRow,
                                       boolean includeStopRow,
                                       long readPt,
                                       boolean includeMemstoreScanner)
                                throws IOException

Recreates the scanners on the current list of active store file scanners

Parameters:

currentFileScanners - the current set of active store file scanners

cacheBlocks - cache the blocks or not

usePread - use pread or not

isCompaction - is the scanner for compaction

matcher - the scan query matcher

startRow - the scan's start row

includeStartRow - should the scan include the start row

stopRow - the scan's stop row

includeStopRow - should the scan include the stop row

readPt - the read point of the current scane

includeMemstoreScanner - whether the current scanner should include memstorescanner

Returns:

list of scanners recreated on the current Scanners

Throws:

IOException

getScanners

default List<KeyValueScanner> getScanners(List<StoreFile> files,
                                          boolean cacheBlocks,
                                          boolean isGet,
                                          boolean usePread,
                                          boolean isCompaction,
                                          ScanQueryMatcher matcher,
                                          byte[] startRow,
                                          byte[] stopRow,
                                          long readPt,
                                          boolean includeMemstoreScanner)
                                   throws IOException

Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).

Parameters:

files - the list of files on which the scanners has to be created

cacheBlocks - cache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

stopRow - the stop row

readPt - the read point of the current scan

includeMemstoreScanner - true if memstore has to be included

Returns:

scanners on the given files and on the memstore if specified

Throws:

IOException

getScanners

List<KeyValueScanner> getScanners(List<StoreFile> files,
                                  boolean cacheBlocks,
                                  boolean usePread,
                                  boolean isCompaction,
                                  ScanQueryMatcher matcher,
                                  byte[] startRow,
                                  boolean includeStartRow,
                                  byte[] stopRow,
                                  boolean includeStopRow,
                                  long readPt,
                                  boolean includeMemstoreScanner)
                           throws IOException

Create scanners on the given files and if needed on the memstore with no filtering based on TTL (that happens further down the line).

Parameters:

files - the list of files on which the scanners has to be created

cacheBlocks - ache the blocks or not

usePread - true to use pread, false if not

isCompaction - true if the scanner is created for compaction

matcher - the scan query matcher

startRow - the start row

includeStartRow - true to include start row, false if not

stopRow - the stop row

includeStopRow - true to include stop row, false if not

readPt - the read point of the current scan

includeMemstoreScanner - true if memstore has to be included

Returns:

scanners on the given files and on the memstore if specified

Throws:

IOException

getScanInfo

ScanInfo getScanInfo()

timeOfOldestEdit

long timeOfOldestEdit()

When was the last edit done in the memstore

getFileSystem

org.apache.hadoop.fs.FileSystem getFileSystem()

createWriterInTmp

StoreFileWriter createWriterInTmp(long maxKeyCount,
                                  Compression.Algorithm compression,
                                  boolean isCompaction,
                                  boolean includeMVCCReadpoint,
                                  boolean includesTags)
                           throws IOException

Parameters:

maxKeyCount -

compression - Compression algorithm to use

isCompaction - whether we are creating a new file in a compaction

includeMVCCReadpoint - whether we should out the MVCC readpoint

Returns:

Writer for a new StoreFile in the tmp dir.

Throws:

IOException

createWriterInTmp

StoreFileWriter createWriterInTmp(long maxKeyCount,
                                  Compression.Algorithm compression,
                                  boolean isCompaction,
                                  boolean includeMVCCReadpoint,
                                  boolean includesTags,
                                  boolean shouldDropBehind)
                           throws IOException

Parameters:

maxKeyCount -

compression - Compression algorithm to use

isCompaction - whether we are creating a new file in a compaction

includeMVCCReadpoint - whether we should out the MVCC readpoint

shouldDropBehind - should the writer drop caches behind writes

Returns:

Writer for a new StoreFile in the tmp dir.

Throws:

IOException

createWriterInTmp

StoreFileWriter createWriterInTmp(long maxKeyCount,
                                  Compression.Algorithm compression,
                                  boolean isCompaction,
                                  boolean includeMVCCReadpoint,
                                  boolean includesTags,
                                  boolean shouldDropBehind,
                                  TimeRangeTracker trt)
                           throws IOException

Parameters:

maxKeyCount -

compression - Compression algorithm to use

isCompaction - whether we are creating a new file in a compaction

includeMVCCReadpoint - whether we should out the MVCC readpoint

shouldDropBehind - should the writer drop caches behind writes

trt - Ready-made timetracker to use.

Returns:

Writer for a new StoreFile in the tmp dir.

Throws:

IOException

throttleCompaction

boolean throttleCompaction(long compactionSize)

getCompactionProgress

CompactionProgress getCompactionProgress()

getter for CompactionProgress object

Returns:

CompactionProgress object; can be null

requestCompaction

default Optional<CompactionContext> requestCompaction()
                                               throws IOException

Throws:

IOException

requestCompaction

Optional<CompactionContext> requestCompaction(int priority,
                                              CompactionLifeCycleTracker tracker,
                                              User user)
                                       throws IOException

Throws:

IOException

cancelRequestedCompaction

void cancelRequestedCompaction(CompactionContext compaction)

compact

@Deprecated
List<StoreFile> compact(CompactionContext compaction,
                                    ThroughputController throughputController)
                             throws IOException

Deprecated. see compact(CompactionContext, ThroughputController, User)

Throws:

IOException

compact

List<StoreFile> compact(CompactionContext compaction,
                        ThroughputController throughputController,
                        User user)
                 throws IOException

Throws:

IOException

isMajorCompaction

boolean isMajorCompaction()
                   throws IOException

Returns:

true if we should run a major compaction.

Throws:

IOException

triggerMajorCompaction

void triggerMajorCompaction()

needsCompaction

boolean needsCompaction()

See if there's too much store files in this store

Returns:

true if number of store files is greater than the number defined in minFilesToCompact

getCompactPriority

int getCompactPriority()

createFlushContext

org.apache.hadoop.hbase.regionserver.StoreFlushContext createFlushContext(long cacheFlushId)

canSplit

boolean canSplit()

getSplitPoint

byte[] getSplitPoint()

Determines if Store should be split

Returns:

byte[] if store should be split, null otherwise.

hasReferences

boolean hasReferences()

Returns:

true if the store has any underlying reference files to older HFiles

getMemStoreSize

@Deprecated
long getMemStoreSize()

Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeOfMemStore() instead.

Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.

Returns:

The size of this store's memstore, in bytes

getSizeOfMemStore

MemstoreSize getSizeOfMemStore()

Returns:

The size of this store's memstore.

getFlushableSize

@Deprecated
long getFlushableSize()

Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeToFlush() instead.

Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.

Returns:

The amount of memory we could flush from this memstore; usually this is equal to getMemStoreSize() unless we are carrying snapshots and then it will be the size of outstanding snapshots.

getSizeToFlush

MemstoreSize getSizeToFlush()

Returns:

The amount of memory we could flush from this memstore; usually this is equal to getSizeOfMemStore() unless we are carrying snapshots and then it will be the size of outstanding snapshots.

getSnapshotSize

@Deprecated
long getSnapshotSize()

Deprecated. Since 2.0 and will be removed in 3.0. Use getSizeOfSnapshot() instead.

Note: When using off heap MSLAB feature, this will not account the cell data bytes size which is in off heap MSLAB area.

Returns the memstore snapshot size

Returns:

size of the memstore snapshot

getSizeOfSnapshot

MemstoreSize getSizeOfSnapshot()

Returns:

size of the memstore snapshot

getColumnFamilyDescriptor

ColumnFamilyDescriptor getColumnFamilyDescriptor()

getMaxSequenceId

long getMaxSequenceId()

Returns:

The maximum sequence id in all store files.

getMaxMemstoreTS

long getMaxMemstoreTS()

Returns:

The maximum memstoreTS in all store files.

getDataBlockEncoder

HFileDataBlockEncoder getDataBlockEncoder()

Returns:

the data block encoder

getLastCompactSize

long getLastCompactSize()

Returns:

aggregate size of all HStores used in the last compaction

getSize

long getSize()

Returns:

aggregate size of HStore

getStorefilesCount

int getStorefilesCount()

Returns:

Count of store files

getCompactedFilesCount

int getCompactedFilesCount()

Returns:

Count of compacted store files

getMaxStoreFileAge

long getMaxStoreFileAge()

Returns:

Max age of store files in this store

getMinStoreFileAge

long getMinStoreFileAge()

Returns:

Min age of store files in this store

getAvgStoreFileAge

long getAvgStoreFileAge()

Returns:

Average age of store files in this store, 0 if no store files

getNumReferenceFiles

long getNumReferenceFiles()

Returns:

Number of reference files in this store

getNumHFiles

long getNumHFiles()

Returns:

Number of HFiles in this store

getStoreSizeUncompressed

long getStoreSizeUncompressed()

Returns:

The size of the store files, in bytes, uncompressed.

getStorefilesSize

long getStorefilesSize()

Returns:

The size of the store files, in bytes.

getHFilesSize

long getHFilesSize()

Returns:

The size of only the store files which are HFiles, in bytes.

getStorefilesIndexSize

long getStorefilesIndexSize()

Returns:

The size of the store file indexes, in bytes.

getTotalStaticIndexSize

long getTotalStaticIndexSize()

Returns the total size of all index blocks in the data block indexes, including the root level, intermediate levels, and the leaf level for multi-level indexes, or just the root level for single-level indexes.

Returns:

the total size of block indexes in the store

getTotalStaticBloomSize

long getTotalStaticBloomSize()

Returns the total byte size of all Bloom filter bit arrays. For compound Bloom filters even the Bloom blocks currently not loaded into the block cache are counted.

Returns:

the total size of all Bloom filters in the store

getCacheConfig

CacheConfig getCacheConfig()

Used for tests.

Returns:

cache configuration for this Store.

getRegionInfo

HRegionInfo getRegionInfo()

Returns:

the parent region info hosting this store

getCoprocessorHost

RegionCoprocessorHost getCoprocessorHost()

areWritesEnabled

boolean areWritesEnabled()

getSmallestReadPoint

long getSmallestReadPoint()

Returns:

The smallest mvcc readPoint across all the scanners in this region. Writes older than this readPoint, are included in every read operation.

getColumnFamilyName

String getColumnFamilyName()

getTableName

TableName getTableName()

getFlushedCellsCount

long getFlushedCellsCount()

Returns:

The number of cells flushed to disk

getFlushedCellsSize

long getFlushedCellsSize()

Returns:

The total size of data flushed to disk, in bytes

getFlushedOutputFileSize

long getFlushedOutputFileSize()

Returns:

The total size of out output files on disk, in bytes

getCompactedCellsCount

long getCompactedCellsCount()

Returns:

The number of cells processed during minor compactions

getCompactedCellsSize

long getCompactedCellsSize()

Returns:

The total amount of data processed during minor compactions, in bytes

getMajorCompactedCellsCount

long getMajorCompactedCellsCount()

Returns:

The number of cells processed during major compactions

getMajorCompactedCellsSize

long getMajorCompactedCellsSize()

Returns:

The total amount of data processed during major compactions, in bytes

addChangedReaderObserver

void addChangedReaderObserver(ChangedReadersObserver o)

deleteChangedReaderObserver

void deleteChangedReaderObserver(ChangedReadersObserver o)

hasTooManyStoreFiles

boolean hasTooManyStoreFiles()

Returns:

Whether this store has too many store files.

refreshStoreFiles

void refreshStoreFiles()
                throws IOException

Checks the underlying store files, and opens the files that have not been opened, and removes the store file readers for store files no longer available. Mainly used by secondary region replicas to keep up to date with the primary region files.

Throws:

IOException

getCompactionPressure

double getCompactionPressure()

This value can represent the degree of emergency of compaction for this store. It should be greater than or equal to 0.0, any value greater than 1.0 means we have too many store files.

• if getStorefilesCount <= getMinFilesToCompact, return 0.0
• return (getStorefilesCount - getMinFilesToCompact) / (blockingFileCount - getMinFilesToCompact)

And for striped stores, we should calculate this value by the files in each stripe separately and return the maximum value.

It is similar to getCompactPriority() except that it is more suitable to use in a linear formula.

refreshStoreFiles

void refreshStoreFiles(Collection<String> newFiles)
                throws IOException

Replaces the store files that the store has with the given files. Mainly used by secondary region replicas to keep up to date with the primary region files.

Throws:

IOException

isPrimaryReplicaStore

boolean isPrimaryReplicaStore()

closeAndArchiveCompactedFiles

void closeAndArchiveCompactedFiles()
                            throws IOException

Closes and archives the compacted files under this store

Throws:

IOException

isSloppyMemstore

boolean isSloppyMemstore()

Returns:

true if the memstore may need some extra memory space