org.apache.hudi.metadata

Class BaseTableMetadata

java.lang.Object

org.apache.hudi.metadata.AbstractHoodieTableMetadata

org.apache.hudi.metadata.BaseTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

Direct Known Subclasses:

HoodieBackedTableMetadata

public abstract class BaseTableMetadata
extends AbstractHoodieTableMetadata

Abstract class for implementing common table metadata operations.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieTableMetaClient	dataMetaClient
protected boolean	hiveStylePartitioningEnabled
protected boolean	isMetadataTableInitialized
protected HoodieMetadataConfig	metadataConfig
protected Option<HoodieMetadataMetrics>	metrics
protected boolean	urlEncodePartitioningEnabled

Fields inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

caseSensitive, dataBasePath, engineContext, storage, storageConf

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String dataBasePath)

Method Summary

Modifier and Type	Method and Description
protected List<String>	fetchAllPartitionPaths() Returns a list of all partitions.
List<StoragePathInfo>	getAllFilesInPartition(StoragePath partitionPath) Return the list of files in a partition.
Map<String,List<StoragePathInfo>>	getAllFilesInPartitions(Collection<String> partitions) Fetch all files for given partition paths.
List<String>	getAllPartitionPaths() Return the list of partitions in the dataset.
Option<BloomFilter>	getBloomFilter(String partitionName, String fileName) Get the bloom filter for the FileID from the metadata table.
Map<Pair<String,String>,BloomFilter>	getBloomFilters(List<Pair<String,String>> partitionNameFileNameList) Get bloom filters for files from the metadata table index.
Map<Pair<String,String>,HoodieMetadataColumnStats>	getColumnStats(List<Pair<String,String>> partitionNameFileNameList, String columnName) Get column stats for files from the metadata table index.
protected HoodieEngineContext	getEngineContext()
protected String	getLatestDataInstantTime()
HoodieMetadataConfig	getMetadataConfig()
protected abstract Option<HoodieRecord<HoodieMetadataPayload>>	getRecordByKey(String key, String partitionName)
protected abstract Map<String,HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeys(List<String> keys, String partitionName)
protected abstract Map<String,List<HoodieRecord<HoodieMetadataPayload>>>	getSecondaryIndexRecords(List<String> keys, String partitionName) Returns a map of (record-key -> list-of-secondary-index-records) for the provided secondary keys.
Map<String,String>	getSecondaryKeys(List<String> recordKeys) Returns a map of (record-key -> secondary-key) for the provided record keys.
protected abstract Map<String,String>	getSecondaryKeysForRecordKeys(List<String> recordKeys, String partitionName)
protected StorageConfiguration<?>	getStorageConf()
boolean	isMetadataTableInitialized()
Map<String,List<HoodieRecordGlobalLocation>>	readRecordIndex(List<String> recordKeys) Reads record keys from record-level index.
Map<String,List<HoodieRecordGlobalLocation>>	readSecondaryIndex(List<String> secondaryKeys, String partitionName) Get record-location using secondary-index and record-index

Methods inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

extractPartitionValues, getPathPartitionLevel

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getAllRecordsByKeys, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getLatestCompactionTime, getMetadataTableBasePath, getMetadataTableBasePath, getNumFileGroupsForPartition, getPartitionPathWithPathPrefixes, getPartitionPathWithPathPrefixUsingFilterExpression, getRecordsByKeyPrefixes, getSyncedInstantTime, isMetadataTable, isMetadataTable, reset

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

dataMetaClient

protected final HoodieTableMetaClient dataMetaClient

metrics

protected final Option<HoodieMetadataMetrics> metrics

metadataConfig

protected final HoodieMetadataConfig metadataConfig

isMetadataTableInitialized

protected boolean isMetadataTableInitialized

hiveStylePartitioningEnabled

protected final boolean hiveStylePartitioningEnabled

urlEncodePartitioningEnabled

protected final boolean urlEncodePartitioningEnabled

Constructor Detail

BaseTableMetadata

protected BaseTableMetadata(HoodieEngineContext engineContext,
                            HoodieStorage storage,
                            HoodieMetadataConfig metadataConfig,
                            String dataBasePath)

Method Detail

getEngineContext

protected HoodieEngineContext getEngineContext()

getAllPartitionPaths

public List<String> getAllPartitionPaths()
                                  throws IOException

Return the list of partitions in the dataset.

If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of partitions is retrieved directly from the underlying FileSystem.

On any errors retrieving the listing from the metadata, defaults to using the file system listings.

Throws:

IOException

getAllFilesInPartition

public List<StoragePathInfo> getAllFilesInPartition(StoragePath partitionPath)
                                             throws IOException

Return the list of files in a partition.

If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of partitions is retrieved directly from the underlying FileSystem.

On any errors retrieving the listing from the metadata, defaults to using the file system listings.

Parameters:

partitionPath - The absolute path of the partition to list

Throws:

IOException

getAllFilesInPartitions

public Map<String,List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitions)
                                                          throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all files for given partition paths. NOTE: Absolute partition paths are expected here

Throws:

IOException

getBloomFilter

public Option<BloomFilter> getBloomFilter(String partitionName,
                                          String fileName)
                                   throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get the bloom filter for the FileID from the metadata table.

Parameters:

partitionName - - Partition name

fileName - - File name for which bloom filter needs to be retrieved

Returns:

BloomFilter if available, otherwise empty

Throws:

HoodieMetadataException

getBloomFilters

public Map<Pair<String,String>,BloomFilter> getBloomFilters(List<Pair<String,String>> partitionNameFileNameList)
                                                     throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get bloom filters for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

Returns:

Map of partition file name pair to its bloom filter

Throws:

HoodieMetadataException

getColumnStats

public Map<Pair<String,String>,HoodieMetadataColumnStats> getColumnStats(List<Pair<String,String>> partitionNameFileNameList,
                                                                         String columnName)
                                                                  throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get column stats for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

columnName - - Column name for which stats are needed

Returns:

Map of partition and file name pair to its column stats

Throws:

HoodieMetadataException

readRecordIndex

public Map<String,List<HoodieRecordGlobalLocation>> readRecordIndex(List<String> recordKeys)

Reads record keys from record-level index.

If the Metadata Table is not enabled, an exception is thrown to distinguish this from the absence of the key.

Parameters:

recordKeys - The list of record keys to read

readSecondaryIndex

public Map<String,List<HoodieRecordGlobalLocation>> readSecondaryIndex(List<String> secondaryKeys,
                                                                       String partitionName)

Get record-location using secondary-index and record-index

If the Metadata Table is not enabled, an exception is thrown to distinguish this from the absence of the key.

Parameters:

secondaryKeys - The list of secondary keys to read

getSecondaryKeys

public Map<String,String> getSecondaryKeys(List<String> recordKeys)

Returns a map of (record-key -> secondary-key) for the provided record keys.

fetchAllPartitionPaths

protected List<String> fetchAllPartitionPaths()

Returns a list of all partitions.

getRecordByKey

protected abstract Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKey(String key,
                                                                              String partitionName)

getRecordsByKeys

protected abstract Map<String,HoodieRecord<HoodieMetadataPayload>> getRecordsByKeys(List<String> keys,
                                                                                    String partitionName)

getSecondaryKeysForRecordKeys

protected abstract Map<String,String> getSecondaryKeysForRecordKeys(List<String> recordKeys,
                                                                    String partitionName)

getSecondaryIndexRecords

protected abstract Map<String,List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys,
                                                                                                  String partitionName)

Returns a map of (record-key -> list-of-secondary-index-records) for the provided secondary keys.

getMetadataConfig

public HoodieMetadataConfig getMetadataConfig()

getStorageConf

protected StorageConfiguration<?> getStorageConf()

getLatestDataInstantTime

protected String getLatestDataInstantTime()

isMetadataTableInitialized

public boolean isMetadataTableInitialized()