org.apache.hudi.metadata

Class HoodieBackedTableMetadata

java.lang.Object

org.apache.hudi.metadata.AbstractHoodieTableMetadata

org.apache.hudi.metadata.BaseTableMetadata

org.apache.hudi.metadata.HoodieBackedTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

public class HoodieBackedTableMetadata
extends BaseTableMetadata

Table metadata provided by an internal DFS backed Hudi metadata table.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.metadata.BaseTableMetadata

dataMetaClient, hiveStylePartitioningEnabled, isMetadataTableInitialized, metadataConfig, metrics, urlEncodePartitioningEnabled

Fields inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

caseSensitive, dataBasePath, engineContext, storage, storageConf

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Constructor and Description
HoodieBackedTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath)
HoodieBackedTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath, boolean reuse)

Method Summary

Modifier and Type	Method and Description
void	close()
boolean	enabled()
Map<String,List<HoodieRecord<HoodieMetadataPayload>>>	getAllRecordsByKeys(List<String> keys, String partitionName) Fetch records for given keys.
Option<String>	getLatestCompactionTime() Returns the timestamp of the latest compaction.
Pair<HoodieMetadataLogRecordReader,Long>	getLogRecordScanner(List<HoodieLogFile> logFiles, String partitionName, Option<Boolean> allowFullScanOverride)
HoodieTableFileSystemView	getMetadataFileSystemView()
HoodieTableMetaClient	getMetadataMetaClient()
int	getNumFileGroupsForPartition(MetadataPartitionType partition) Returns the number of shards in a metadata table partition.
List<String>	getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes) Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
List<String>	getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes, Types.RecordType partitionFields, Expression expression) Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.
protected Option<HoodieRecord<HoodieMetadataPayload>>	getRecordByKey(String key, String partitionName)
HoodieData<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes, String partitionName, boolean shouldLoadInMemory) Fetch records by key prefixes.
protected Map<String,HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeys(List<String> keys, String partitionName)
protected Map<String,List<HoodieRecord<HoodieMetadataPayload>>>	getSecondaryIndexRecords(List<String> keys, String partitionName) Returns a map of (record-key -> list-of-secondary-index-records) for the provided secondary keys.
protected Map<String,String>	getSecondaryKeysForRecordKeys(List<String> recordKeys, String partitionName)
Option<String>	getSyncedInstantTime() Get the instant time to which the metadata is synced w.r.t data timeline.
void	reset() Clear the states of the table metadata.
Map<String,String>	stats()

Methods inherited from class org.apache.hudi.metadata.BaseTableMetadata

fetchAllPartitionPaths, getAllFilesInPartition, getAllFilesInPartitions, getAllPartitionPaths, getBloomFilter, getBloomFilters, getColumnStats, getEngineContext, getLatestDataInstantTime, getMetadataConfig, getSecondaryKeys, getStorageConf, isMetadataTableInitialized, readRecordIndex, readSecondaryIndex

Methods inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

extractPartitionValues, getPathPartitionLevel

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getMetadataTableBasePath, getMetadataTableBasePath, isMetadataTable, isMetadataTable

Constructor Detail

HoodieBackedTableMetadata

public HoodieBackedTableMetadata(HoodieEngineContext engineContext,
                                 HoodieStorage storage,
                                 HoodieMetadataConfig metadataConfig,
                                 String datasetBasePath)

HoodieBackedTableMetadata

public HoodieBackedTableMetadata(HoodieEngineContext engineContext,
                                 HoodieStorage storage,
                                 HoodieMetadataConfig metadataConfig,
                                 String datasetBasePath,
                                 boolean reuse)

Method Detail

getRecordByKey

protected Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKey(String key,
                                                                     String partitionName)

Specified by:

getRecordByKey in class BaseTableMetadata

getPartitionPathWithPathPrefixUsingFilterExpression

public List<String> getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes,
                                                                        Types.RecordType partitionFields,
                                                                        Expression expression)
                                                                 throws IOException

Description copied from interface: HoodieTableMetadata

Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.

Throws:

IOException

getPartitionPathWithPathPrefixes

public List<String> getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes)
                                              throws IOException

Description copied from interface: HoodieTableMetadata

Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.

E.g., Table has partition 4 partitions: year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04 The relative path "year=2022" returns all partitions, while the relative path "year=2022/month=07" returns only two partitions.

Throws:

IOException

getRecordsByKeyPrefixes

public HoodieData<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes,
                                                                               String partitionName,
                                                                               boolean shouldLoadInMemory)

Description copied from interface: HoodieTableMetadata

Fetch records by key prefixes. Key prefix passed is expected to match the same prefix as stored in Metadata table partitions. For eg, in case of col stats partition, actual keys in metadata partition is encoded values of column name, partition name and file name. So, key prefixes passed to this method is expected to be encoded already.

Parameters:

keyPrefixes - list of key prefixes for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

HoodieData of HoodieRecords with records matching the passed in key prefixes.

getRecordsByKeys

protected Map<String,HoodieRecord<HoodieMetadataPayload>> getRecordsByKeys(List<String> keys,
                                                                           String partitionName)

Specified by:

getRecordsByKeys in class BaseTableMetadata

getAllRecordsByKeys

public Map<String,List<HoodieRecord<HoodieMetadataPayload>>> getAllRecordsByKeys(List<String> keys,
                                                                                 String partitionName)

Description copied from interface: HoodieTableMetadata

Fetch records for given keys. A key could have multiple records associated with it. This method returns all the records for given keys.

Parameters:

keys - list of key for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

Map of key to List of HoodieRecords with records matching the passed in keys.

getLogRecordScanner

public Pair<HoodieMetadataLogRecordReader,Long> getLogRecordScanner(List<HoodieLogFile> logFiles,
                                                                    String partitionName,
                                                                    Option<Boolean> allowFullScanOverride)

close

public void close()

enabled

public boolean enabled()

getMetadataMetaClient

public HoodieTableMetaClient getMetadataMetaClient()

getMetadataFileSystemView

public HoodieTableFileSystemView getMetadataFileSystemView()

stats

public Map<String,String> stats()

getSyncedInstantTime

public Option<String> getSyncedInstantTime()

Description copied from interface: HoodieTableMetadata

Get the instant time to which the metadata is synced w.r.t data timeline.

getLatestCompactionTime

public Option<String> getLatestCompactionTime()

Description copied from interface: HoodieTableMetadata

Returns the timestamp of the latest compaction.

reset

public void reset()

Description copied from interface: HoodieTableMetadata

Clear the states of the table metadata.

getNumFileGroupsForPartition

public int getNumFileGroupsForPartition(MetadataPartitionType partition)

Description copied from interface: HoodieTableMetadata

Returns the number of shards in a metadata table partition.

getSecondaryKeysForRecordKeys

protected Map<String,String> getSecondaryKeysForRecordKeys(List<String> recordKeys,
                                                           String partitionName)

Specified by:

getSecondaryKeysForRecordKeys in class BaseTableMetadata

getSecondaryIndexRecords

protected Map<String,List<HoodieRecord<HoodieMetadataPayload>>> getSecondaryIndexRecords(List<String> keys,
                                                                                         String partitionName)

Description copied from class: BaseTableMetadata

Returns a map of (record-key -> list-of-secondary-index-records) for the provided secondary keys.

Specified by:

getSecondaryIndexRecords in class BaseTableMetadata