org.apache.hudi.metadata

Class HoodieBackedTableMetadataWriter

java.lang.Object

org.apache.hudi.metadata.HoodieBackedTableMetadataWriter

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadataWriter

public abstract class HoodieBackedTableMetadataWriter
extends Object
implements HoodieTableMetadataWriter

Writer implementation backed by an internal hudi table. Partition and file listing are saved within an internal MOR table called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync using the instants on the main dataset.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieTableMetaClient	dataMetaClient
protected HoodieWriteConfig	dataWriteConfig
protected boolean	enabled
protected List<MetadataPartitionType>	enabledPartitionTypes
protected HoodieEngineContext	engineContext
protected SerializableConfiguration	hadoopConf
protected HoodieBackedTableMetadata	metadata
protected HoodieTableMetaClient	metadataMetaClient
protected HoodieWriteConfig	metadataWriteConfig
protected Option<HoodieMetadataMetrics>	metrics
protected String	tableName

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, HoodieEngineContext engineContext)
protected	HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, HoodieEngineContext engineContext, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Hudi backed table metadata writer.

Method Summary

Modifier and Type	Method and Description
void	buildMetadataPartitions(HoodieEngineContext engineContext, List<HoodieIndexPartitionInfo> indexPartitionInfos) Builds the given metadata partitions to create index.
protected void	cleanIfNecessary(BaseHoodieWriteClient writeClient, String instantTime)
void	close()
protected abstract void	commit(String instantTime, Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap, boolean canTriggerTableService) Commit the HoodieRecords to Metadata Table as a new delta-commit.
protected void	compactIfNecessary(BaseHoodieWriteClient writeClient, String instantTime) Perform a compaction on the Metadata Table.
void	dropMetadataPartitions(List<MetadataPartitionType> metadataPartitions) Drop the given metadata partitions.
List<MetadataPartitionType>	getEnabledPartitionTypes()
HoodieBackedTableMetadata	getTableMetadata()
HoodieWriteConfig	getWriteConfig()
protected abstract <T extends org.apache.avro.specific.SpecificRecordBase> void	initialize(HoodieEngineContext engineContext, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Initialize the metadata table if it does not exist.
protected <T extends org.apache.avro.specific.SpecificRecordBase> void	initializeIfNeeded(HoodieTableMetaClient dataMetaClient, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Initialize the metadata table if needed.
void	initializeMetadataPartitions(HoodieTableMetaClient dataMetaClient, List<MetadataPartitionType> metadataPartitions, String instantTime) Initialize file groups for the given metadata partitions when indexing is requested.
protected abstract void	initRegistry()
void	initTableMetadata()
protected HoodieData<HoodieRecord>	prepRecords(Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap) Tag each record with the location in the given partition.
void	update(HoodieCleanMetadata cleanMetadata, String instantTime) Update from HoodieCleanMetadata.
void	update(HoodieCommitMetadata commitMetadata, String instantTime, boolean isTableServiceAction) Update from HoodieCommitMetadata.
void	update(HoodieRestoreMetadata restoreMetadata, String instantTime) Update from HoodieRestoreMetadata.
void	update(HoodieRollbackMetadata rollbackMetadata, String instantTime) Update from HoodieRollbackMetadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadataWriter

deletePartitions

Field Detail

metadataWriteConfig

protected HoodieWriteConfig metadataWriteConfig

dataWriteConfig

protected HoodieWriteConfig dataWriteConfig

tableName

protected String tableName

metadata

protected HoodieBackedTableMetadata metadata

metadataMetaClient

protected HoodieTableMetaClient metadataMetaClient

dataMetaClient

protected HoodieTableMetaClient dataMetaClient

metrics

protected Option<HoodieMetadataMetrics> metrics

enabled

protected boolean enabled

hadoopConf

protected SerializableConfiguration hadoopConf

engineContext

protected final transient HoodieEngineContext engineContext

enabledPartitionTypes

protected final List<MetadataPartitionType> enabledPartitionTypes

Constructor Detail

HoodieBackedTableMetadataWriter

protected HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                          HoodieWriteConfig writeConfig,
                                          HoodieEngineContext engineContext,
                                          Option<T> actionMetadata,
                                          Option<String> inflightInstantTimestamp)

Hudi backed table metadata writer.

Type Parameters:

T - - Action metadata types extending Avro generated SpecificRecordBase

Parameters:

hadoopConf - - Hadoop configuration to use for the metadata writer

writeConfig - - Writer config

engineContext - - Engine context

actionMetadata - - Optional action metadata to help decide initialize operations

inflightInstantTimestamp - - Timestamp of any instant in progress

HoodieBackedTableMetadataWriter

public HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                       HoodieWriteConfig writeConfig,
                                       HoodieEngineContext engineContext)

Method Detail

initRegistry

protected abstract void initRegistry()

getWriteConfig

public HoodieWriteConfig getWriteConfig()

getTableMetadata

public HoodieBackedTableMetadata getTableMetadata()

getEnabledPartitionTypes

public List<MetadataPartitionType> getEnabledPartitionTypes()

initialize

protected abstract <T extends org.apache.avro.specific.SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
                                                                                           Option<T> actionMetadata,
                                                                                           Option<String> inflightInstantTimestamp)

Initialize the metadata table if it does not exist.

If the metadata table does not exist, then file and partition listing is used to initialize the table.

Parameters:

engineContext -

actionMetadata - Action metadata types extending Avro generated SpecificRecordBase

inflightInstantTimestamp - Timestamp of an instant in progress on the dataset. This instant is ignored while deciding to initialize the metadata table.

initTableMetadata

public void initTableMetadata()

initializeIfNeeded

protected <T extends org.apache.avro.specific.SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
                                                                                          Option<T> actionMetadata,
                                                                                          Option<String> inflightInstantTimestamp)
                                                                                   throws IOException

Initialize the metadata table if needed.

Type Parameters:

T - - action metadata types extending Avro generated SpecificRecordBase

Parameters:

dataMetaClient - - meta client for the data table

actionMetadata - - optional action metadata

inflightInstantTimestamp - - timestamp of an instant in progress on the dataset

Throws:

IOException

initializeMetadataPartitions

public void initializeMetadataPartitions(HoodieTableMetaClient dataMetaClient,
                                         List<MetadataPartitionType> metadataPartitions,
                                         String instantTime)
                                  throws IOException

Description copied from interface: HoodieTableMetadataWriter

Initialize file groups for the given metadata partitions when indexing is requested.

Specified by:

initializeMetadataPartitions in interface HoodieTableMetadataWriter

Parameters:

dataMetaClient - - meta client for the data table

metadataPartitions - - metadata partitions for which file groups needs to be initialized

instantTime - - instant time of the index action

Throws:

IOException

dropMetadataPartitions

public void dropMetadataPartitions(List<MetadataPartitionType> metadataPartitions)
                            throws IOException

Description copied from interface: HoodieTableMetadataWriter

Drop the given metadata partitions.

Specified by:

dropMetadataPartitions in interface HoodieTableMetadataWriter

Throws:

IOException

buildMetadataPartitions

public void buildMetadataPartitions(HoodieEngineContext engineContext,
                                    List<HoodieIndexPartitionInfo> indexPartitionInfos)

Description copied from interface: HoodieTableMetadataWriter

Builds the given metadata partitions to create index.

Specified by:

buildMetadataPartitions in interface HoodieTableMetadataWriter

indexPartitionInfos - - information about partitions to build such as partition type and base instant time

update

public void update(HoodieCommitMetadata commitMetadata,
                   String instantTime,
                   boolean isTableServiceAction)

Update from HoodieCommitMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

commitMetadata - HoodieCommitMetadata

instantTime - Timestamp at which the commit was performed

isTableServiceAction - true if commit metadata is pertaining to a table service. false otherwise.

update

public void update(HoodieCleanMetadata cleanMetadata,
                   String instantTime)

Update from HoodieCleanMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

cleanMetadata - HoodieCleanMetadata

instantTime - Timestamp at which the clean was completed

update

public void update(HoodieRestoreMetadata restoreMetadata,
                   String instantTime)

Update from HoodieRestoreMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

restoreMetadata - HoodieRestoreMetadata

instantTime - Timestamp at which the restore was performed

update

public void update(HoodieRollbackMetadata rollbackMetadata,
                   String instantTime)

Update from HoodieRollbackMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

rollbackMetadata - HoodieRollbackMetadata

instantTime - Timestamp at which the rollback was performed

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

commit

protected abstract void commit(String instantTime,
                               Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap,
                               boolean canTriggerTableService)

Commit the HoodieRecords to Metadata Table as a new delta-commit.

Parameters:

instantTime - - Action instant time for this commit

partitionRecordsMap - - Map of partition name to its records to commit

canTriggerTableService - true if table services can be scheduled and executed. false otherwise.

prepRecords

protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap)

Tag each record with the location in the given partition. The record is tagged with respective file slice's location based on its record key.

compactIfNecessary

protected void compactIfNecessary(BaseHoodieWriteClient writeClient,
                                  String instantTime)

Perform a compaction on the Metadata Table. Cases to be handled: 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx. 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a deltacommit.

cleanIfNecessary

protected void cleanIfNecessary(BaseHoodieWriteClient writeClient,
                                String instantTime)