org.apache.hudi.metadata

Class HoodieBackedTableMetadataWriter

java.lang.Object

org.apache.hudi.metadata.HoodieBackedTableMetadataWriter

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadataWriter

public abstract class HoodieBackedTableMetadataWriter
extends Object
implements HoodieTableMetadataWriter

Writer implementation backed by an internal hudi table. Partition and file listing are saved within an internal MOR table called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync using the instants on the main dataset.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieTableMetaClient	dataMetaClient
protected HoodieWriteConfig	dataWriteConfig
protected boolean	enabled
protected HoodieEngineContext	engineContext
protected SerializableConfiguration	hadoopConf
protected HoodieBackedTableMetadata	metadata
protected HoodieTableMetaClient	metadataMetaClient
protected HoodieWriteConfig	metadataWriteConfig
protected Option<HoodieMetadataMetrics>	metrics
protected String	tableName

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, HoodieEngineContext engineContext)
protected	HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, HoodieEngineContext engineContext, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Hudi backed table metadata writer.

Method Summary

Modifier and Type	Method and Description
protected void	bootstrapCommit(List<org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.DirectoryInfo> partitionInfoList, String createInstantTime) This is invoked to bootstrap metadata table for a dataset.
protected <T extends org.apache.avro.specific.SpecificRecordBase> void	bootstrapIfNeeded(HoodieEngineContext engineContext, HoodieTableMetaClient dataMetaClient, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Bootstrap the metadata table if needed.
protected void	cleanIfNecessary(AbstractHoodieWriteClient writeClient, String instantTime)
void	close()
protected abstract void	commit(HoodieData<HoodieRecord> records, String partitionName, String instantTime, boolean canTriggerTableService) Commit the HoodieRecords to Metadata Table as a new delta-commit.
protected void	compactIfNecessary(AbstractHoodieWriteClient writeClient, String instantTime) Perform a compaction on the Metadata Table.
HoodieWriteConfig	getWriteConfig()
protected abstract <T extends org.apache.avro.specific.SpecificRecordBase> void	initialize(HoodieEngineContext engineContext, Option<T> actionMetadata, Option<String> inflightInstantTimestamp) Initialize the metadata table if it does not exist.
protected abstract void	initRegistry()
void	initTableMetadata()
HoodieBackedTableMetadata	metadata()
void	update(HoodieCleanMetadata cleanMetadata, String instantTime) Update from HoodieCleanMetadata.
void	update(HoodieCommitMetadata commitMetadata, String instantTime, boolean isTableServiceAction) Update from HoodieCommitMetadata.
void	update(HoodieRestoreMetadata restoreMetadata, String instantTime) Update from HoodieRestoreMetadata.
void	update(HoodieRollbackMetadata rollbackMetadata, String instantTime) Update from HoodieRollbackMetadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metadataWriteConfig

protected HoodieWriteConfig metadataWriteConfig

dataWriteConfig

protected HoodieWriteConfig dataWriteConfig

tableName

protected String tableName

metadata

protected HoodieBackedTableMetadata metadata

metadataMetaClient

protected HoodieTableMetaClient metadataMetaClient

dataMetaClient

protected HoodieTableMetaClient dataMetaClient

metrics

protected Option<HoodieMetadataMetrics> metrics

enabled

protected boolean enabled

hadoopConf

protected SerializableConfiguration hadoopConf

engineContext

protected final transient HoodieEngineContext engineContext

Constructor Detail

HoodieBackedTableMetadataWriter

protected HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                          HoodieWriteConfig writeConfig,
                                          HoodieEngineContext engineContext,
                                          Option<T> actionMetadata,
                                          Option<String> inflightInstantTimestamp)

Hudi backed table metadata writer.

Type Parameters:

T - - Action metadata types extending Avro generated SpecificRecordBase

Parameters:

hadoopConf - - Hadoop configuration to use for the metadata writer

writeConfig - - Writer config

engineContext - - Engine context

actionMetadata - - Optional action metadata to help decide bootstrap operations

inflightInstantTimestamp - - Timestamp of any instant in progress

HoodieBackedTableMetadataWriter

public HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                       HoodieWriteConfig writeConfig,
                                       HoodieEngineContext engineContext)

Method Detail

initRegistry

protected abstract void initRegistry()

getWriteConfig

public HoodieWriteConfig getWriteConfig()

metadata

public HoodieBackedTableMetadata metadata()

initialize

protected abstract <T extends org.apache.avro.specific.SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
                                                                                           Option<T> actionMetadata,
                                                                                           Option<String> inflightInstantTimestamp)

Initialize the metadata table if it does not exist. If the metadata table does not exist, then file and partition listing is used to bootstrap the table.

Parameters:

engineContext -

actionMetadata - Action metadata types extending Avro generated SpecificRecordBase

inflightInstantTimestamp - Timestamp of an instant in progress on the dataset. This instant is ignored while deciding to bootstrap the metadata table.

initTableMetadata

public void initTableMetadata()

bootstrapIfNeeded

protected <T extends org.apache.avro.specific.SpecificRecordBase> void bootstrapIfNeeded(HoodieEngineContext engineContext,
                                                                                         HoodieTableMetaClient dataMetaClient,
                                                                                         Option<T> actionMetadata,
                                                                                         Option<String> inflightInstantTimestamp)
                                                                                  throws IOException

Bootstrap the metadata table if needed.

Type Parameters:

T - - Action metadata types extending Avro generated SpecificRecordBase

Parameters:

engineContext - - Engine context

dataMetaClient - - Meta client for the data table

actionMetadata - - Optional action metadata

inflightInstantTimestamp - - Timestamp of an instant in progress on the dataset. This instant is ignored

Throws:

IOException

update

public void update(HoodieCommitMetadata commitMetadata,
                   String instantTime,
                   boolean isTableServiceAction)

Update from HoodieCommitMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

commitMetadata - HoodieCommitMetadata

instantTime - Timestamp at which the commit was performed

isTableServiceAction - true if commit metadata is pertaining to a table service. false otherwise.

update

public void update(HoodieCleanMetadata cleanMetadata,
                   String instantTime)

Update from HoodieCleanMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

cleanMetadata - HoodieCleanMetadata

instantTime - Timestamp at which the clean was completed

update

public void update(HoodieRestoreMetadata restoreMetadata,
                   String instantTime)

Update from HoodieRestoreMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

restoreMetadata - HoodieRestoreMetadata

instantTime - Timestamp at which the restore was performed

update

public void update(HoodieRollbackMetadata rollbackMetadata,
                   String instantTime)

Update from HoodieRollbackMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

rollbackMetadata - HoodieRollbackMetadata

instantTime - Timestamp at which the rollback was performed

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

commit

protected abstract void commit(HoodieData<HoodieRecord> records,
                               String partitionName,
                               String instantTime,
                               boolean canTriggerTableService)

Commit the HoodieRecords to Metadata Table as a new delta-commit.

Parameters:

records - The HoodieData of records to be written.

partitionName - The partition to which the records are to be written.

instantTime - The timestamp to use for the deltacommit.

canTriggerTableService - true if table services can be scheduled and executed. false otherwise.

compactIfNecessary

protected void compactIfNecessary(AbstractHoodieWriteClient writeClient,
                                  String instantTime)

Perform a compaction on the Metadata Table. Cases to be handled: 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx. 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a deltacommit.

cleanIfNecessary

protected void cleanIfNecessary(AbstractHoodieWriteClient writeClient,
                                String instantTime)

bootstrapCommit

protected void bootstrapCommit(List<org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.DirectoryInfo> partitionInfoList,
                               String createInstantTime)

This is invoked to bootstrap metadata table for a dataset. Bootstrap Commit has special handling mechanism due to its scale compared to other regular commits.