org.apache.hudi.client

Class BaseHoodieTableServiceClient<I,T,O>

java.lang.Object

org.apache.hudi.client.BaseHoodieClient

org.apache.hudi.client.BaseHoodieTableServiceClient<I,T,O>

Type Parameters:

I - The HoodieTable implementation's input type

T - The HoodieTable implementation's output type

O - The BaseHoodieWriteClient implementation's output type (differs in case of spark)

All Implemented Interfaces:

Serializable, AutoCloseable, RunsTableService

public abstract class BaseHoodieTableServiceClient<I,T,O>
extends BaseHoodieClient
implements RunsTableService

Base class for all shared logic between table service clients regardless of engine.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected AsyncArchiveService	asyncArchiveService
protected AsyncCleanerService	asyncCleanerService
protected com.codahale.metrics.Timer.Context	clusteringTimer
protected com.codahale.metrics.Timer.Context	compactionTimer
protected com.codahale.metrics.Timer.Context	logCompactionTimer
protected Set<String>	pendingInflightAndRequestedInstants

Fields inherited from class org.apache.hudi.client.BaseHoodieClient

basePath, config, context, heartbeatClient, metrics, storage, storageConf, txnManager

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, Option<EmbeddedTimelineService> timelineService)

Method Summary

Modifier and Type	Method and Description
protected void	archive(HoodieTable table) Trigger archival for the table.
protected void	asyncArchive()
protected void	asyncClean()
HoodieCleanMetadata	clean(String cleanInstantTime, boolean scheduleInline) Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used.
HoodieCleanMetadata	clean(String cleanInstantTime, boolean scheduleInline, boolean skipLocking) Deprecated.
void	close() Releases any resources used by the client.
HoodieWriteMetadata<O>	cluster(String clusteringInstant, boolean shouldComplete) Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
void	commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata, Option<Map<String,String>> extraMetadata) Commit a compaction operation.
protected HoodieWriteMetadata<O>	compact(String compactionInstantTime, boolean shouldComplete) Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
protected void	completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime) Commit Compaction and track metrics.
protected void	completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) Commit Log Compaction and track metrics.
protected abstract HoodieWriteMetadata<O>	convertToOutputMetadata(HoodieWriteMetadata<T> writeMetadata)
protected HoodieTable<?,I,?,T>	createTable(HoodieWriteConfig config, StorageConfiguration<?> storageConf)
protected abstract HoodieTable<?,I,?,T>	createTable(HoodieWriteConfig config, StorageConfiguration<?> storageConf, boolean skipValidation)
protected HoodieTable	createTableAndValidate(HoodieWriteConfig config, BiFunction<HoodieWriteConfig,HoodieEngineContext,HoodieTable> createTableFn, boolean skipValidation)
protected List<String>	getInstantsToRollback(HoodieTableMetaClient metaClient, HoodieFailedWritesCleaningPolicy cleaningPolicy, Option<String> curInstantTime)
protected Option<HoodiePendingRollbackInfo>	getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback)
Option<HoodiePendingRollbackInfo>	getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback, boolean ignoreCompactionAndClusteringInstants)
protected Map<String,Option<HoodiePendingRollbackInfo>>	getPendingRollbackInfos(HoodieTableMetaClient metaClient)
protected Map<String,Option<HoodiePendingRollbackInfo>>	getPendingRollbackInfos(HoodieTableMetaClient metaClient, boolean ignoreCompactionAndClusteringInstants) Fetch map of pending commits to be rolled-back to HoodiePendingRollbackInfo.
protected void	handleWriteErrors(List<HoodieWriteStat> writeStats, TableServiceType tableServiceType)
protected Option<String>	inlineClustering(Option<Map<String,String>> extraMetadata) Executes a clustering plan on a table, serially before or after an insert/upsert action.
protected Option<String>	inlineCompaction(Option<Map<String,String>> extraMetadata) Performs a compaction operation on a table, serially before or after an insert/upsert action.
protected Option<String>	inlineLogCompact(Option<Map<String,String>> extraMetadata) Performs a log compaction operation on a table, serially before or after an insert/upsert action.
protected Option<String>	inlineScheduleClustering(Option<Map<String,String>> extraMetadata) Schedules clustering inline.
protected Option<String>	inlineScheduleCompaction(Option<Map<String,String>> extraMetadata) Schedules compaction inline.
protected boolean	isPreCommitRequired() Some writers use SparkAllowUpdateStrategy and treat replacecommit plan as revocable plan.
HoodieWriteMetadata<O>	logCompact(String logCompactionInstantTime) Performs Log Compaction for the workload stored in instant-time.
protected HoodieWriteMetadata<O>	logCompact(String logCompactionInstantTime, boolean shouldComplete) Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
HoodieWriteMetadata<T>	managePartitionTTL(String instantTime) Delete expired partition by config.
protected void	preCommit(HoodieCommitMetadata metadata) Any pre-commit actions like conflict resolution goes here.
boolean	purgePendingClustering(String clusteringInstant)
protected void	releaseResources(String instantTime) Called after each commit of a compaction or clustering table service, to release any resources used.
boolean	rollback(String commitInstantTime, Option<HoodiePendingRollbackInfo> pendingRollbackInfo, boolean skipLocking, boolean skipVersionCheck) Deprecated.
boolean	rollback(String commitInstantTime, Option<HoodiePendingRollbackInfo> pendingRollbackInfo, String rollbackInstantTime, boolean skipLocking, boolean skipVersionCheck) Deprecated.
void	rollbackFailedBootstrap() Main API to rollback failed bootstrap.
protected boolean	rollbackFailedIndexingCommits() Rolls back the failed delta commits corresponding to the indexing action.
protected Boolean	rollbackFailedWrites() Rollback all failed writes.
protected void	rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback)
protected void	rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback, boolean skipLocking, boolean skipVersionCheck)
protected void	runAnyPendingClustering(HoodieTable table)
protected void	runAnyPendingCompactions(HoodieTable table)
protected void	runAnyPendingLogCompactions(HoodieTable table)
protected void	runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String,String>> extraMetadata)
Option<String>	scheduleClustering(Option<Map<String,String>> extraMetadata) Schedules a new clustering instant.
boolean	scheduleClusteringAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new clustering instant with passed-in instant time.
Option<String>	scheduleCompaction(Option<Map<String,String>> extraMetadata) Schedules a new compaction instant.
boolean	scheduleCompactionAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new compaction instant with passed-in instant time.
Option<String>	scheduleLogCompaction(Option<Map<String,String>> extraMetadata) Schedules a new log compaction instant.
boolean	scheduleLogCompactionAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new log compaction instant with passed-in instant time.
Option<String>	scheduleTableService(String instantTime, Option<Map<String,String>> extraMetadata, TableServiceType tableServiceType) Schedule table services such as clustering, compaction & cleaning.
protected Option<String>	scheduleTableServiceInternal(String instantTime, Option<Map<String,String>> extraMetadata, TableServiceType tableServiceType)
protected void	setPendingInflightAndRequestedInstants(Set<String> pendingInflightAndRequestedInstants)
protected void	setTableServiceTimer(WriteOperationType operationType)
protected void	startAsyncArchiveService(BaseHoodieWriteClient writeClient)
protected void	startAsyncCleanerService(BaseHoodieWriteClient writeClient)
protected abstract void	validateClusteringCommit(HoodieWriteMetadata<O> clusteringMetadata, String clusteringCommitTime, HoodieTable table)

Methods inherited from class org.apache.hudi.client.BaseHoodieClient

createMetaClient, createNewInstantTime, createNewInstantTime, finalizeWrite, getConfig, getEngineContext, getHeartbeatClient, getTimelineServer, initWrapperFSMetrics, resolveWriteConflict, writeTableMetadata

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.client.RunsTableService

shouldDelegateToTableServiceManager, tableServicesEnabled

Field Detail

compactionTimer

protected transient com.codahale.metrics.Timer.Context compactionTimer

clusteringTimer

protected transient com.codahale.metrics.Timer.Context clusteringTimer

logCompactionTimer

protected transient com.codahale.metrics.Timer.Context logCompactionTimer

asyncCleanerService

protected transient AsyncCleanerService asyncCleanerService

asyncArchiveService

protected transient AsyncArchiveService asyncArchiveService

pendingInflightAndRequestedInstants

protected Set<String> pendingInflightAndRequestedInstants

Constructor Detail

BaseHoodieTableServiceClient

protected BaseHoodieTableServiceClient(HoodieEngineContext context,
                                       HoodieWriteConfig clientConfig,
                                       Option<EmbeddedTimelineService> timelineService)

Method Detail

startAsyncCleanerService

protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient)

startAsyncArchiveService

protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient)

asyncClean

protected void asyncClean()

asyncArchive

protected void asyncArchive()

setTableServiceTimer

protected void setTableServiceTimer(WriteOperationType operationType)

setPendingInflightAndRequestedInstants

protected void setPendingInflightAndRequestedInstants(Set<String> pendingInflightAndRequestedInstants)

preCommit

protected void preCommit(HoodieCommitMetadata metadata)

Any pre-commit actions like conflict resolution goes here.

Parameters:

metadata - commit metadata for which pre commit is being invoked.

inlineCompaction

protected Option<String> inlineCompaction(Option<Map<String,String>> extraMetadata)

Performs a compaction operation on a table, serially before or after an insert/upsert action. Scheduling and execution is done inline.

logCompact

protected HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime,
                                            boolean shouldComplete)

Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s

Parameters:

logCompactionInstantTime - Compaction Instant Time

Returns:

Collection of Write Status

inlineLogCompact

protected Option<String> inlineLogCompact(Option<Map<String,String>> extraMetadata)

Performs a log compaction operation on a table, serially before or after an insert/upsert action.

runAnyPendingCompactions

protected void runAnyPendingCompactions(HoodieTable table)

runAnyPendingLogCompactions

protected void runAnyPendingLogCompactions(HoodieTable table)

inlineScheduleCompaction

protected Option<String> inlineScheduleCompaction(Option<Map<String,String>> extraMetadata)

Schedules compaction inline.

Parameters:

extraMetadata - extra metadata to be used.

Returns:

compaction instant if scheduled.

scheduleCompaction

public Option<String> scheduleCompaction(Option<Map<String,String>> extraMetadata)
                                  throws HoodieIOException

Schedules a new compaction instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

compact

protected HoodieWriteMetadata<O> compact(String compactionInstantTime,
                                         boolean shouldComplete)

Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.

Parameters:

compactionInstantTime - Compaction Instant Time

Returns:

Collection of Write Status

commitCompaction

public void commitCompaction(String compactionInstantTime,
                             HoodieCommitMetadata metadata,
                             Option<Map<String,String>> extraMetadata)

Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.

Parameters:

compactionInstantTime - Compaction Instant Time

metadata - All the metadata that gets stored along with a commit

extraMetadata - Extra Metadata to be stored

completeCompaction

protected void completeCompaction(HoodieCommitMetadata metadata,
                                  HoodieTable table,
                                  String compactionCommitTime)

Commit Compaction and track metrics.

scheduleLogCompaction

public Option<String> scheduleLogCompaction(Option<Map<String,String>> extraMetadata)
                                     throws HoodieIOException

Schedules a new log compaction instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleLogCompactionAtInstant

public boolean scheduleLogCompactionAtInstant(String instantTime,
                                              Option<Map<String,String>> extraMetadata)
                                       throws HoodieIOException

Schedules a new log compaction instant with passed-in instant time.

Parameters:

instantTime - Log Compaction Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

logCompact

public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime)

Performs Log Compaction for the workload stored in instant-time.

Parameters:

logCompactionInstantTime - Log Compaction Instant Time

Returns:

Collection of WriteStatus to inspect errors and counts

completeLogCompaction

protected void completeLogCompaction(HoodieCommitMetadata metadata,
                                     HoodieTable table,
                                     String logCompactionCommitTime)

Commit Log Compaction and track metrics.

scheduleCompactionAtInstant

public boolean scheduleCompactionAtInstant(String instantTime,
                                           Option<Map<String,String>> extraMetadata)
                                    throws HoodieIOException

Schedules a new compaction instant with passed-in instant time.

Parameters:

instantTime - Compaction Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleClustering

public Option<String> scheduleClustering(Option<Map<String,String>> extraMetadata)
                                  throws HoodieIOException

Schedules a new clustering instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleClusteringAtInstant

public boolean scheduleClusteringAtInstant(String instantTime,
                                           Option<Map<String,String>> extraMetadata)
                                    throws HoodieIOException

Schedules a new clustering instant with passed-in instant time.

Parameters:

instantTime - clustering Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

cluster

public HoodieWriteMetadata<O> cluster(String clusteringInstant,
                                      boolean shouldComplete)

Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.

Parameters:

clusteringInstant - Clustering Instant Time

Returns:

Collection of Write Status

purgePendingClustering

public boolean purgePendingClustering(String clusteringInstant)

managePartitionTTL

public HoodieWriteMetadata<T> managePartitionTTL(String instantTime)

Delete expired partition by config.

Parameters:

instantTime - Instant Time for the action

Returns:

HoodieWriteMetadata

validateClusteringCommit

protected abstract void validateClusteringCommit(HoodieWriteMetadata<O> clusteringMetadata,
                                                 String clusteringCommitTime,
                                                 HoodieTable table)

convertToOutputMetadata

protected abstract HoodieWriteMetadata<O> convertToOutputMetadata(HoodieWriteMetadata<T> writeMetadata)

runTableServicesInline

protected void runTableServicesInline(HoodieTable table,
                                      HoodieCommitMetadata metadata,
                                      Option<Map<String,String>> extraMetadata)

scheduleTableService

public Option<String> scheduleTableService(String instantTime,
                                           Option<Map<String,String>> extraMetadata,
                                           TableServiceType tableServiceType)

Schedule table services such as clustering, compaction & cleaning.

Parameters:

extraMetadata - Metadata to pass onto the scheduled service instant

tableServiceType - Type of table service to schedule

Returns:

scheduleTableServiceInternal

protected Option<String> scheduleTableServiceInternal(String instantTime,
                                                      Option<Map<String,String>> extraMetadata,
                                                      TableServiceType tableServiceType)

createTableAndValidate

protected HoodieTable createTableAndValidate(HoodieWriteConfig config,
                                             BiFunction<HoodieWriteConfig,HoodieEngineContext,HoodieTable> createTableFn,
                                             boolean skipValidation)

createTable

protected HoodieTable<?,I,?,T> createTable(HoodieWriteConfig config,
                                           StorageConfiguration<?> storageConf)

createTable

protected abstract HoodieTable<?,I,?,T> createTable(HoodieWriteConfig config,
                                                    StorageConfiguration<?> storageConf,
                                                    boolean skipValidation)

inlineClustering

protected Option<String> inlineClustering(Option<Map<String,String>> extraMetadata)

Executes a clustering plan on a table, serially before or after an insert/upsert action. Schedules and executes clustering inline.

inlineScheduleClustering

protected Option<String> inlineScheduleClustering(Option<Map<String,String>> extraMetadata)

Schedules clustering inline.

Parameters:

extraMetadata - extra metadata to use.

Returns:

clustering instant if scheduled.

runAnyPendingClustering

protected void runAnyPendingClustering(HoodieTable table)

clean

@Nullable
 @Deprecated
public HoodieCleanMetadata clean(String cleanInstantTime,
                                                        boolean scheduleInline,
                                                        boolean skipLocking)
                                                 throws HoodieIOException

Deprecated.

Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used. (typically files that no longer can be used by a running query can be cleaned). This API provides the flexibility to schedule clean instant asynchronously via scheduleTableService(String, Option, TableServiceType) and disable inline scheduling of clean.

Parameters:

cleanInstantTime - instant time for clean.

scheduleInline - true if needs to be scheduled inline. false otherwise.

Throws:

HoodieIOException

clean

@Nullable
public HoodieCleanMetadata clean(String cleanInstantTime,
                                           boolean scheduleInline)
                                    throws HoodieIOException

Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used. (typically files that no longer can be used by a running query can be cleaned). This API provides the flexibility to schedule clean instant asynchronously via scheduleTableService(String, Option, TableServiceType) and disable inline scheduling of clean.

Parameters:

cleanInstantTime - instant time for clean.

scheduleInline - true if needs to be scheduled inline. false otherwise.

Throws:

HoodieIOException

archive

protected void archive(HoodieTable table)

Trigger archival for the table. This ensures that the number of commits do not explode and keep increasing unbounded over time.

Parameters:

table - table to commit on.

getPendingRollbackInfo

protected Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient,
                                                                   String commitToRollback)

getPendingRollbackInfo

public Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient,
                                                                String commitToRollback,
                                                                boolean ignoreCompactionAndClusteringInstants)

getPendingRollbackInfos

protected Map<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient)

getPendingRollbackInfos

protected Map<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient,
                                                                                boolean ignoreCompactionAndClusteringInstants)

Fetch map of pending commits to be rolled-back to HoodiePendingRollbackInfo.

Parameters:

metaClient - instance of HoodieTableMetaClient to use.

Returns:

map of pending commits to be rolled-back instants to Rollback Instant and Rollback plan Pair.

rollbackFailedIndexingCommits

protected boolean rollbackFailedIndexingCommits()

Rolls back the failed delta commits corresponding to the indexing action.

TODO(HUDI-5733): This should be cleaned up once the proper fix of rollbacks in the metadata table is landed.

Returns:

true if rollback happens; false otherwise.

rollbackFailedWrites

protected Boolean rollbackFailedWrites()

Rollback all failed writes.

Returns:

true if rollback was triggered. false otherwise.

rollbackFailedWrites

protected void rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback)

rollbackFailedWrites

protected void rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback,
                                    boolean skipLocking,
                                    boolean skipVersionCheck)

getInstantsToRollback

protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient,
                                             HoodieFailedWritesCleaningPolicy cleaningPolicy,
                                             Option<String> curInstantTime)

rollback

@Deprecated
public boolean rollback(String commitInstantTime,
                                    Option<HoodiePendingRollbackInfo> pendingRollbackInfo,
                                    boolean skipLocking,
                                    boolean skipVersionCheck)
                             throws HoodieRollbackException

Deprecated.

Parameters:

commitInstantTime - Instant time of the commit

pendingRollbackInfo - pending rollback instant and plan if rollback failed from previous attempt.

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Throws:

HoodieRollbackException - if rollback cannot be performed successfully

rollback

@Deprecated
public boolean rollback(String commitInstantTime,
                                    Option<HoodiePendingRollbackInfo> pendingRollbackInfo,
                                    String rollbackInstantTime,
                                    boolean skipLocking,
                                    boolean skipVersionCheck)
                             throws HoodieRollbackException

Deprecated.

Parameters:

commitInstantTime - Instant time of the commit

pendingRollbackInfo - pending rollback instant and plan if rollback failed from previous attempt.

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Throws:

HoodieRollbackException - if rollback cannot be performed successfully

rollbackFailedBootstrap

public void rollbackFailedBootstrap()

Main API to rollback failed bootstrap.

isPreCommitRequired

protected boolean isPreCommitRequired()

Some writers use SparkAllowUpdateStrategy and treat replacecommit plan as revocable plan. In those cases, their ConflictResolutionStrategy implementation should run conflict resolution even for clustering operations.

Returns:

boolean

close

public void close()

Description copied from class: BaseHoodieClient

Releases any resources used by the client.

Specified by:

close in interface AutoCloseable

Overrides:

close in class BaseHoodieClient

handleWriteErrors

protected void handleWriteErrors(List<HoodieWriteStat> writeStats,
                                 TableServiceType tableServiceType)

releaseResources

protected void releaseResources(String instantTime)

Called after each commit of a compaction or clustering table service, to release any resources used.