org.apache.hudi.client

Class BaseHoodieWriteClient<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.client.BaseHoodieClient

org.apache.hudi.client.BaseHoodieWriteClient<T,I,K,O>

Type Parameters:

T - Sub type of HoodieRecordPayload

I - Type of inputs

K - Type of keys

O - Type of outputs

All Implemented Interfaces:

Serializable, AutoCloseable, RunsTableService

public abstract class BaseHoodieWriteClient<T extends HoodieRecordPayload,I,K,O>
extends BaseHoodieClient
implements RunsTableService

Abstract Write Client providing functionality for performing commit, index updates and rollback Reused for regular write operations like upsert/insert/bulk-insert.. as well as bootstrap

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected AsyncArchiveService	asyncArchiveService
protected AsyncCleanerService	asyncCleanerService
protected com.codahale.metrics.Timer.Context	clusteringTimer
protected com.codahale.metrics.Timer.Context	compactionTimer
protected Option<Pair<HoodieInstant,Map<String,String>>>	lastCompletedTxnAndMetadata
protected static String	LOOKUP_STR
protected HoodieMetrics	metrics
protected Set<String>	pendingInflightAndRequestedInstants
protected TransactionManager	txnManager
protected com.codahale.metrics.Timer.Context	writeTimer

Fields inherited from class org.apache.hudi.client.BaseHoodieClient

basePath, config, context, fs, hadoopConf, heartbeatClient

Constructor Summary

Constructors

Constructor and Description
BaseHoodieWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> timelineService, SupportsUpgradeDowngrade upgradeDowngradeHelper) Deprecated.
BaseHoodieWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, SupportsUpgradeDowngrade upgradeDowngradeHelper) Deprecated.

Method Summary

Modifier and Type	Method and Description
void	addColumn(String colName, org.apache.avro.Schema schema)
void	addColumn(String colName, org.apache.avro.Schema schema, String doc, String position, TableChange.ColumnPositionChange.ColumnPositionType positionType) add columns to table.
void	archive() Trigger archival for the table.
protected void	archive(HoodieTable table, boolean acquireLockForArchival) Trigger archival for the table.
protected void	autoArchiveOnCommit(HoodieTable table, boolean acquireLockForArchival)
protected void	autoCleanOnCommit()
void	bootstrap(Option<Map<String,String>> extraMetadata) Main API to run bootstrap to hudi.
abstract O	bulkInsert(I records, String instantTime) Loads the given HoodieRecords, as inserts into the table.
abstract O	bulkInsert(I records, String instantTime, Option<BulkInsertPartitioner> userDefinedBulkInsertPartitioner) Loads the given HoodieRecords, as inserts into the table.
abstract O	bulkInsertPreppedRecords(I preppedRecords, String instantTime, Option<BulkInsertPartitioner> bulkInsertPartitioner) Loads the given HoodieRecords, as inserts into the table.
HoodieCleanMetadata	clean()
HoodieCleanMetadata	clean(boolean skipLocking) Triggers clean for the table.
HoodieCleanMetadata	clean(String cleanInstantTime) Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used.
HoodieCleanMetadata	clean(String cleanInstantTime, boolean skipLocking) Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used.
HoodieCleanMetadata	clean(String cleanInstantTime, boolean scheduleInline, boolean skipLocking) Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used.
void	close() Releases any resources used by the client.
abstract HoodieWriteMetadata<O>	cluster(String clusteringInstant, boolean shouldComplete) Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
protected void	commit(HoodieTable table, String commitActionType, String instantTime, HoodieCommitMetadata metadata, List<HoodieWriteStat> stats)
boolean	commit(String instantTime, O writeStatuses) Commit changes performed at the given instantTime marker.
boolean	commit(String instantTime, O writeStatuses, Option<Map<String,String>> extraMetadata) Commit changes performed at the given instantTime marker.
abstract boolean	commit(String instantTime, O writeStatuses, Option<Map<String,String>> extraMetadata, String commitActionType, Map<String,List<String>> partitionToReplacedFileIds)
abstract void	commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata, Option<Map<String,String>> extraMetadata) Commit a compaction operation.
boolean	commitStats(String instantTime, List<HoodieWriteStat> stats, Option<Map<String,String>> extraMetadata, String commitActionType)
boolean	commitStats(String instantTime, List<HoodieWriteStat> stats, Option<Map<String,String>> extraMetadata, String commitActionType, Map<String,List<String>> partitionToReplaceFileIds)
HoodieWriteMetadata<O>	compact(String compactionInstantTime) Performs Compaction for the workload stored in instant-time.
protected abstract HoodieWriteMetadata<O>	compact(String compactionInstantTime, boolean shouldComplete) Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
protected abstract void	completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime) Commit Compaction and track metrics.
protected abstract HoodieIndex<?,?>	createIndex(HoodieWriteConfig writeConfig)
protected HoodieTable	createTable(HoodieWriteConfig config, org.apache.hadoop.conf.Configuration hadoopConf)
protected abstract HoodieTable	createTable(HoodieWriteConfig config, org.apache.hadoop.conf.Configuration hadoopConf, boolean refreshTimeline)
abstract O	delete(K keys, String instantTime) Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.
void	deleteColumns(String... colNames) delete columns to table.
void	deleteSavepoint(String savepointTime) Delete a savepoint that was created.
protected abstract HoodieTable	doInitTable(HoodieTableMetaClient metaClient, Option<String> instantTime, boolean initialMetadataTableIfNecessary) Instantiates engine-specific instance of HoodieTable as well as performs necessary bootstrapping operations (for ex, validating whether Metadata Table has to be bootstrapped) NOTE: THIS OPERATION IS EXECUTED UNDER LOCK, THEREFORE SHOULD AVOID ANY OPERATIONS NOT REQUIRING EXTERNAL SYNCHRONIZATION
void	dropIndex(List<MetadataPartitionType> partitionTypes) Drops the index and removes the metadata partitions.
abstract I	filterExists(I hoodieRecords) Filter out HoodieRecords that already exists in the output folder.
protected void	finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) Finalize Write operation.
HoodieIndex<?,?>	getIndex()
protected List<String>	getInstantsToRollback(HoodieTableMetaClient metaClient, HoodieFailedWritesCleaningPolicy cleaningPolicy, Option<String> curInstantTime)
HoodieMetrics	getMetrics()
WriteOperationType	getOperationType()
protected Option<HoodiePendingRollbackInfo>	getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback)
protected Option<HoodiePendingRollbackInfo>	getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback, boolean ignoreCompactionAndClusteringInstants)
protected Map<String,Option<HoodiePendingRollbackInfo>>	getPendingRollbackInfos(HoodieTableMetaClient metaClient)
protected Map<String,Option<HoodiePendingRollbackInfo>>	getPendingRollbackInfos(HoodieTableMetaClient metaClient, boolean ignoreCompactionAndClusteringInstants) Fetch map of pending commits to be rolled-back to HoodiePendingRollbackInfo.
Option<HoodieIndexCommitMetadata>	index(String indexInstantTime) Runs INDEX action to build out the metadata partitions as planned for the given instant time.
protected HoodieTable	initTable(WriteOperationType operationType, Option<String> instantTime)
protected HoodieTable	initTable(WriteOperationType operationType, Option<String> instantTime, boolean initialMetadataTableIfNecessary) Instantiates and initializes instance of HoodieTable, performing crucial bootstrapping operations such as: NOTE: This method is engine-agnostic and SHOULD NOT be overloaded, please check on doInitTable(HoodieTableMetaClient, Option, boolean) instead Checking whether upgrade/downgrade is required Bootstrapping Metadata Table (if required) Initializing metrics contexts
protected Option<String>	inlineClustering(Option<Map<String,String>> extraMetadata) Executes a clustering plan on a table, serially before or after an insert/upsert action.
protected Option<String>	inlineCompaction(Option<Map<String,String>> extraMetadata) Performs a compaction operation on a table, serially before or after an insert/upsert action.
protected Option<String>	inlineScheduleClustering(Option<Map<String,String>> extraMetadata) Schedules clustering inline.
protected Option<String>	inlineScheduleCompaction(Option<Map<String,String>> extraMetadata) Schedules compaction inline.
abstract O	insert(I records, String instantTime) Inserts the given HoodieRecords, into the table.
abstract O	insertPreppedRecords(I preppedRecords, String instantTime) Inserts the given prepared records into the Hoodie table, at the supplied instantTime.
protected void	postCommit(HoodieTable table, HoodieCommitMetadata metadata, String instantTime, Option<Map<String,String>> extraMetadata, boolean acquireLockForArchival) Post Commit Hook.
protected abstract O	postWrite(HoodieWriteMetadata<O> result, String instantTime, HoodieTable hoodieTable) Common method containing steps to be performed after write (upsert/insert/..) operations including auto-commit.
protected void	preCommit(HoodieInstant inflightInstant, HoodieCommitMetadata metadata) Any pre-commit actions like conflict resolution goes here.
protected void	preWrite(String instantTime, WriteOperationType writeOperationType, HoodieTableMetaClient metaClient) Common method containing steps to be performed before write (upsert/insert/...
protected void	releaseResources() Called after each write, to release any resources used.
void	renameColumn(String colName, String newName) rename col name for hudi table.
void	reOrderColPosition(String colName, String referColName, TableChange.ColumnPositionChange.ColumnPositionType orderType) reorder the position of col.
HoodieRestoreMetadata	restoreToInstant(String instantTime, boolean initialMetadataTableIfNecessary) NOTE : This action requires all writers (ingest and compact) to a table to be stopped before proceeding.
void	restoreToSavepoint(String savepointTime) Restore the data to the savepoint.
boolean	rollback(String commitInstantTime) Deprecated.
boolean	rollback(String commitInstantTime, Option<HoodiePendingRollbackInfo> pendingRollbackInfo, boolean skipLocking) Deprecated.
protected void	rollbackFailedBootstrap() Main API to rollback failed bootstrap.
protected Boolean	rollbackFailedWrites() Rollback all failed writes.
protected Boolean	rollbackFailedWrites(boolean skipLocking) Rollback all failed writes.
protected void	rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback, boolean skipLocking)
protected void	rollbackInflightClustering(HoodieInstant inflightInstant, HoodieTable table)
protected void	runAnyPendingClustering(HoodieTable table)
void	runAnyPendingCompactions() Run any pending compactions.
protected void	runAnyPendingCompactions(HoodieTable table)
protected void	runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String,String>> extraMetadata)
void	savepoint(String user, String comment) Create a savepoint based on the latest commit action on the timeline.
void	savepoint(String instantTime, String user, String comment) Savepoint a specific commit instant time.
protected Option<String>	scheduleCleaning(Option<Map<String,String>> extraMetadata) Schedules a new cleaning instant.
protected boolean	scheduleCleaningAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new cleaning instant with passed-in instant time.
Option<String>	scheduleClustering(Option<Map<String,String>> extraMetadata) Schedules a new clustering instant.
boolean	scheduleClusteringAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new clustering instant with passed-in instant time.
Option<String>	scheduleCompaction(Option<Map<String,String>> extraMetadata) Schedules a new compaction instant.
boolean	scheduleCompactionAtInstant(String instantTime, Option<Map<String,String>> extraMetadata) Schedules a new compaction instant with passed-in instant time.
Option<String>	scheduleIndexing(List<MetadataPartitionType> partitionTypes) Schedules INDEX action.
Option<String>	scheduleTableService(Option<Map<String,String>> extraMetadata, TableServiceType tableServiceType) Schedule table services such as clustering, compaction & cleaning.
Option<String>	scheduleTableService(String instantTime, Option<Map<String,String>> extraMetadata, TableServiceType tableServiceType) Schedule table services such as clustering, compaction & cleaning.
void	setOperationType(WriteOperationType operationType)
protected void	setWriteSchemaForDeletes(HoodieTableMetaClient metaClient) Sets write schema from last instant since deletes may not have schema set in the config.
String	startCommit() Provides a new commit time for a write operation (insert/update/delete).
String	startCommit(String actionType, HoodieTableMetaClient metaClient) Provides a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) with specified action.
void	startCommitWithTime(String instantTime) Provides a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) without specified action.
void	startCommitWithTime(String instantTime, String actionType) Completes a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) with specified action.
void	updateColumnComment(String colName, String doc) update col comment for hudi table.
void	updateColumnNullability(String colName, boolean nullable) update col nullable attribute for hudi table.
void	updateColumnType(String colName, Type newType) update col Type for hudi table.
abstract O	upsert(I records, String instantTime) Upsert a batch of new records into Hoodie table at the supplied instantTime.
abstract O	upsertPreppedRecords(I preppedRecords, String instantTime) Upserts the given prepared records into the Hoodie table, at the supplied instantTime.
protected void	writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) Write the HoodieCommitMetadata to metadata table if available.

Methods inherited from class org.apache.hudi.client.BaseHoodieClient

createMetaClient, getConfig, getEngineContext, getHeartbeatClient, getTimelineServer, initWrapperFSMetrics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.client.RunsTableService

tableServicesEnabled

Field Detail

LOOKUP_STR

protected static final String LOOKUP_STR

See Also:

Constant Field Values

metrics

protected final transient HoodieMetrics metrics

writeTimer

protected transient com.codahale.metrics.Timer.Context writeTimer

compactionTimer

protected transient com.codahale.metrics.Timer.Context compactionTimer

clusteringTimer

protected transient com.codahale.metrics.Timer.Context clusteringTimer

asyncCleanerService

protected transient AsyncCleanerService asyncCleanerService

asyncArchiveService

protected transient AsyncArchiveService asyncArchiveService

txnManager

protected final TransactionManager txnManager

lastCompletedTxnAndMetadata

protected Option<Pair<HoodieInstant,Map<String,String>>> lastCompletedTxnAndMetadata

pendingInflightAndRequestedInstants

protected Set<String> pendingInflightAndRequestedInstants

Constructor Detail

BaseHoodieWriteClient

@Deprecated
public BaseHoodieWriteClient(HoodieEngineContext context,
                                         HoodieWriteConfig writeConfig,
                                         SupportsUpgradeDowngrade upgradeDowngradeHelper)

Deprecated.

Create a write client, with new hudi index.

Parameters:

context - HoodieEngineContext

writeConfig - instance of HoodieWriteConfig

upgradeDowngradeHelper - engine-specific instance of SupportsUpgradeDowngrade

BaseHoodieWriteClient

@Deprecated
public BaseHoodieWriteClient(HoodieEngineContext context,
                                         HoodieWriteConfig writeConfig,
                                         Option<EmbeddedTimelineService> timelineService,
                                         SupportsUpgradeDowngrade upgradeDowngradeHelper)

Deprecated.

Create a write client, allows to specify all parameters.

Parameters:

context - HoodieEngineContext

writeConfig - instance of HoodieWriteConfig

timelineService - Timeline Service that runs as part of write client.

Method Detail

createIndex

protected abstract HoodieIndex<?,?> createIndex(HoodieWriteConfig writeConfig)

setOperationType

public void setOperationType(WriteOperationType operationType)

getOperationType

public WriteOperationType getOperationType()

commit

public boolean commit(String instantTime,
                      O writeStatuses)

Commit changes performed at the given instantTime marker.

commit

public boolean commit(String instantTime,
                      O writeStatuses,
                      Option<Map<String,String>> extraMetadata)

Commit changes performed at the given instantTime marker.

commit

public abstract boolean commit(String instantTime,
                               O writeStatuses,
                               Option<Map<String,String>> extraMetadata,
                               String commitActionType,
                               Map<String,List<String>> partitionToReplacedFileIds)

commitStats

public boolean commitStats(String instantTime,
                           List<HoodieWriteStat> stats,
                           Option<Map<String,String>> extraMetadata,
                           String commitActionType)

commitStats

public boolean commitStats(String instantTime,
                           List<HoodieWriteStat> stats,
                           Option<Map<String,String>> extraMetadata,
                           String commitActionType,
                           Map<String,List<String>> partitionToReplaceFileIds)

commit

protected void commit(HoodieTable table,
                      String commitActionType,
                      String instantTime,
                      HoodieCommitMetadata metadata,
                      List<HoodieWriteStat> stats)
               throws IOException

Throws:

IOException

createTable

protected HoodieTable createTable(HoodieWriteConfig config,
                                  org.apache.hadoop.conf.Configuration hadoopConf)

createTable

protected abstract HoodieTable createTable(HoodieWriteConfig config,
                                           org.apache.hadoop.conf.Configuration hadoopConf,
                                           boolean refreshTimeline)

preCommit

protected void preCommit(HoodieInstant inflightInstant,
                         HoodieCommitMetadata metadata)

Any pre-commit actions like conflict resolution goes here.

Parameters:

inflightInstant - instant of inflight operation.

metadata - commit metadata for which pre commit is being invoked.

writeTableMetadata

protected void writeTableMetadata(HoodieTable table,
                                  String instantTime,
                                  String actionType,
                                  HoodieCommitMetadata metadata)

Write the HoodieCommitMetadata to metadata table if available.

Parameters:

table - HoodieTable of interest.

instantTime - instant time of the commit.

actionType - action type of the commit.

metadata - instance of HoodieCommitMetadata.

filterExists

public abstract I filterExists(I hoodieRecords)

Filter out HoodieRecords that already exists in the output folder. This is useful in deduplication.

Parameters:

hoodieRecords - Input Hoodie records.

Returns:

A subset of hoodieRecords, with existing records filtered out.

bootstrap

public void bootstrap(Option<Map<String,String>> extraMetadata)

Main API to run bootstrap to hudi.

rollbackFailedBootstrap

protected void rollbackFailedBootstrap()

Main API to rollback failed bootstrap.

upsert

public abstract O upsert(I records,
                         String instantTime)

Upsert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

records - hoodieRecords to upsert

instantTime - Instant time of the commit

Returns:

WriteStatus to inspect errors and counts

upsertPreppedRecords

public abstract O upsertPreppedRecords(I preppedRecords,
                                       String instantTime)

Upserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

preppedRecords - Prepared HoodieRecords to upsert

instantTime - Instant time of the commit

Returns:

Collection of WriteStatus to inspect errors and counts

insert

public abstract O insert(I records,
                         String instantTime)

Inserts the given HoodieRecords, into the table. This API is intended to be used for normal writes.

This implementation skips the index check and is able to leverage benefits such as small file handling/blocking alignment, as with upsert(), by profiling the workload

Parameters:

records - HoodieRecords to insert

instantTime - Instant time of the commit

Returns:

Collection of WriteStatus to inspect errors and counts

insertPreppedRecords

public abstract O insertPreppedRecords(I preppedRecords,
                                       String instantTime)

Inserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation skips the index check, skips de-duping and is able to leverage benefits such as small file handling/blocking alignment, as with insert(), by profiling the workload. The prepared HoodieRecords should be de-duped if needed.

Parameters:

preppedRecords - HoodieRecords to insert

instantTime - Instant time of the commit

Returns:

Collection of WriteStatus to inspect errors and counts

bulkInsert

public abstract O bulkInsert(I records,
                             String instantTime)

Loads the given HoodieRecords, as inserts into the table. This is suitable for doing big bulk loads into a Hoodie table for the very first time (e.g: converting an existing table to Hoodie).

This implementation uses sortBy (which does range partitioning based on reservoir sampling) and attempts to control the numbers of files with less memory compared to the BaseHoodieWriteClient#insert(I, String)

Parameters:

records - HoodieRecords to insert

instantTime - Instant time of the commit

Returns:

Collection of WriteStatus to inspect errors and counts

bulkInsert

public abstract O bulkInsert(I records,
                             String instantTime,
                             Option<BulkInsertPartitioner> userDefinedBulkInsertPartitioner)

Loads the given HoodieRecords, as inserts into the table. This is suitable for doing big bulk loads into a Hoodie table for the very first time (e.g: converting an existing table to Hoodie).

This implementation uses sortBy (which does range partitioning based on reservoir sampling) and attempts to control the numbers of files with less memory compared to the BaseHoodieWriteClient#insert(I, String). Optionally it allows users to specify their own partitioner. If specified then it will be used for repartitioning records. See BulkInsertPartitioner.

Parameters:

records - HoodieRecords to insert

instantTime - Instant time of the commit

userDefinedBulkInsertPartitioner - If specified then it will be used to partition input records before they are inserted into hoodie.

Returns:

Collection of WriteStatus to inspect errors and counts

bulkInsertPreppedRecords

public abstract O bulkInsertPreppedRecords(I preppedRecords,
                                           String instantTime,
                                           Option<BulkInsertPartitioner> bulkInsertPartitioner)

Loads the given HoodieRecords, as inserts into the table. This is suitable for doing big bulk loads into a Hoodie table for the very first time (e.g: converting an existing table to Hoodie). The input records should contain no duplicates if needed.

This implementation uses sortBy (which does range partitioning based on reservoir sampling) and attempts to control the numbers of files with less memory compared to the BaseHoodieWriteClient#insert(I, String). Optionally it allows users to specify their own partitioner. If specified then it will be used for repartitioning records. See BulkInsertPartitioner.

Parameters:

preppedRecords - HoodieRecords to insert

instantTime - Instant time of the commit

bulkInsertPartitioner - If specified then it will be used to partition input records before they are inserted into hoodie.

Returns:

Collection of WriteStatus to inspect errors and counts

delete

public abstract O delete(K keys,
                         String instantTime)

Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.

Parameters:

keys - List of HoodieKeys to be deleted

instantTime - Commit time handle

Returns:

Collection of WriteStatus to inspect errors and counts

preWrite

protected void preWrite(String instantTime,
                        WriteOperationType writeOperationType,
                        HoodieTableMetaClient metaClient)

Common method containing steps to be performed before write (upsert/insert/...

Parameters:

instantTime -

writeOperationType -

metaClient -

postWrite

protected abstract O postWrite(HoodieWriteMetadata<O> result,
                               String instantTime,
                               HoodieTable hoodieTable)

Common method containing steps to be performed after write (upsert/insert/..) operations including auto-commit.

Parameters:

result - Commit Action Result

instantTime - Instant Time

hoodieTable - Hoodie Table

Returns:

Write Status

postCommit

protected void postCommit(HoodieTable table,
                          HoodieCommitMetadata metadata,
                          String instantTime,
                          Option<Map<String,String>> extraMetadata,
                          boolean acquireLockForArchival)

Post Commit Hook. Derived classes use this method to perform post-commit processing

Parameters:

table - table to commit on

metadata - Commit Metadata corresponding to committed instant

instantTime - Instant Time

extraMetadata - Additional Metadata passed by user

acquireLockForArchival - true if lock has to be acquired for archival. false otherwise.

runTableServicesInline

protected void runTableServicesInline(HoodieTable table,
                                      HoodieCommitMetadata metadata,
                                      Option<Map<String,String>> extraMetadata)

runAnyPendingCompactions

protected void runAnyPendingCompactions(HoodieTable table)

runAnyPendingClustering

protected void runAnyPendingClustering(HoodieTable table)

autoCleanOnCommit

protected void autoCleanOnCommit()

autoArchiveOnCommit

protected void autoArchiveOnCommit(HoodieTable table,
                                   boolean acquireLockForArchival)

runAnyPendingCompactions

public void runAnyPendingCompactions()

Run any pending compactions.

savepoint

public void savepoint(String user,
                      String comment)

Create a savepoint based on the latest commit action on the timeline.

Parameters:

user - - User creating the savepoint

comment - - Comment for the savepoint

savepoint

public void savepoint(String instantTime,
                      String user,
                      String comment)

Savepoint a specific commit instant time. Latest version of data files as of the passed in instantTime will be referenced in the savepoint and will never be cleaned. The savepointed commit will never be rolledback or archived.

This gives an option to rollback the state to the savepoint anytime. Savepoint needs to be manually created and deleted.

Savepoint should be on a commit that could not have been cleaned.

Parameters:

instantTime - - commit that should be savepointed

user - - User creating the savepoint

comment - - Comment for the savepoint

deleteSavepoint

public void deleteSavepoint(String savepointTime)

Delete a savepoint that was created. Once the savepoint is deleted, the commit can be rolledback and cleaner may clean up data files.

Parameters:

savepointTime - - delete the savepoint

restoreToSavepoint

public void restoreToSavepoint(String savepointTime)

Restore the data to the savepoint. WARNING: This rolls back recent commits and deleted data files and also pending compactions after savepoint time. Queries accessing the files will mostly fail. This is expected to be a manual operation and no concurrent write or compaction is expected to be running

Parameters:

savepointTime - - savepoint time to rollback to

rollback

@Deprecated
public boolean rollback(String commitInstantTime)
                             throws HoodieRollbackException

Deprecated.

Throws:

HoodieRollbackException

rollback

@Deprecated
public boolean rollback(String commitInstantTime,
                                    Option<HoodiePendingRollbackInfo> pendingRollbackInfo,
                                    boolean skipLocking)
                             throws HoodieRollbackException

Deprecated.

Parameters:

commitInstantTime - Instant time of the commit

pendingRollbackInfo - pending rollback instant and plan if rollback failed from previous attempt.

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Throws:

HoodieRollbackException - if rollback cannot be performed successfully

restoreToInstant

public HoodieRestoreMetadata restoreToInstant(String instantTime,
                                              boolean initialMetadataTableIfNecessary)
                                       throws HoodieRestoreException

NOTE : This action requires all writers (ingest and compact) to a table to be stopped before proceeding. Revert the (inflight/committed) record changes for all commits after the provided instant time.

Parameters:

instantTime - Instant time to which restoration is requested

Throws:

HoodieRestoreException

clean

public HoodieCleanMetadata clean(String cleanInstantTime)
                          throws HoodieIOException

Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used. (typically files that no longer can be used by a running query can be cleaned)

Throws:

HoodieIOException

clean

public HoodieCleanMetadata clean(String cleanInstantTime,
                                 boolean skipLocking)
                          throws HoodieIOException

Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used. (typically files that no longer can be used by a running query can be cleaned)

Parameters:

cleanInstantTime - instant time for clean.

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Returns:

instance of HoodieCleanMetadata.

Throws:

HoodieIOException

clean

public HoodieCleanMetadata clean(String cleanInstantTime,
                                 boolean scheduleInline,
                                 boolean skipLocking)
                          throws HoodieIOException

Clean up any stale/old files/data lying around (either on file storage or index storage) based on the configurations and CleaningPolicy used. (typically files that no longer can be used by a running query can be cleaned). This API provides the flexibility to schedule clean instant asynchronously via scheduleTableService(String, Option, TableServiceType) and disable inline scheduling of clean.

Parameters:

cleanInstantTime - instant time for clean.

scheduleInline - true if needs to be scheduled inline. false otherwise.

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Throws:

HoodieIOException

clean

public HoodieCleanMetadata clean()

clean

public HoodieCleanMetadata clean(boolean skipLocking)

Triggers clean for the table. This refers to Clean up any stale/old files/data lying around (either on file storage or index storage) based on the * configurations and CleaningPolicy used.

Parameters:

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

Returns:

instance of HoodieCleanMetadata.

archive

protected void archive(HoodieTable table,
                       boolean acquireLockForArchival)

Trigger archival for the table. This ensures that the number of commits do not explode and keep increasing unbounded over time.

Parameters:

table - table to commit on.

acquireLockForArchival - true if lock has to be acquired for archival. false otherwise.

archive

public void archive()

Trigger archival for the table. This ensures that the number of commits do not explode and keep increasing unbounded over time.

startCommit

public String startCommit()

Provides a new commit time for a write operation (insert/update/delete).

startCommit

public String startCommit(String actionType,
                          HoodieTableMetaClient metaClient)

Provides a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) with specified action.

startCommitWithTime

public void startCommitWithTime(String instantTime)

Provides a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) without specified action.

Parameters:

instantTime - Instant time to be generated

startCommitWithTime

public void startCommitWithTime(String instantTime,
                                String actionType)

Completes a new commit time for a write operation (insert/update/delete/insert_overwrite/insert_overwrite_table) with specified action.

scheduleCompaction

public Option<String> scheduleCompaction(Option<Map<String,String>> extraMetadata)
                                  throws HoodieIOException

Schedules a new compaction instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleCompactionAtInstant

public boolean scheduleCompactionAtInstant(String instantTime,
                                           Option<Map<String,String>> extraMetadata)
                                    throws HoodieIOException

Schedules a new compaction instant with passed-in instant time.

Parameters:

instantTime - Compaction Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleIndexing

public Option<String> scheduleIndexing(List<MetadataPartitionType> partitionTypes)

Schedules INDEX action.

Parameters:

partitionTypes - - list of MetadataPartitionType which needs to be indexed

Returns:

instant time for the requested INDEX action

index

public Option<HoodieIndexCommitMetadata> index(String indexInstantTime)

Runs INDEX action to build out the metadata partitions as planned for the given instant time.

Parameters:

indexInstantTime - - instant time for the requested INDEX action

Returns:

Option after successful indexing.

dropIndex

public void dropIndex(List<MetadataPartitionType> partitionTypes)

Drops the index and removes the metadata partitions.

Parameters:

partitionTypes - - list of MetadataPartitionType which needs to be indexed

compact

public HoodieWriteMetadata<O> compact(String compactionInstantTime)

Performs Compaction for the workload stored in instant-time.

Parameters:

compactionInstantTime - Compaction Instant Time

Returns:

Collection of WriteStatus to inspect errors and counts

commitCompaction

public abstract void commitCompaction(String compactionInstantTime,
                                      HoodieCommitMetadata metadata,
                                      Option<Map<String,String>> extraMetadata)

Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.

Parameters:

compactionInstantTime - Compaction Instant Time

metadata - All the metadata that gets stored along with a commit

extraMetadata - Extra Metadata to be stored

completeCompaction

protected abstract void completeCompaction(HoodieCommitMetadata metadata,
                                           HoodieTable table,
                                           String compactionCommitTime)

Commit Compaction and track metrics.

getPendingRollbackInfo

protected Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient,
                                                                   String commitToRollback)

getPendingRollbackInfo

protected Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient,
                                                                   String commitToRollback,
                                                                   boolean ignoreCompactionAndClusteringInstants)

getPendingRollbackInfos

protected Map<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient)

getPendingRollbackInfos

protected Map<String,Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient,
                                                                                boolean ignoreCompactionAndClusteringInstants)

Fetch map of pending commits to be rolled-back to HoodiePendingRollbackInfo.

Parameters:

metaClient - instance of HoodieTableMetaClient to use.

Returns:

map of pending commits to be rolled-back instants to Rollback Instant and Rollback plan Pair.

rollbackFailedWrites

protected Boolean rollbackFailedWrites()

Rollback all failed writes.

rollbackFailedWrites

protected Boolean rollbackFailedWrites(boolean skipLocking)

Rollback all failed writes.

Parameters:

skipLocking - if this is triggered by another parent transaction, locking can be skipped.

rollbackFailedWrites

protected void rollbackFailedWrites(Map<String,Option<HoodiePendingRollbackInfo>> instantsToRollback,
                                    boolean skipLocking)

getInstantsToRollback

protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient,
                                             HoodieFailedWritesCleaningPolicy cleaningPolicy,
                                             Option<String> curInstantTime)

compact

protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime,
                                                  boolean shouldComplete)

Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.

Parameters:

compactionInstantTime - Compaction Instant Time

Returns:

Collection of Write Status

inlineCompaction

protected Option<String> inlineCompaction(Option<Map<String,String>> extraMetadata)

Performs a compaction operation on a table, serially before or after an insert/upsert action. Scheduling and execution is done inline.

inlineScheduleCompaction

protected Option<String> inlineScheduleCompaction(Option<Map<String,String>> extraMetadata)

Schedules compaction inline.

Parameters:

extraMetadata - extrametada to be used.

Returns:

compaction instant if scheduled.

scheduleClustering

public Option<String> scheduleClustering(Option<Map<String,String>> extraMetadata)
                                  throws HoodieIOException

Schedules a new clustering instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleClusteringAtInstant

public boolean scheduleClusteringAtInstant(String instantTime,
                                           Option<Map<String,String>> extraMetadata)
                                    throws HoodieIOException

Schedules a new clustering instant with passed-in instant time.

Parameters:

instantTime - clustering Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleCleaning

protected Option<String> scheduleCleaning(Option<Map<String,String>> extraMetadata)
                                   throws HoodieIOException

Schedules a new cleaning instant.

Parameters:

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

scheduleCleaningAtInstant

protected boolean scheduleCleaningAtInstant(String instantTime,
                                            Option<Map<String,String>> extraMetadata)
                                     throws HoodieIOException

Schedules a new cleaning instant with passed-in instant time.

Parameters:

instantTime - cleaning Instant Time

extraMetadata - Extra Metadata to be stored

Throws:

HoodieIOException

cluster

public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant,
                                               boolean shouldComplete)

Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.

Parameters:

clusteringInstant - Clustering Instant Time

Returns:

Collection of Write Status

scheduleTableService

public Option<String> scheduleTableService(Option<Map<String,String>> extraMetadata,
                                           TableServiceType tableServiceType)

Schedule table services such as clustering, compaction & cleaning.

Parameters:

extraMetadata - Metadata to pass onto the scheduled service instant

tableServiceType - Type of table service to schedule

Returns:

scheduleTableService

public Option<String> scheduleTableService(String instantTime,
                                           Option<Map<String,String>> extraMetadata,
                                           TableServiceType tableServiceType)

Schedule table services such as clustering, compaction & cleaning.

Parameters:

extraMetadata - Metadata to pass onto the scheduled service instant

tableServiceType - Type of table service to schedule

Returns:

inlineClustering

protected Option<String> inlineClustering(Option<Map<String,String>> extraMetadata)

Executes a clustering plan on a table, serially before or after an insert/upsert action. Schedules and executes clustering inline.

inlineScheduleClustering

protected Option<String> inlineScheduleClustering(Option<Map<String,String>> extraMetadata)

Schedules clustering inline.

Parameters:

extraMetadata - extrametadata to use.

Returns:

clustering instant if scheduled.

rollbackInflightClustering

protected void rollbackInflightClustering(HoodieInstant inflightInstant,
                                          HoodieTable table)

finalizeWrite

protected void finalizeWrite(HoodieTable table,
                             String instantTime,
                             List<HoodieWriteStat> stats)

Finalize Write operation.

Parameters:

table - HoodieTable

instantTime - Instant Time

stats - Hoodie Write Stat

getMetrics

public HoodieMetrics getMetrics()

getIndex

public HoodieIndex<?,?> getIndex()

doInitTable

protected abstract HoodieTable doInitTable(HoodieTableMetaClient metaClient,
                                           Option<String> instantTime,
                                           boolean initialMetadataTableIfNecessary)

Instantiates engine-specific instance of HoodieTable as well as performs necessary bootstrapping operations (for ex, validating whether Metadata Table has to be bootstrapped) NOTE: THIS OPERATION IS EXECUTED UNDER LOCK, THEREFORE SHOULD AVOID ANY OPERATIONS NOT REQUIRING EXTERNAL SYNCHRONIZATION

Parameters:

metaClient - instance of HoodieTableMetaClient

instantTime - current inflight instant time

Returns:

instantiated HoodieTable

initTable

protected final HoodieTable initTable(WriteOperationType operationType,
                                      Option<String> instantTime,
                                      boolean initialMetadataTableIfNecessary)

Instantiates and initializes instance of HoodieTable, performing crucial bootstrapping operations such as: NOTE: This method is engine-agnostic and SHOULD NOT be overloaded, please check on doInitTable(HoodieTableMetaClient, Option, boolean) instead

• Checking whether upgrade/downgrade is required
• Bootstrapping Metadata Table (if required)
• Initializing metrics contexts

initTable

protected final HoodieTable initTable(WriteOperationType operationType,
                                      Option<String> instantTime)

setWriteSchemaForDeletes

protected void setWriteSchemaForDeletes(HoodieTableMetaClient metaClient)

Sets write schema from last instant since deletes may not have schema set in the config.

releaseResources

protected void releaseResources()

Called after each write, to release any resources used.

close

public void close()

Description copied from class: BaseHoodieClient

Releases any resources used by the client.

Specified by:

close in interface AutoCloseable

Overrides:

close in class BaseHoodieClient

addColumn

public void addColumn(String colName,
                      org.apache.avro.Schema schema,
                      String doc,
                      String position,
                      TableChange.ColumnPositionChange.ColumnPositionType positionType)

add columns to table.

Parameters:

colName - col name to be added. if we want to add col to a nested filed, the fullName should be specify

schema - col type to be added.

doc - col doc to be added.

position - col position to be added

positionType - col position change type. now support three change types: first/after/before

addColumn

public void addColumn(String colName,
                      org.apache.avro.Schema schema)

deleteColumns

public void deleteColumns(String... colNames)

delete columns to table.

Parameters:

colNames - col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify

renameColumn

public void renameColumn(String colName,
                         String newName)

rename col name for hudi table.

Parameters:

colName - col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify

newName - new name for current col. no need to specify fullName.

updateColumnNullability

public void updateColumnNullability(String colName,
                                    boolean nullable)

update col nullable attribute for hudi table.

Parameters:

colName - col name to be changed. if we want to change col from a nested filed, the fullName should be specify

nullable - .

updateColumnType

public void updateColumnType(String colName,
                             Type newType)

update col Type for hudi table. only support update primitive type to primitive type. cannot update nest type to nest type or primitive type eg: RecordType -> MapType, MapType -> LongType.

Parameters:

colName - col name to be changed. if we want to change col from a nested filed, the fullName should be specify

newType - .

updateColumnComment

public void updateColumnComment(String colName,
                                String doc)

update col comment for hudi table.

Parameters:

colName - col name to be changed. if we want to change col from a nested filed, the fullName should be specify

doc - .

reOrderColPosition

public void reOrderColPosition(String colName,
                               String referColName,
                               TableChange.ColumnPositionChange.ColumnPositionType orderType)

reorder the position of col.

Parameters:

colName - column which need to be reordered. if we want to change col from a nested filed, the fullName should be specify.

referColName - reference position.

orderType - col position change type. now support three change types: first/after/before