org.apache.hudi.table

Class HoodieTable<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.table.HoodieTable<T,I,K,O>

Type Parameters:

T - Sub type of HoodieRecordPayload

I - Type of inputs

K - Type of keys

O - Type of outputs

All Implemented Interfaces:

Serializable

public abstract class HoodieTable<T extends HoodieRecordPayload,I,K,O>
extends Object
implements Serializable

Abstract implementation of a HoodieTable.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieWriteConfig	config
protected HoodieEngineContext	context
protected HoodieIndex<T,?,?,?>	index
protected HoodieTableMetaClient	metaClient
protected TaskContextSupplier	taskContextSupplier

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HoodieTable(HoodieWriteConfig config, HoodieEngineContext context, HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
abstract HoodieBootstrapWriteMetadata<O>	bootstrap(HoodieEngineContext context, Option<Map<String,String>> extraMetadata) Perform metadata/full bootstrap of a Hudi table.
abstract HoodieWriteMetadata<O>	bulkInsert(HoodieEngineContext context, String instantTime, I records, Option<BulkInsertPartitioner<I>> bulkInsertPartitioner) Bulk Insert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	bulkInsertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords, Option<BulkInsertPartitioner<I>> bulkInsertPartitioner) Bulk Insert the given prepared records into the Hoodie table, at the supplied instantTime.
abstract HoodieCleanMetadata	clean(HoodieEngineContext context, String cleanInstantTime, boolean skipLocking) Executes a new clean action.
abstract HoodieWriteMetadata<O>	cluster(HoodieEngineContext context, String clusteringInstantTime) Execute Clustering on the table.
abstract HoodieWriteMetadata<O>	compact(HoodieEngineContext context, String compactionInstantTime) Run Compaction on the table.
abstract HoodieWriteMetadata<O>	delete(HoodieEngineContext context, String instantTime, K keys) Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.
abstract HoodieWriteMetadata	deletePartitions(HoodieEngineContext context, String instantTime, List<String> partitions) Deletes all data of partitions.
void	finalizeWrite(HoodieEngineContext context, String instantTs, List<HoodieWriteStat> stats) Finalize the written data onto storage.
HoodieActiveTimeline	getActiveTimeline()
String	getBaseFileExtension()
HoodieFileFormat	getBaseFileFormat()
TableFileSystemView.BaseFileOnlyView	getBaseFileOnlyView() Get the base file only view of the file system for this table.
HoodieTimeline	getCleanTimeline() Get clean timeline.
HoodieTimeline	getCompletedCleanTimeline() Get only the completed (no-inflights) clean timeline.
HoodieTimeline	getCompletedCommitsTimeline() Get only the completed (no-inflights) commit + deltacommit timeline.
HoodieTimeline	getCompletedCommitTimeline() Get only the completed (no-inflights) commit timeline.
HoodieTimeline	getCompletedSavepointTimeline() Get only the completed (no-inflights) savepoint timeline.
HoodieWriteConfig	getConfig()
static ConsistencyGuard	getConsistencyGuard(org.apache.hadoop.fs.FileSystem fs, ConsistencyGuardConfig consistencyGuardConfig) Instantiate ConsistencyGuard based on configs.
HoodieEngineContext	getContext()
TableFileSystemView	getFileSystemView() Get the view of the file system for this table.
org.apache.hadoop.conf.Configuration	getHadoopConf()
SyncableFileSystemView	getHoodieView() Get complete view of the file system for this table with ability to force sync.
HoodieIndex<T,?,?,?>	getIndex() Return the index.
protected abstract HoodieIndex<T,?,?,?>	getIndex(HoodieWriteConfig config, HoodieEngineContext context)
protected Set<String>	getInvalidDataPaths(WriteMarkers markers) Returns the possible invalid data file name with given marker files.
HoodieLogBlock.HoodieLogBlockType	getLogDataBlockFormat()
HoodieFileFormat	getLogFileFormat()
HoodieTableMetaClient	getMetaClient()
Option<HoodieTableMetadataWriter>	getMetadataWriter(String triggeringInstantTimestamp) Get Table metadata writer.
<T extends org.apache.avro.specific.SpecificRecordBase> Option<HoodieTableMetadataWriter>	getMetadataWriter(String triggeringInstantTimestamp, Option<T> actionMetadata) Get Table metadata writer.
HoodieTimeline	getPendingCommitTimeline() Get only the inflights (no-completed) commit timeline.
HoodieTimeline	getRollbackTimeline() Get rollback timeline.
List<String>	getSavepoints() Get the list of savepoints in this table.
TableFileSystemView.SliceView	getSliceView() Get the full view of the file system for this table.
TaskContextSupplier	getTaskContextSupplier()
abstract HoodieWriteMetadata<O>	insert(HoodieEngineContext context, String instantTime, I records) Insert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	insertOverwrite(HoodieEngineContext context, String instantTime, I records) Replaces all the existing records and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.
abstract HoodieWriteMetadata<O>	insertOverwriteTable(HoodieEngineContext context, String instantTime, I records) Delete all the existing records of the Hoodie table and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.
abstract HoodieWriteMetadata<O>	insertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords) Inserts the given prepared records into the Hoodie table, at the supplied instantTime.
abstract boolean	isTableServiceAction(String actionType) Check if action type is a table service.
protected void	reconcileAgainstMarkers(HoodieEngineContext context, String instantTs, List<HoodieWriteStat> stats, boolean consistencyCheckEnabled) Reconciles WriteStats and marker files to detect and safely delete duplicate data files created because of Spark retries.
boolean	requireSortedRecords()
abstract HoodieRestoreMetadata	restore(HoodieEngineContext context, String restoreInstantTime, String instantToRestore) Restore the table to the given instant.
abstract HoodieRollbackMetadata	rollback(HoodieEngineContext context, String rollbackInstantTime, HoodieInstant commitInstant, boolean deleteInstants, boolean skipLocking) Rollback the (inflight/committed) record changes with the given commit time.
abstract void	rollbackBootstrap(HoodieEngineContext context, String instantTime) Perform rollback of bootstrap of a Hudi table.
void	rollbackInflightCompaction(HoodieInstant inflightInstant) Rollback failed compactions.
abstract HoodieSavepointMetadata	savepoint(HoodieEngineContext context, String instantToSavepoint, String user, String comment) Create a savepoint at the specified instant, so that the table can be restored to this point-in-timeline later if needed.
abstract Option<HoodieCleanerPlan>	scheduleCleaning(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule cleaning for the instant time.
abstract Option<HoodieClusteringPlan>	scheduleClustering(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule clustering for the instant time.
abstract Option<HoodieCompactionPlan>	scheduleCompaction(HoodieEngineContext context, String instantTime, Option<Map<String,String>> extraMetadata) Schedule compaction for the instant time.
abstract Option<HoodieRollbackPlan>	scheduleRollback(HoodieEngineContext context, String instantTime, HoodieInstant instantToRollback, boolean skipTimelinePublish, boolean shouldRollbackUsingMarkers) Schedule rollback for the instant time.
abstract void	updateMetadataIndexes(HoodieEngineContext context, List<HoodieWriteStat> stats, String instantTime) Updates Metadata Indexes (like Z-Index) TODO rebase onto metadata table (post RFC-27)
abstract HoodieWriteMetadata<O>	upsert(HoodieEngineContext context, String instantTime, I records) Upsert a batch of new records into Hoodie table at the supplied instantTime.
abstract HoodieWriteMetadata<O>	upsertPrepped(HoodieEngineContext context, String instantTime, I preppedRecords) Upserts the given prepared records into the Hoodie table, at the supplied instantTime.
void	validateInsertSchema()
void	validateUpsertSchema()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected final HoodieWriteConfig config

metaClient

protected final HoodieTableMetaClient metaClient

index

protected final HoodieIndex<T extends HoodieRecordPayload,?,?,?> index

taskContextSupplier

protected final TaskContextSupplier taskContextSupplier

context

protected final transient HoodieEngineContext context

Constructor Detail

HoodieTable

protected HoodieTable(HoodieWriteConfig config,
                      HoodieEngineContext context,
                      HoodieTableMetaClient metaClient)

Method Detail

getIndex

protected abstract HoodieIndex<T,?,?,?> getIndex(HoodieWriteConfig config,
                                                 HoodieEngineContext context)

upsert

public abstract HoodieWriteMetadata<O> upsert(HoodieEngineContext context,
                                              String instantTime,
                                              I records)

Upsert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

insert

public abstract HoodieWriteMetadata<O> insert(HoodieEngineContext context,
                                              String instantTime,
                                              I records)

Insert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

bulkInsert

public abstract HoodieWriteMetadata<O> bulkInsert(HoodieEngineContext context,
                                                  String instantTime,
                                                  I records,
                                                  Option<BulkInsertPartitioner<I>> bulkInsertPartitioner)

Bulk Insert a batch of new records into Hoodie table at the supplied instantTime.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

records - hoodieRecords to upsert

bulkInsertPartitioner - User Defined Partitioner

Returns:

HoodieWriteMetadata

delete

public abstract HoodieWriteMetadata<O> delete(HoodieEngineContext context,
                                              String instantTime,
                                              K keys)

Deletes a list of HoodieKeys from the Hoodie table, at the supplied instantTime HoodieKeys will be de-duped and non existent keys will be removed before deleting.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

keys - List of HoodieKeys to be deleted

Returns:

HoodieWriteMetadata

deletePartitions

public abstract HoodieWriteMetadata deletePartitions(HoodieEngineContext context,
                                                     String instantTime,
                                                     List<String> partitions)

Deletes all data of partitions.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

partitions - List of partition to be deleted

Returns:

HoodieWriteMetadata

upsertPrepped

public abstract HoodieWriteMetadata<O> upsertPrepped(HoodieEngineContext context,
                                                     String instantTime,
                                                     I preppedRecords)

Upserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

insertPrepped

public abstract HoodieWriteMetadata<O> insertPrepped(HoodieEngineContext context,
                                                     String instantTime,
                                                     I preppedRecords)

Inserts the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

Returns:

HoodieWriteMetadata

bulkInsertPrepped

public abstract HoodieWriteMetadata<O> bulkInsertPrepped(HoodieEngineContext context,
                                                         String instantTime,
                                                         I preppedRecords,
                                                         Option<BulkInsertPartitioner<I>> bulkInsertPartitioner)

Bulk Insert the given prepared records into the Hoodie table, at the supplied instantTime.

This implementation requires that the input records are already tagged, and de-duped if needed.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for the action

preppedRecords - hoodieRecords to upsert

bulkInsertPartitioner - User Defined Partitioner

Returns:

HoodieWriteMetadata

insertOverwrite

public abstract HoodieWriteMetadata<O> insertOverwrite(HoodieEngineContext context,
                                                       String instantTime,
                                                       I records)

Replaces all the existing records and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.

Parameters:

context - HoodieEngineContext

instantTime - Instant time for the replace action

records - input records

Returns:

HoodieWriteMetadata

insertOverwriteTable

public abstract HoodieWriteMetadata<O> insertOverwriteTable(HoodieEngineContext context,
                                                            String instantTime,
                                                            I records)

Delete all the existing records of the Hoodie table and inserts the specified new records into Hoodie table at the supplied instantTime, for the partition paths contained in input records.

Parameters:

context - HoodieEngineContext

instantTime - Instant time for the replace action

records - input records

Returns:

HoodieWriteMetadata

updateMetadataIndexes

public abstract void updateMetadataIndexes(@Nonnull
                                           HoodieEngineContext context,
                                           @Nonnull
                                           List<HoodieWriteStat> stats,
                                           @Nonnull
                                           String instantTime)
                                    throws Exception

Updates Metadata Indexes (like Z-Index) TODO rebase onto metadata table (post RFC-27)

Parameters:

context - instance of HoodieEngineContext

instantTime - instant of the carried operation triggering the update

Throws:

Exception

getConfig

public HoodieWriteConfig getConfig()

getMetaClient

public HoodieTableMetaClient getMetaClient()

getHadoopConf

public org.apache.hadoop.conf.Configuration getHadoopConf()

getFileSystemView

public TableFileSystemView getFileSystemView()

Get the view of the file system for this table.

getBaseFileOnlyView

public TableFileSystemView.BaseFileOnlyView getBaseFileOnlyView()

Get the base file only view of the file system for this table.

getSliceView

public TableFileSystemView.SliceView getSliceView()

Get the full view of the file system for this table.

getHoodieView

public SyncableFileSystemView getHoodieView()

Get complete view of the file system for this table with ability to force sync.

getCompletedCommitsTimeline

public HoodieTimeline getCompletedCommitsTimeline()

Get only the completed (no-inflights) commit + deltacommit timeline.

getCompletedCommitTimeline

public HoodieTimeline getCompletedCommitTimeline()

Get only the completed (no-inflights) commit timeline.

getPendingCommitTimeline

public HoodieTimeline getPendingCommitTimeline()

Get only the inflights (no-completed) commit timeline.

getCompletedCleanTimeline

public HoodieTimeline getCompletedCleanTimeline()

Get only the completed (no-inflights) clean timeline.

getCleanTimeline

public HoodieTimeline getCleanTimeline()

Get clean timeline.

getRollbackTimeline

public HoodieTimeline getRollbackTimeline()

Get rollback timeline.

getCompletedSavepointTimeline

public HoodieTimeline getCompletedSavepointTimeline()

Get only the completed (no-inflights) savepoint timeline.

getSavepoints

public List<String> getSavepoints()

Get the list of savepoints in this table.

getActiveTimeline

public HoodieActiveTimeline getActiveTimeline()

getIndex

public HoodieIndex<T,?,?,?> getIndex()

Return the index.

scheduleCompaction

public abstract Option<HoodieCompactionPlan> scheduleCompaction(HoodieEngineContext context,
                                                                String instantTime,
                                                                Option<Map<String,String>> extraMetadata)

Schedule compaction for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling compaction

extraMetadata - additional metadata to write into plan

Returns:

compact

public abstract HoodieWriteMetadata<O> compact(HoodieEngineContext context,
                                               String compactionInstantTime)

Run Compaction on the table. Compaction arranges the data so that it is optimized for data access.

Parameters:

context - HoodieEngineContext

compactionInstantTime - Instant Time

scheduleClustering

public abstract Option<HoodieClusteringPlan> scheduleClustering(HoodieEngineContext context,
                                                                String instantTime,
                                                                Option<Map<String,String>> extraMetadata)

Schedule clustering for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling clustering

extraMetadata - additional metadata to write into plan

Returns:

HoodieClusteringPlan, if there is enough data for clustering.

cluster

public abstract HoodieWriteMetadata<O> cluster(HoodieEngineContext context,
                                               String clusteringInstantTime)

Execute Clustering on the table. Clustering re-arranges the data so that it is optimized for data access.

Parameters:

context - HoodieEngineContext

clusteringInstantTime - Instant Time

bootstrap

public abstract HoodieBootstrapWriteMetadata<O> bootstrap(HoodieEngineContext context,
                                                          Option<Map<String,String>> extraMetadata)

Perform metadata/full bootstrap of a Hudi table.

Parameters:

context - HoodieEngineContext

extraMetadata - Additional Metadata for storing in commit file.

Returns:

HoodieBootstrapWriteMetadata

rollbackBootstrap

public abstract void rollbackBootstrap(HoodieEngineContext context,
                                       String instantTime)

Perform rollback of bootstrap of a Hudi table.

Parameters:

context - HoodieEngineContext

scheduleCleaning

public abstract Option<HoodieCleanerPlan> scheduleCleaning(HoodieEngineContext context,
                                                           String instantTime,
                                                           Option<Map<String,String>> extraMetadata)

Schedule cleaning for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling cleaning

extraMetadata - additional metadata to write into plan

Returns:

HoodieCleanerPlan, if there is anything to clean.

clean

public abstract HoodieCleanMetadata clean(HoodieEngineContext context,
                                          String cleanInstantTime,
                                          boolean skipLocking)

Executes a new clean action.

Returns:

information on cleaned file slices

scheduleRollback

public abstract Option<HoodieRollbackPlan> scheduleRollback(HoodieEngineContext context,
                                                            String instantTime,
                                                            HoodieInstant instantToRollback,
                                                            boolean skipTimelinePublish,
                                                            boolean shouldRollbackUsingMarkers)

Schedule rollback for the instant time.

Parameters:

context - HoodieEngineContext

instantTime - Instant Time for scheduling rollback

instantToRollback - instant to be rolled back

shouldRollbackUsingMarkers - uses marker based rollback strategy when set to true. uses list based rollback when false.

Returns:

HoodieRollbackPlan containing info on rollback.

rollback

public abstract HoodieRollbackMetadata rollback(HoodieEngineContext context,
                                                String rollbackInstantTime,
                                                HoodieInstant commitInstant,
                                                boolean deleteInstants,
                                                boolean skipLocking)

Rollback the (inflight/committed) record changes with the given commit time.

   Three steps:
   (1) Atomically unpublish this commit
   (2) clean indexing data
   (3) clean new generated parquet files.
   (4) Finally delete .commit or .inflight file, if deleteInstants = true
 

savepoint

public abstract HoodieSavepointMetadata savepoint(HoodieEngineContext context,
                                                  String instantToSavepoint,
                                                  String user,
                                                  String comment)

Create a savepoint at the specified instant, so that the table can be restored to this point-in-timeline later if needed.

restore

public abstract HoodieRestoreMetadata restore(HoodieEngineContext context,
                                              String restoreInstantTime,
                                              String instantToRestore)

Restore the table to the given instant. Note that this is a admin table recovery operation that would cause any running queries that are accessing file slices written after the instant to fail.

rollbackInflightCompaction

public void rollbackInflightCompaction(HoodieInstant inflightInstant)

Rollback failed compactions. Inflight rollbacks for compactions revert the .inflight file to the .requested file.

Parameters:

inflightInstant - Inflight Compaction Instant

finalizeWrite

public void finalizeWrite(HoodieEngineContext context,
                          String instantTs,
                          List<HoodieWriteStat> stats)
                   throws HoodieIOException

Finalize the written data onto storage. Perform any final cleanups.

Parameters:

context - HoodieEngineContext

stats - List of HoodieWriteStats

Throws:

HoodieIOException - if some paths can't be finalized on storage

getInvalidDataPaths

protected Set<String> getInvalidDataPaths(WriteMarkers markers)
                                   throws IOException

Returns the possible invalid data file name with given marker files.

Throws:

IOException

reconcileAgainstMarkers

protected void reconcileAgainstMarkers(HoodieEngineContext context,
                                       String instantTs,
                                       List<HoodieWriteStat> stats,
                                       boolean consistencyCheckEnabled)
                                throws HoodieIOException

Reconciles WriteStats and marker files to detect and safely delete duplicate data files created because of Spark retries.

Parameters:

context - HoodieEngineContext

instantTs - Instant Timestamp

stats - Hoodie Write Stat

consistencyCheckEnabled - Consistency Check Enabled

Throws:

HoodieIOException

getConsistencyGuard

public static ConsistencyGuard getConsistencyGuard(org.apache.hadoop.fs.FileSystem fs,
                                                   ConsistencyGuardConfig consistencyGuardConfig)
                                            throws IOException

Instantiate ConsistencyGuard based on configs.

Default consistencyGuard class is OptimisticConsistencyGuard.

Throws:

IOException

getTaskContextSupplier

public TaskContextSupplier getTaskContextSupplier()

validateUpsertSchema

public void validateUpsertSchema()
                          throws HoodieUpsertException

Throws:

HoodieUpsertException

validateInsertSchema

public void validateInsertSchema()
                          throws HoodieInsertException

Throws:

HoodieInsertException

getBaseFileFormat

public HoodieFileFormat getBaseFileFormat()

getLogFileFormat

public HoodieFileFormat getLogFileFormat()

getLogDataBlockFormat

public HoodieLogBlock.HoodieLogBlockType getLogDataBlockFormat()

getBaseFileExtension

public String getBaseFileExtension()

requireSortedRecords

public boolean requireSortedRecords()

getContext

public HoodieEngineContext getContext()

getMetadataWriter

public final Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp)

Get Table metadata writer.

Parameters:

triggeringInstantTimestamp - - The instant that is triggering this metadata write

Returns:

instance of {@link HoodieTableMetadataWriter

isTableServiceAction

public abstract boolean isTableServiceAction(String actionType)

Check if action type is a table service.

Parameters:

actionType - action type of interest.

Returns:

true if action represents a table service. false otherwise.

getMetadataWriter

public <T extends org.apache.avro.specific.SpecificRecordBase> Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp,
                                                                                                                   Option<T> actionMetadata)

Get Table metadata writer.

Note: Get the metadata writer for the conf. If the metadata table doesn't exist, this wil trigger the creation of the table and the initial bootstrapping. Since this call is under the transaction lock, other concurrent writers are blocked from doing the similar initial metadata table creation and the bootstrapping.

Parameters:

triggeringInstantTimestamp - - The instant that is triggering this metadata write

Returns:

instance of HoodieTableMetadataWriter