org.apache.hudi.client

Class CompactionAdminClient

java.lang.Object

org.apache.hudi.client.BaseHoodieClient

org.apache.hudi.client.CompactionAdminClient

All Implemented Interfaces:

Serializable, AutoCloseable

public class CompactionAdminClient
extends BaseHoodieClient

Client to perform admin operations related to compaction.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompactionAdminClient.CompactionValidationException
static class	CompactionAdminClient.RenameInfo
static class	CompactionAdminClient.RenameOpResult Holds Operation result for Renaming.
static class	CompactionAdminClient.ValidationOpResult Holds Operation result for Renaming.

Field Summary

Fields inherited from class org.apache.hudi.client.BaseHoodieClient

basePath, config, context, fs, hadoopConf, heartbeatClient

Constructor Summary

Constructors

Constructor and Description
CompactionAdminClient(HoodieEngineContext context, String basePath)

Method Summary

Modifier and Type	Method and Description
List<Pair<HoodieLogFile,HoodieLogFile>>	getRenamingActionsForUnschedulingCompactionForFileId(HoodieTableMetaClient metaClient, HoodieFileGroupId fgId, Option<HoodieTableFileSystemView> fsViewOpt, boolean skipValidation) Generate renaming actions for unscheduling a fileId from pending compaction.
List<Pair<HoodieLogFile,HoodieLogFile>>	getRenamingActionsForUnschedulingCompactionOperation(HoodieTableMetaClient metaClient, String compactionInstant, CompactionOperation operation, Option<HoodieTableFileSystemView> fsViewOpt, boolean skipValidation) Generate renaming actions for unscheduling a compaction operation NOTE: Can only be used safely when no writer (ingestion/compaction) is running.
List<Pair<HoodieLogFile,HoodieLogFile>>	getRenamingActionsForUnschedulingCompactionPlan(HoodieTableMetaClient metaClient, String compactionInstant, int parallelism, Option<HoodieTableFileSystemView> fsViewOpt, boolean skipValidation) Generate renaming actions for unscheduling a pending compaction plan.
protected static List<Pair<HoodieLogFile,HoodieLogFile>>	getRenamingActionsToAlignWithCompactionOperation(HoodieTableMetaClient metaClient, String compactionInstant, CompactionOperation op, Option<HoodieTableFileSystemView> fsViewOpt) Get Renaming actions to ensure the log-files of merged file-slices is aligned with compaction operation.
protected static void	renameLogFile(HoodieTableMetaClient metaClient, HoodieLogFile oldLogFile, HoodieLogFile newLogFile) Rename log files.
List<CompactionAdminClient.RenameOpResult>	repairCompaction(String compactionInstant, int parallelism, boolean dryRun) Renames delta files to make file-slices consistent with the timeline as dictated by Hoodie metadata.
List<CompactionAdminClient.RenameOpResult>	unscheduleCompactionFileId(HoodieFileGroupId fgId, boolean skipValidation, boolean dryRun) Remove a fileId from pending compaction.
List<CompactionAdminClient.RenameOpResult>	unscheduleCompactionPlan(String compactionInstant, boolean skipValidation, int parallelism, boolean dryRun) Un-schedules compaction plan.
List<CompactionAdminClient.ValidationOpResult>	validateCompactionPlan(HoodieTableMetaClient metaClient, String compactionInstant, int parallelism) Validate all compaction operations in a compaction plan.

Methods inherited from class org.apache.hudi.client.BaseHoodieClient

close, createMetaClient, getConfig, getEngineContext, getHeartbeatClient, getTimelineServer, initWrapperFSMetrics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompactionAdminClient

public CompactionAdminClient(HoodieEngineContext context,
                             String basePath)

Method Detail

validateCompactionPlan

public List<CompactionAdminClient.ValidationOpResult> validateCompactionPlan(HoodieTableMetaClient metaClient,
                                                                             String compactionInstant,
                                                                             int parallelism)
                                                                      throws IOException

Validate all compaction operations in a compaction plan. Verifies the file-slices are consistent with corresponding compaction operations.

Parameters:

metaClient - Hoodie Table Meta Client

compactionInstant - Compaction Instant

Throws:

IOException

unscheduleCompactionPlan

public List<CompactionAdminClient.RenameOpResult> unscheduleCompactionPlan(String compactionInstant,
                                                                           boolean skipValidation,
                                                                           int parallelism,
                                                                           boolean dryRun)
                                                                    throws Exception

Un-schedules compaction plan. Remove All compaction operation scheduled and re-arrange delta-files that were created after the compaction was scheduled. This operation MUST be executed with compactions and writer turned OFF.

Parameters:

compactionInstant - Compaction Instant

skipValidation - Skip validation step

parallelism - Parallelism

dryRun - Dry Run

Throws:

Exception

unscheduleCompactionFileId

public List<CompactionAdminClient.RenameOpResult> unscheduleCompactionFileId(HoodieFileGroupId fgId,
                                                                             boolean skipValidation,
                                                                             boolean dryRun)
                                                                      throws Exception

Remove a fileId from pending compaction. Removes the associated compaction operation and rename delta-files that were generated for that file-id after the compaction operation was scheduled. This operation MUST be executed with compactions and writer turned OFF.

Parameters:

fgId - FileGroupId to be unscheduled

skipValidation - Skip validation

dryRun - Dry Run Mode

Throws:

Exception

repairCompaction

public List<CompactionAdminClient.RenameOpResult> repairCompaction(String compactionInstant,
                                                                   int parallelism,
                                                                   boolean dryRun)
                                                            throws Exception

Renames delta files to make file-slices consistent with the timeline as dictated by Hoodie metadata. Use when compaction unschedule fails partially. This operation MUST be executed with compactions and writer turned OFF.

Parameters:

compactionInstant - Compaction Instant to be repaired

dryRun - Dry Run Mode

Throws:

Exception

getRenamingActionsToAlignWithCompactionOperation

protected static List<Pair<HoodieLogFile,HoodieLogFile>> getRenamingActionsToAlignWithCompactionOperation(HoodieTableMetaClient metaClient,
                                                                                                          String compactionInstant,
                                                                                                          CompactionOperation op,
                                                                                                          Option<HoodieTableFileSystemView> fsViewOpt)

Get Renaming actions to ensure the log-files of merged file-slices is aligned with compaction operation. This method is used to recover from failures during unschedule compaction operations.

Parameters:

metaClient - Hoodie Table Meta Client

compactionInstant - Compaction Instant

op - Compaction Operation

fsViewOpt - File System View

renameLogFile

protected static void renameLogFile(HoodieTableMetaClient metaClient,
                                    HoodieLogFile oldLogFile,
                                    HoodieLogFile newLogFile)
                             throws IOException

Rename log files. This is done for un-scheduling a pending compaction operation NOTE: Can only be used safely when no writer (ingestion/compaction) is running.

Parameters:

metaClient - Hoodie Table Meta-Client

oldLogFile - Old Log File

newLogFile - New Log File

Throws:

IOException

getRenamingActionsForUnschedulingCompactionPlan

public List<Pair<HoodieLogFile,HoodieLogFile>> getRenamingActionsForUnschedulingCompactionPlan(HoodieTableMetaClient metaClient,
                                                                                               String compactionInstant,
                                                                                               int parallelism,
                                                                                               Option<HoodieTableFileSystemView> fsViewOpt,
                                                                                               boolean skipValidation)
                                                                                        throws IOException

Generate renaming actions for unscheduling a pending compaction plan. NOTE: Can only be used safely when no writer (ingestion/compaction) is running.

Parameters:

metaClient - Hoodie Table MetaClient

compactionInstant - Compaction Instant to be unscheduled

fsViewOpt - Cached File System View

skipValidation - Skip Validation

Returns:

list of pairs of log-files (old, new) and for each pair, rename must be done to successfully unschedule compaction.

Throws:

IOException

getRenamingActionsForUnschedulingCompactionOperation

public List<Pair<HoodieLogFile,HoodieLogFile>> getRenamingActionsForUnschedulingCompactionOperation(HoodieTableMetaClient metaClient,
                                                                                                    String compactionInstant,
                                                                                                    CompactionOperation operation,
                                                                                                    Option<HoodieTableFileSystemView> fsViewOpt,
                                                                                                    boolean skipValidation)
                                                                                             throws IOException

Generate renaming actions for unscheduling a compaction operation NOTE: Can only be used safely when no writer (ingestion/compaction) is running.

Parameters:

metaClient - Hoodie Table MetaClient

compactionInstant - Compaction Instant

operation - Compaction Operation

fsViewOpt - Cached File System View

skipValidation - Skip Validation

Returns:

list of pairs of log-files (old, new) and for each pair, rename must be done to successfully unschedule compaction.

Throws:

IOException

getRenamingActionsForUnschedulingCompactionForFileId

public List<Pair<HoodieLogFile,HoodieLogFile>> getRenamingActionsForUnschedulingCompactionForFileId(HoodieTableMetaClient metaClient,
                                                                                                    HoodieFileGroupId fgId,
                                                                                                    Option<HoodieTableFileSystemView> fsViewOpt,
                                                                                                    boolean skipValidation)
                                                                                             throws IOException

Generate renaming actions for unscheduling a fileId from pending compaction. NOTE: Can only be used safely when no writer (ingestion/compaction) is running.

Parameters:

metaClient - Hoodie Table MetaClient

fgId - FileGroupId to remove compaction

fsViewOpt - Cached File System View

skipValidation - Skip Validation

Returns:

list of pairs of log-files (old, new) and for each pair, rename must be done to successfully unschedule compaction.

Throws:

IOException