org.apache.hudi.common.util

Class CompactionUtils

java.lang.Object

org.apache.hudi.common.util.CompactionUtils

public class CompactionUtils
extends Object

Helper class to generate compaction plan from FileGroup/FileSlice abstraction.

Field Summary

Fields

Modifier and Type	Field and Description
static Integer	COMPACTION_METADATA_VERSION_1
static Integer	COMPACTION_METADATA_VERSION_2
static Integer	LATEST_COMPACTION_METADATA_VERSION

Constructor Summary

Constructors

Constructor and Description
CompactionUtils()

Method Summary

Modifier and Type	Method and Description
static CompactionOperation	buildCompactionOperation(HoodieCompactionOperation hc) Build Compaction operation payload from Avro version for using in Spark executors.
static HoodieCompactionOperation	buildFromFileSlice(String partitionPath, FileSlice fileSlice, Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction) Generate compaction operation from file-slice.
static HoodieCompactionPlan	buildFromFileSlices(List<Pair<String,FileSlice>> partitionFileSlicePairs, Option<Map<String,String>> extraMetadata, Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction) Generate compaction plan from file-slices.
static HoodieCompactionOperation	buildHoodieCompactionOperation(CompactionOperation op) Build Avro generated Compaction operation payload from compaction operation POJO for serialization.
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>	getAllPendingCompactionOperations(HoodieTableMetaClient metaClient) Get all PartitionPath + file-ids with pending Compaction operations and their target compaction instant time.
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>	getAllPendingCompactionOperationsInPendingCompactionPlans(List<Pair<HoodieInstant,HoodieCompactionPlan>> pendingLogCompactionPlanWithInstants) Get all partition + file Ids with pending Log Compaction operations and their target log compaction instant time.
static List<Pair<HoodieInstant,HoodieCompactionPlan>>	getAllPendingCompactionPlans(HoodieTableMetaClient metaClient) Get all pending compaction plans along with their instants.
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>	getAllPendingLogCompactionOperations(HoodieTableMetaClient metaClient) Get all partition + file Ids with pending Log Compaction operations and their target log compaction instant time.
static List<Pair<HoodieInstant,HoodieCompactionPlan>>	getAllPendingLogCompactionPlans(HoodieTableMetaClient metaClient) Get all pending logcompaction plans along with their instants.
static HoodieCompactionPlan	getCompactionPlan(HoodieTableMetaClient metaClient, Option<byte[]> planContent) Util method to fetch both compaction and log compaction plan from requestedInstant.
static HoodieCompactionPlan	getCompactionPlan(HoodieTableMetaClient metaClient, String compactionInstant) This method will serve only Compaction instants because we use same HoodieCompactionPlan for both the operations.
static Option<Pair<HoodieTimeline,HoodieInstant>>	getCompletedDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline) Returns a pair of (timeline containing the completed delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the completed delta commits, the first delta commit instant), if there is no completed compaction commit.
static Option<Pair<HoodieTimeline,HoodieInstant>>	getDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline) Returns a pair of (timeline containing the delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the delta commits, the first delta commit instant), if there is no completed compaction commit.
static Option<Pair<HoodieTimeline,HoodieInstant>>	getDeltaCommitsSinceLatestCompactionRequest(HoodieActiveTimeline activeTimeline)
static Option<HoodieInstant>	getEarliestInstantToRetainForCompaction(HoodieActiveTimeline activeTimeline, int maxDeltaCommits) Gets the earliest instant to retain for MOR compaction.
static HoodieCompactionPlan	getLogCompactionPlan(HoodieTableMetaClient metaClient, String logCompactionInstant) This method will serve only log compaction instants, because we use same HoodieCompactionPlan for both the operations.
static List<HoodieInstant>	getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient) Return all pending compaction instant times.
static Stream<Pair<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>>	getPendingCompactionOperations(HoodieInstant instant, HoodieCompactionPlan compactionPlan) Get pending compaction operations for both major and minor compaction.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COMPACTION_METADATA_VERSION_1

public static final Integer COMPACTION_METADATA_VERSION_1

COMPACTION_METADATA_VERSION_2

public static final Integer COMPACTION_METADATA_VERSION_2

LATEST_COMPACTION_METADATA_VERSION

public static final Integer LATEST_COMPACTION_METADATA_VERSION

Constructor Detail

CompactionUtils

public CompactionUtils()

Method Detail

buildFromFileSlice

public static HoodieCompactionOperation buildFromFileSlice(String partitionPath,
                                                           FileSlice fileSlice,
                                                           Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction)

Generate compaction operation from file-slice.

Parameters:

partitionPath - Partition path

fileSlice - File Slice

metricsCaptureFunction - Metrics Capture function

Returns:

Compaction Operation

buildFromFileSlices

public static HoodieCompactionPlan buildFromFileSlices(List<Pair<String,FileSlice>> partitionFileSlicePairs,
                                                       Option<Map<String,String>> extraMetadata,
                                                       Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction)

Generate compaction plan from file-slices.

Parameters:

partitionFileSlicePairs - list of partition file-slice pairs

extraMetadata - Extra Metadata

metricsCaptureFunction - Metrics Capture function

buildHoodieCompactionOperation

public static HoodieCompactionOperation buildHoodieCompactionOperation(CompactionOperation op)

Build Avro generated Compaction operation payload from compaction operation POJO for serialization.

buildCompactionOperation

public static CompactionOperation buildCompactionOperation(HoodieCompactionOperation hc)

Build Compaction operation payload from Avro version for using in Spark executors.

Parameters:

hc - HoodieCompactionOperation

getAllPendingCompactionPlans

public static List<Pair<HoodieInstant,HoodieCompactionPlan>> getAllPendingCompactionPlans(HoodieTableMetaClient metaClient)

Get all pending compaction plans along with their instants.

Parameters:

metaClient - Hoodie Meta Client

getAllPendingLogCompactionPlans

public static List<Pair<HoodieInstant,HoodieCompactionPlan>> getAllPendingLogCompactionPlans(HoodieTableMetaClient metaClient)

Get all pending logcompaction plans along with their instants.

Parameters:

metaClient - Hoodie Meta Client

getCompactionPlan

public static HoodieCompactionPlan getCompactionPlan(HoodieTableMetaClient metaClient,
                                                     String compactionInstant)

This method will serve only Compaction instants because we use same HoodieCompactionPlan for both the operations.

getLogCompactionPlan

public static HoodieCompactionPlan getLogCompactionPlan(HoodieTableMetaClient metaClient,
                                                        String logCompactionInstant)

This method will serve only log compaction instants, because we use same HoodieCompactionPlan for both the operations.

getCompactionPlan

public static HoodieCompactionPlan getCompactionPlan(HoodieTableMetaClient metaClient,
                                                     Option<byte[]> planContent)

Util method to fetch both compaction and log compaction plan from requestedInstant.

getAllPendingCompactionOperations

public static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> getAllPendingCompactionOperations(HoodieTableMetaClient metaClient)

Get all PartitionPath + file-ids with pending Compaction operations and their target compaction instant time.

Parameters:

metaClient - Hoodie Table Meta Client

getAllPendingLogCompactionOperations

public static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> getAllPendingLogCompactionOperations(HoodieTableMetaClient metaClient)

Get all partition + file Ids with pending Log Compaction operations and their target log compaction instant time.

getAllPendingCompactionOperationsInPendingCompactionPlans

public static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> getAllPendingCompactionOperationsInPendingCompactionPlans(List<Pair<HoodieInstant,HoodieCompactionPlan>> pendingLogCompactionPlanWithInstants)

Get all partition + file Ids with pending Log Compaction operations and their target log compaction instant time.

getPendingCompactionOperations

public static Stream<Pair<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>> getPendingCompactionOperations(HoodieInstant instant,
                                                                                                                    HoodieCompactionPlan compactionPlan)

Get pending compaction operations for both major and minor compaction.

getPendingCompactionInstantTimes

public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient)

Return all pending compaction instant times.

Returns:

getCompletedDeltaCommitsSinceLatestCompaction

public static Option<Pair<HoodieTimeline,HoodieInstant>> getCompletedDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline)

Returns a pair of (timeline containing the completed delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the completed delta commits, the first delta commit instant), if there is no completed compaction commit.

Parameters:

activeTimeline - Active timeline of a table.

Returns:

Pair of timeline containing delta commits and an instant.

getDeltaCommitsSinceLatestCompaction

public static Option<Pair<HoodieTimeline,HoodieInstant>> getDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline)

Returns a pair of (timeline containing the delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the delta commits, the first delta commit instant), if there is no completed compaction commit.

Parameters:

activeTimeline - Active timeline of a table.

Returns:

Pair of timeline containing delta commits and an instant.

getDeltaCommitsSinceLatestCompactionRequest

public static Option<Pair<HoodieTimeline,HoodieInstant>> getDeltaCommitsSinceLatestCompactionRequest(HoodieActiveTimeline activeTimeline)

getEarliestInstantToRetainForCompaction

public static Option<HoodieInstant> getEarliestInstantToRetainForCompaction(HoodieActiveTimeline activeTimeline,
                                                                            int maxDeltaCommits)

Gets the earliest instant to retain for MOR compaction. If there is no completed compaction, num delta commits >= "hoodie.compact.inline.max.delta.commits" If there is a completed compaction, num delta commits after latest completed compaction >= "hoodie.compact.inline.max.delta.commits"

Parameters:

activeTimeline - Active timeline of a table.

maxDeltaCommits - Maximum number of delta commits that trigger the compaction plan, i.e., "hoodie.compact.inline.max.delta.commits".

Returns:

the earliest instant to keep for MOR compaction.