org.apache.hudi.common.util

Class ClusteringUtils

java.lang.Object

org.apache.hudi.common.util.ClusteringUtils

public class ClusteringUtils
extends Object

Helper class to generate clustering plan from metadata.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TOTAL_IO_READ_MB
static String	TOTAL_LOG_FILE_SIZE
static String	TOTAL_LOG_FILES

Constructor Summary

Constructors

Constructor and Description
ClusteringUtils()

Method Summary

Modifier and Type	Method and Description
static HoodieClusteringPlan	createClusteringPlan(String strategyClassName, Map<String,String> strategyParams, List<FileSlice>[] fileSliceGroups, Map<String,String> extraMetadata) Create clustering plan from input fileSliceGroups.
static Map<HoodieFileGroupId,HoodieInstant>	getAllFileGroupsInPendingClusteringPlans(HoodieTableMetaClient metaClient) Get filegroups to pending clustering instant mapping for all pending clustering plans.
static Stream<Pair<HoodieInstant,HoodieClusteringPlan>>	getAllPendingClusteringPlans(HoodieTableMetaClient metaClient) Get all pending clustering plans along with their instants.
static Option<Pair<HoodieInstant,HoodieClusteringPlan>>	getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant pendingReplaceInstant) Get Clustering plan from timeline.
static Option<Pair<HoodieInstant,HoodieClusteringPlan>>	getClusteringPlan(HoodieTimeline timeline, HoodieInstant pendingReplaceInstant) Get Clustering plan from timeline.
static Option<HoodieInstant>	getEarliestInstantToRetainForClustering(HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient) Returns the earliest instant to retain.
static Stream<HoodieFileGroupId>	getFileGroupsFromClusteringGroup(HoodieClusteringGroup group)
static Stream<HoodieFileGroupId>	getFileGroupsFromClusteringPlan(HoodieClusteringPlan clusteringPlan)
static Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClusteringInstant(HoodieInstant instant, HoodieClusteringPlan clusteringPlan)
static List<HoodieInstant>	getPendingClusteringInstantTimes(HoodieTableMetaClient metaClient)
static boolean	isClusteringInstant(HoodieTimeline timeline, HoodieInstant replaceInstant) Checks if the requested, inflight, or completed instant of replacecommit action is a clustering operation, by checking whether the requested instant contains a clustering plan.
static boolean	isCompletedClusteringInstant(HoodieInstant instant, HoodieTimeline timeline)
static boolean	isInsertOverwriteInstant(HoodieInstant instant, HoodieTimeline timeline) Returns whether the given instant instant is with insert overwrite operation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOTAL_IO_READ_MB

public static final String TOTAL_IO_READ_MB

See Also:

Constant Field Values

TOTAL_LOG_FILE_SIZE

public static final String TOTAL_LOG_FILE_SIZE

See Also:

Constant Field Values

TOTAL_LOG_FILES

public static final String TOTAL_LOG_FILES

See Also:

Constant Field Values

Constructor Detail

ClusteringUtils

public ClusteringUtils()

Method Detail

getAllPendingClusteringPlans

public static Stream<Pair<HoodieInstant,HoodieClusteringPlan>> getAllPendingClusteringPlans(HoodieTableMetaClient metaClient)

Get all pending clustering plans along with their instants.

isClusteringInstant

public static boolean isClusteringInstant(HoodieTimeline timeline,
                                          HoodieInstant replaceInstant)

Checks if the requested, inflight, or completed instant of replacecommit action is a clustering operation, by checking whether the requested instant contains a clustering plan.

Parameters:

timeline - Hudi timeline.

replaceInstant - the instant of replacecommit action to check.

Returns:

whether the instant is a clustering operation.

getClusteringPlan

public static Option<Pair<HoodieInstant,HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient,
                                                                                 HoodieInstant pendingReplaceInstant)

Get Clustering plan from timeline.

Parameters:

metaClient - used to get the active timeline

pendingReplaceInstant - can be in any state, because it will always be converted to requested state

Returns:

option of the replace metadata if present, else empty

getClusteringPlan

public static Option<Pair<HoodieInstant,HoodieClusteringPlan>> getClusteringPlan(HoodieTimeline timeline,
                                                                                 HoodieInstant pendingReplaceInstant)

Get Clustering plan from timeline.

Parameters:

timeline -

pendingReplaceInstant -

Returns:

getAllFileGroupsInPendingClusteringPlans

public static Map<HoodieFileGroupId,HoodieInstant> getAllFileGroupsInPendingClusteringPlans(HoodieTableMetaClient metaClient)

Get filegroups to pending clustering instant mapping for all pending clustering plans. This includes all clustering operations in 'requested' and 'inflight' states.

getFileGroupsInPendingClusteringInstant

public static Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClusteringInstant(HoodieInstant instant,
                                                                                                    HoodieClusteringPlan clusteringPlan)

getFileGroupsFromClusteringPlan

public static Stream<HoodieFileGroupId> getFileGroupsFromClusteringPlan(HoodieClusteringPlan clusteringPlan)

getFileGroupsFromClusteringGroup

public static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group)

createClusteringPlan

public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
                                                        Map<String,String> strategyParams,
                                                        List<FileSlice>[] fileSliceGroups,
                                                        Map<String,String> extraMetadata)

Create clustering plan from input fileSliceGroups.

getPendingClusteringInstantTimes

public static List<HoodieInstant> getPendingClusteringInstantTimes(HoodieTableMetaClient metaClient)

getEarliestInstantToRetainForClustering

public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(HoodieActiveTimeline activeTimeline,
                                                                            HoodieTableMetaClient metaClient)
                                                                     throws IOException

Returns the earliest instant to retain. Make sure the clustering instant won't be archived before cleaned, and the earliest inflight clustering instant has a previous commit.

Parameters:

activeTimeline - The active timeline

metaClient - The meta client

Returns:

the earliest instant to retain for clustering

Throws:

IOException

isCompletedClusteringInstant

public static boolean isCompletedClusteringInstant(HoodieInstant instant,
                                                   HoodieTimeline timeline)

Parameters:

instant - Hudi instant to check.

timeline - Hudi timeline.

Returns:

whether the given instant is a completed clustering operation.

isInsertOverwriteInstant

public static boolean isInsertOverwriteInstant(HoodieInstant instant,
                                               HoodieTimeline timeline)

Returns whether the given instant instant is with insert overwrite operation.