org.apache.hudi.table.action.cluster.strategy

Class BaseConsistentHashingBucketClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy<T,I,K,O>

org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy<T,I,K,O>

org.apache.hudi.table.action.cluster.strategy.BaseConsistentHashingBucketClusteringPlanStrategy<T,I,K,O>

All Implemented Interfaces:

Serializable

public abstract class BaseConsistentHashingBucketClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>
extends PartitionAwareClusteringPlanStrategy<T,I,K,O>

Clustering plan strategy specifically for consistent bucket index.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	METADATA_CHILD_NODE_KEY
static String	METADATA_PARTITION_KEY
static String	METADATA_SEQUENCE_NUMBER_KEY

Fields inherited from class org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy

CLUSTERING_PLAN_VERSION_1, hoodieTable

Constructor Summary

Constructors

Constructor and Description
BaseConsistentHashingBucketClusteringPlanStrategy(HoodieTable table, HoodieEngineContext engineContext, HoodieWriteConfig writeConfig)

Method Summary

Modifier and Type	Method and Description
protected Stream<HoodieClusteringGroup>	buildClusteringGroupsForPartition(String partitionPath, List<FileSlice> fileSlices) Generate cluster group based on split, merge and sort rules
protected Triple<List<HoodieClusteringGroup>,Integer,List<FileSlice>>	buildMergeClusteringGroup(ConsistentBucketIdentifier identifier, List<FileSlice> fileSlices, int mergeSlot) Generate clustering group according to merge rules
protected Triple<List<HoodieClusteringGroup>,Integer,List<FileSlice>>	buildSplitClusteringGroups(ConsistentBucketIdentifier identifier, List<FileSlice> fileSlices, int splitSlot) Generate clustering groups according to split rules.
boolean	checkPrecondition() TODO maybe add force config to schedule the clustering.
protected Stream<FileSlice>	getFileSlicesEligibleForClustering(String partition) Generate candidate clustering file slices of the given partition.
protected Map<String,String>	getStrategyParams() Get parameters specific to strategy.
protected boolean	isBucketClusteringMergeEnabled() Whether enable buckets merged when using consistent hashing bucket index.
protected boolean	isBucketClusteringSortEnabled() Whether generate regular sort clustering plans for buckets that are not involved in merge or split.

Methods inherited from class org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy

filterPartitionPaths, generateClusteringPlan, getNumberOfOutputFileGroups, getRegexPatternMatchedPartitions

Methods inherited from class org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy

buildMetrics, checkAndGetClusteringPlanStrategy, getEngineContext, getExtraMetadata, getFileSliceInfo, getHoodieTable, getPlanVersion, getWriteConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METADATA_PARTITION_KEY

public static final String METADATA_PARTITION_KEY

See Also:

Constant Field Values

METADATA_CHILD_NODE_KEY

public static final String METADATA_CHILD_NODE_KEY

See Also:

Constant Field Values

METADATA_SEQUENCE_NUMBER_KEY

public static final String METADATA_SEQUENCE_NUMBER_KEY

See Also:

Constant Field Values

Constructor Detail

BaseConsistentHashingBucketClusteringPlanStrategy

public BaseConsistentHashingBucketClusteringPlanStrategy(HoodieTable table,
                                                         HoodieEngineContext engineContext,
                                                         HoodieWriteConfig writeConfig)

Method Detail

checkPrecondition

public boolean checkPrecondition()

TODO maybe add force config to schedule the clustering. It could allow clustering on partitions that are not doing write operation. Block clustering if there is any ongoing concurrent writers

Overrides:

checkPrecondition in class ClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

Returns:

true if the schedule can proceed

buildClusteringGroupsForPartition

protected Stream<HoodieClusteringGroup> buildClusteringGroupsForPartition(String partitionPath,
                                                                          List<FileSlice> fileSlices)

Generate cluster group based on split, merge and sort rules

Overrides:

buildClusteringGroupsForPartition in class PartitionAwareClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

isBucketClusteringMergeEnabled

protected boolean isBucketClusteringMergeEnabled()

Whether enable buckets merged when using consistent hashing bucket index.

Returns:

true if bucket merge is enabled, false otherwise.

isBucketClusteringSortEnabled

protected boolean isBucketClusteringSortEnabled()

Whether generate regular sort clustering plans for buckets that are not involved in merge or split.

Returns:

true if generate regular sort clustering plans for buckets that are not involved in merge or split, false otherwise.

getFileSlicesEligibleForClustering

protected Stream<FileSlice> getFileSlicesEligibleForClustering(String partition)

Generate candidate clustering file slices of the given partition. If there is inflight / requested clustering working on the partition, then return empty list to ensure serialized update to the hashing metadata.

Overrides:

getFileSlicesEligibleForClustering in class ClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

Returns:

candidate file slices to be clustered (i.e., sort, bucket split or merge)

getStrategyParams

protected Map<String,String> getStrategyParams()

Description copied from class: ClusteringPlanStrategy

Get parameters specific to strategy. These parameters are passed from 'schedule clustering' step to 'execute clustering' step. 'execute clustering' step is typically async. So these params help with passing any required context from schedule to run step.

Specified by:

getStrategyParams in class ClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

buildSplitClusteringGroups

protected Triple<List<HoodieClusteringGroup>,Integer,List<FileSlice>> buildSplitClusteringGroups(ConsistentBucketIdentifier identifier,
                                                                                                 List<FileSlice> fileSlices,
                                                                                                 int splitSlot)

Generate clustering groups according to split rules. Currently, we always split bucket into two sub-buckets.

Parameters:

identifier - bucket identifier

fileSlices - file slice candidate to be built as split clustering groups

splitSlot - number of new bucket allowed to produce, in order to constrain the upper bound of the total number of bucket

Returns:

list of clustering group, number of new buckets generated, remaining file slice (that does not split)

buildMergeClusteringGroup

protected Triple<List<HoodieClusteringGroup>,Integer,List<FileSlice>> buildMergeClusteringGroup(ConsistentBucketIdentifier identifier,
                                                                                                List<FileSlice> fileSlices,
                                                                                                int mergeSlot)

Generate clustering group according to merge rules

Parameters:

identifier - bucket identifier

fileSlices - file slice candidates to be built as merge clustering groups

mergeSlot - number of bucket allowed to be merged, in order to guarantee the lower bound of the total number of bucket

Returns:

list of clustering group, number of buckets merged (removed), remaining file slice (that does not be merged)