org.apache.hudi.table.action.compact.strategy

Class CompactionStrategy

java.lang.Object

org.apache.hudi.table.action.compact.strategy.CompactionStrategy

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

BoundedIOCompactionStrategy, CompositeCompactionStrategy, PartitionRegexBasedCompactionStrategy, UnBoundedCompactionStrategy, UnBoundedPartitionAwareCompactionStrategy

public abstract class CompactionStrategy
extends Object
implements Serializable

Strategy for compaction. Pluggable implementation to define how compaction should be done. The over-ridden implementations of this abstract class can capture the relevant metrics to order and filter the final list of compaction operation to run in a single compaction. Implementation of CompactionStrategy cannot hold any state. Difference instantiations can be passed in every time

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	TOTAL_IO_MB
static String	TOTAL_IO_READ_MB
static String	TOTAL_IO_WRITE_MB
static String	TOTAL_LOG_FILE_SIZE
static String	TOTAL_LOG_FILES

Constructor Summary

Constructors

Constructor and Description
CompactionStrategy()

Method Summary

Modifier and Type	Method and Description
Map<String,Double>	captureMetrics(HoodieWriteConfig writeConfig, FileSlice slice) Callback hook when a HoodieCompactionOperation is created.
List<String>	filterPartitionPaths(HoodieWriteConfig writeConfig, List<String> allPartitionPaths) Filter the partition paths based on compaction strategy.
HoodieCompactionPlan	generateCompactionPlan(HoodieWriteConfig writeConfig, List<HoodieCompactionOperation> operations, List<HoodieCompactionPlan> pendingCompactionPlans) Generate Compaction plan.
List<HoodieCompactionOperation>	orderAndFilter(HoodieWriteConfig writeConfig, List<HoodieCompactionOperation> operations, List<HoodieCompactionPlan> pendingCompactionPlans) Order and Filter the list of compactions.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOTAL_IO_READ_MB

public static final String TOTAL_IO_READ_MB

See Also:

Constant Field Values

TOTAL_IO_WRITE_MB

public static final String TOTAL_IO_WRITE_MB

See Also:

Constant Field Values

TOTAL_IO_MB

public static final String TOTAL_IO_MB

See Also:

Constant Field Values

TOTAL_LOG_FILE_SIZE

public static final String TOTAL_LOG_FILE_SIZE

See Also:

Constant Field Values

TOTAL_LOG_FILES

public static final String TOTAL_LOG_FILES

See Also:

Constant Field Values

Constructor Detail

CompactionStrategy

public CompactionStrategy()

Method Detail

captureMetrics

public Map<String,Double> captureMetrics(HoodieWriteConfig writeConfig,
                                         FileSlice slice)

Callback hook when a HoodieCompactionOperation is created. Individual strategies can capture the metrics they need to decide on the priority.

Parameters:

writeConfig - write configuration.

slice - fileSlice to capture metrics for.

Returns:

Map[String, Object] - metrics captured

generateCompactionPlan

public HoodieCompactionPlan generateCompactionPlan(HoodieWriteConfig writeConfig,
                                                   List<HoodieCompactionOperation> operations,
                                                   List<HoodieCompactionPlan> pendingCompactionPlans)

Generate Compaction plan. Allows clients to order and filter the list of compactions to be set. The default implementation takes care of setting compactor Id from configuration allowing subclasses to only worry about ordering and filtering compaction operations

Parameters:

writeConfig - Hoodie Write Config

operations - Compaction Operations to be ordered and filtered

pendingCompactionPlans - Pending Compaction Plans for strategy to schedule next compaction plan

Returns:

Compaction plan to be scheduled.

orderAndFilter

public List<HoodieCompactionOperation> orderAndFilter(HoodieWriteConfig writeConfig,
                                                      List<HoodieCompactionOperation> operations,
                                                      List<HoodieCompactionPlan> pendingCompactionPlans)

Order and Filter the list of compactions. Use the metrics captured with the captureMetrics to order and filter out compactions

Parameters:

writeConfig - config for this compaction is passed in

operations - list of compactions collected

pendingCompactionPlans - Pending Compaction Plans for strategy to schedule next compaction plan

Returns:

list of compactions to perform in this run

filterPartitionPaths

public List<String> filterPartitionPaths(HoodieWriteConfig writeConfig,
                                         List<String> allPartitionPaths)

Filter the partition paths based on compaction strategy.

Parameters:

writeConfig -

allPartitionPaths -

Returns: