org.apache.hudi.config

Class HoodieClusteringConfig

java.lang.Object

org.apache.hudi.common.config.HoodieConfig

org.apache.hudi.config.HoodieClusteringConfig

All Implemented Interfaces:

Serializable

public class HoodieClusteringConfig
extends HoodieConfig

Clustering specific configs.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieClusteringConfig.BuildCurveStrategyType strategy types for build z-ordering/space-filling curves.
static class	HoodieClusteringConfig.Builder
static class	HoodieClusteringConfig.BuildLayoutOptimizationStrategy strategy types for optimize layout for hudi data.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigProperty<String>	ASYNC_CLUSTERING_ENABLE
static String	ASYNC_CLUSTERING_ENABLE_OPT_KEY Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead
static ConfigProperty<String>	ASYNC_CLUSTERING_MAX_COMMITS
static String	CLUSTERING_EXECUTION_STRATEGY_CLASS Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead
static String	CLUSTERING_MAX_BYTES_PER_GROUP Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead
static String	CLUSTERING_MAX_NUM_GROUPS Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead
static String	CLUSTERING_PLAN_SMALL_FILE_LIMIT Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead
static String	CLUSTERING_PLAN_STRATEGY_CLASS Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead
static String	CLUSTERING_SORT_COLUMNS_PROPERTY Deprecated. Use PLAN_STRATEGY_SORT_COLUMNS and its methods instead
static String	CLUSTERING_STRATEGY_PARAM_PREFIX
static String	CLUSTERING_TARGET_FILE_MAX_BYTES Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead
static String	CLUSTERING_TARGET_PARTITIONS Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead
static String	CLUSTERING_UPDATES_STRATEGY_PROP Deprecated. Use UPDATES_STRATEGY and its methods instead
static ConfigProperty<String>	DAYBASED_LOOKBACK_PARTITIONS
static String	DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead
static String	DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead
static String	DEFAULT_CLUSTERING_MAX_GROUP_SIZE Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead
static String	DEFAULT_CLUSTERING_MAX_NUM_GROUPS Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead
static String	DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead
static String	DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead
static String	DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead
static String	DEFAULT_CLUSTERING_TARGET_PARTITIONS Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead
static String	DEFAULT_CLUSTERING_UPDATES_STRATEGY Deprecated. Use UPDATES_STRATEGY and its methods instead
static ConfigProperty<String>	EXECUTION_STRATEGY_CLASS_NAME
static ConfigProperty<String>	INLINE_CLUSTERING
static String	INLINE_CLUSTERING_MAX_COMMIT_PROP Deprecated. Use INLINE_CLUSTERING_MAX_COMMITS and its methods instead
static ConfigProperty<String>	INLINE_CLUSTERING_MAX_COMMITS
static String	INLINE_CLUSTERING_PROP Deprecated. Use INLINE_CLUSTERING and its methods instead
static String	JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY
static String	JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY
static ConfigProperty	LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE Doing sample for table data is the first step in Boundary-based Interleaved Index method.
static ConfigProperty	LAYOUT_OPTIMIZE_CURVE_BUILD_METHOD There exists two method to build z-curve.
static ConfigProperty	LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE The best way to use Z-order/Space-filling curves is to cooperate with Data-Skipping with data-skipping query engine can greatly reduce the number of table files to be read.
static ConfigProperty	LAYOUT_OPTIMIZE_ENABLE Using space-filling curves to optimize the layout of table to boost query performance.
static String	LAYOUT_OPTIMIZE_PARAM_PREFIX
static ConfigProperty	LAYOUT_OPTIMIZE_STRATEGY
static ConfigProperty<String>	PLAN_STRATEGY_CLASS_NAME
static ConfigProperty<String>	PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP
static ConfigProperty<String>	PLAN_STRATEGY_MAX_GROUPS
static ConfigProperty<String>	PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST
static ConfigProperty<String>	PLAN_STRATEGY_SMALL_FILE_LIMIT
static ConfigProperty<String>	PLAN_STRATEGY_SORT_COLUMNS
static ConfigProperty<String>	PLAN_STRATEGY_TARGET_FILE_MAX_BYTES
static ConfigProperty<Boolean>	PRESERVE_COMMIT_METADATA
static ConfigProperty<Boolean>	ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT
static String	SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY
static String	SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY
static ConfigProperty<String>	UPDATES_STRATEGY

Fields inherited from class org.apache.hudi.common.config.HoodieConfig

props

Constructor Summary

Constructors

Constructor and Description
HoodieClusteringConfig()

Method Summary

Modifier and Type	Method and Description
static HoodieClusteringConfig	from(TypedProperties props)
boolean	isAsyncClusteringEnabled()
boolean	isInlineClusteringEnabled()
static HoodieClusteringConfig.Builder	newBuilder()

Methods inherited from class org.apache.hudi.common.config.HoodieConfig

contains, contains, create, getBoolean, getBooleanOrDefault, getDouble, getFloat, getInt, getLong, getProps, getProps, getString, getString, getStringOrDefault, getStringOrDefault, getStringOrThrow, setAll, setDefaultOnCondition, setDefaults, setDefaultValue, setDefaultValue, setValue, setValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLUSTERING_STRATEGY_PARAM_PREFIX

public static final String CLUSTERING_STRATEGY_PARAM_PREFIX

See Also:

Constant Field Values

SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

public static final String SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

See Also:

Constant Field Values

JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

public static final String JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

See Also:

Constant Field Values

SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY

public static final String SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY

See Also:

Constant Field Values

JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY

public static final String JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY

See Also:

Constant Field Values

LAYOUT_OPTIMIZE_PARAM_PREFIX

public static final String LAYOUT_OPTIMIZE_PARAM_PREFIX

See Also:

Constant Field Values

DAYBASED_LOOKBACK_PARTITIONS

public static final ConfigProperty<String> DAYBASED_LOOKBACK_PARTITIONS

PLAN_STRATEGY_SMALL_FILE_LIMIT

public static final ConfigProperty<String> PLAN_STRATEGY_SMALL_FILE_LIMIT

PLAN_STRATEGY_CLASS_NAME

public static final ConfigProperty<String> PLAN_STRATEGY_CLASS_NAME

EXECUTION_STRATEGY_CLASS_NAME

public static final ConfigProperty<String> EXECUTION_STRATEGY_CLASS_NAME

INLINE_CLUSTERING

public static final ConfigProperty<String> INLINE_CLUSTERING

INLINE_CLUSTERING_MAX_COMMITS

public static final ConfigProperty<String> INLINE_CLUSTERING_MAX_COMMITS

ASYNC_CLUSTERING_MAX_COMMITS

public static final ConfigProperty<String> ASYNC_CLUSTERING_MAX_COMMITS

PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST

public static final ConfigProperty<String> PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST

PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP

public static final ConfigProperty<String> PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP

PLAN_STRATEGY_MAX_GROUPS

public static final ConfigProperty<String> PLAN_STRATEGY_MAX_GROUPS

PLAN_STRATEGY_TARGET_FILE_MAX_BYTES

public static final ConfigProperty<String> PLAN_STRATEGY_TARGET_FILE_MAX_BYTES

PLAN_STRATEGY_SORT_COLUMNS

public static final ConfigProperty<String> PLAN_STRATEGY_SORT_COLUMNS

UPDATES_STRATEGY

public static final ConfigProperty<String> UPDATES_STRATEGY

ASYNC_CLUSTERING_ENABLE

public static final ConfigProperty<String> ASYNC_CLUSTERING_ENABLE

PRESERVE_COMMIT_METADATA

public static final ConfigProperty<Boolean> PRESERVE_COMMIT_METADATA

LAYOUT_OPTIMIZE_ENABLE

public static final ConfigProperty LAYOUT_OPTIMIZE_ENABLE

Using space-filling curves to optimize the layout of table to boost query performance. The table data which sorted by space-filling curve has better aggregation; combine with min-max filtering, it can achieve good performance improvement. Notice: when we use this feature, we need specify the sort columns. The more columns involved in sorting, the worse the aggregation, and the smaller the query performance improvement. Choose the filter columns which commonly used in query sql as sort columns. It is recommend that 2 ~ 4 columns participate in sorting.

LAYOUT_OPTIMIZE_STRATEGY

public static final ConfigProperty LAYOUT_OPTIMIZE_STRATEGY

LAYOUT_OPTIMIZE_CURVE_BUILD_METHOD

public static final ConfigProperty LAYOUT_OPTIMIZE_CURVE_BUILD_METHOD

There exists two method to build z-curve. one is directly mapping sort cols to z-value to build z-curve; we can find this method in Amazon DynamoDB https://aws.amazon.com/cn/blogs/database/tag/z-order/ the other one is Boundary-based Interleaved Index method which we proposed. simply call it sample method. Refer to rfc-28 for specific algorithm flow. Boundary-based Interleaved Index method has better generalization, but the build speed is slower than direct method.

LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE

public static final ConfigProperty LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE

Doing sample for table data is the first step in Boundary-based Interleaved Index method. larger sample number means better optimize result, but more memory consumption

LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE

public static final ConfigProperty LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE

The best way to use Z-order/Space-filling curves is to cooperate with Data-Skipping with data-skipping query engine can greatly reduce the number of table files to be read. otherwise query engine can only do row-group skipping for files (parquet/orc)

ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT

public static final ConfigProperty<Boolean> ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT

CLUSTERING_PLAN_STRATEGY_CLASS

@Deprecated
public static final String CLUSTERING_PLAN_STRATEGY_CLASS

Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead

DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS

@Deprecated
public static final String DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS

Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead

CLUSTERING_EXECUTION_STRATEGY_CLASS

@Deprecated
public static final String CLUSTERING_EXECUTION_STRATEGY_CLASS

Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead

DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS

@Deprecated
public static final String DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS

Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead

INLINE_CLUSTERING_PROP

@Deprecated
public static final String INLINE_CLUSTERING_PROP

Deprecated. Use INLINE_CLUSTERING and its methods instead

INLINE_CLUSTERING_MAX_COMMIT_PROP

@Deprecated
public static final String INLINE_CLUSTERING_MAX_COMMIT_PROP

Deprecated. Use INLINE_CLUSTERING_MAX_COMMITS and its methods instead

CLUSTERING_TARGET_PARTITIONS

@Deprecated
public static final String CLUSTERING_TARGET_PARTITIONS

Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead

DEFAULT_CLUSTERING_TARGET_PARTITIONS

@Deprecated
public static final String DEFAULT_CLUSTERING_TARGET_PARTITIONS

Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead

CLUSTERING_PLAN_SMALL_FILE_LIMIT

@Deprecated
public static final String CLUSTERING_PLAN_SMALL_FILE_LIMIT

Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead

DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT

@Deprecated
public static final String DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT

Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead

CLUSTERING_MAX_BYTES_PER_GROUP

@Deprecated
public static final String CLUSTERING_MAX_BYTES_PER_GROUP

Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead

DEFAULT_CLUSTERING_MAX_GROUP_SIZE

@Deprecated
public static final String DEFAULT_CLUSTERING_MAX_GROUP_SIZE

Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead

CLUSTERING_MAX_NUM_GROUPS

@Deprecated
public static final String CLUSTERING_MAX_NUM_GROUPS

Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead

DEFAULT_CLUSTERING_MAX_NUM_GROUPS

@Deprecated
public static final String DEFAULT_CLUSTERING_MAX_NUM_GROUPS

Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead

CLUSTERING_TARGET_FILE_MAX_BYTES

@Deprecated
public static final String CLUSTERING_TARGET_FILE_MAX_BYTES

Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead

DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES

@Deprecated
public static final String DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES

Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead

CLUSTERING_SORT_COLUMNS_PROPERTY

@Deprecated
public static final String CLUSTERING_SORT_COLUMNS_PROPERTY

Deprecated. Use PLAN_STRATEGY_SORT_COLUMNS and its methods instead

CLUSTERING_UPDATES_STRATEGY_PROP

@Deprecated
public static final String CLUSTERING_UPDATES_STRATEGY_PROP

Deprecated. Use UPDATES_STRATEGY and its methods instead

DEFAULT_CLUSTERING_UPDATES_STRATEGY

@Deprecated
public static final String DEFAULT_CLUSTERING_UPDATES_STRATEGY

Deprecated. Use UPDATES_STRATEGY and its methods instead

ASYNC_CLUSTERING_ENABLE_OPT_KEY

@Deprecated
public static final String ASYNC_CLUSTERING_ENABLE_OPT_KEY

Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead

DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL

@Deprecated
public static final String DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL

Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead

Constructor Detail

HoodieClusteringConfig

public HoodieClusteringConfig()

Method Detail

isAsyncClusteringEnabled

public boolean isAsyncClusteringEnabled()

isInlineClusteringEnabled

public boolean isInlineClusteringEnabled()

from

public static HoodieClusteringConfig from(TypedProperties props)

newBuilder

public static HoodieClusteringConfig.Builder newBuilder()