org.apache.hudi.util

Class CompactionUtil

java.lang.Object

org.apache.hudi.util.CompactionUtil

public class CompactionUtil
extends Object

Utilities for flink hudi compaction.

Constructor Summary

Constructors

Constructor and Description
CompactionUtil()

Method Summary

Modifier and Type	Method and Description
static Option<String>	getCompactionInstantTime(HoodieTableMetaClient metaClient) Gets compaction Instant time.
static void	inferChangelogMode(org.apache.flink.configuration.Configuration conf, HoodieTableMetaClient metaClient) Infers the changelog mode based on the data file schema(including metadata fields).
static void	inferMetadataConf(org.apache.flink.configuration.Configuration conf, HoodieTableMetaClient metaClient) Infers the metadata config based on the existence of metadata folder.
static boolean	isLIFO(String seq) Returns whether the execution sequence is LIFO.
static void	rollbackCompaction(HoodieFlinkTable<?> table) Force rolls back all the inflight compaction instants, especially for job failover restart.
static void	rollbackCompaction(HoodieFlinkTable<?> table, String instantTime)
static void	rollbackEarliestCompaction(HoodieFlinkTable<?> table, org.apache.flink.configuration.Configuration conf) Rolls back the earliest compaction if there exists.
static void	scheduleCompaction(HoodieTableMetaClient metaClient, HoodieFlinkWriteClient<?> writeClient, boolean deltaTimeCompaction, boolean committed) Schedules a new compaction instant.
static void	setAvroSchema(org.apache.flink.configuration.Configuration conf, HoodieTableMetaClient metaClient) Sets up the avro schema string into the give configuration conf through reading from the hoodie table metadata.
static void	setAvroSchema(HoodieWriteConfig writeConfig, HoodieTableMetaClient metaClient) Sets up the avro schema string into the HoodieWriteConfig HoodieWriteConfig through reading from the hoodie table metadata.
static void	setPreCombineField(org.apache.flink.configuration.Configuration conf, HoodieTableMetaClient metaClient) Sets up the preCombine field into the given configuration conf through reading from the hoodie table metadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompactionUtil

public CompactionUtil()

Method Detail

scheduleCompaction

public static void scheduleCompaction(HoodieTableMetaClient metaClient,
                                      HoodieFlinkWriteClient<?> writeClient,
                                      boolean deltaTimeCompaction,
                                      boolean committed)

Schedules a new compaction instant.

Parameters:

metaClient - The metadata client

writeClient - The write client

deltaTimeCompaction - Whether the compaction is trigger by elapsed delta time

committed - Whether the last instant was committed successfully

getCompactionInstantTime

public static Option<String> getCompactionInstantTime(HoodieTableMetaClient metaClient)

Gets compaction Instant time.

setAvroSchema

public static void setAvroSchema(org.apache.flink.configuration.Configuration conf,
                                 HoodieTableMetaClient metaClient)
                          throws Exception

Sets up the avro schema string into the give configuration conf through reading from the hoodie table metadata.

Parameters:

conf - The configuration

Throws:

Exception

setAvroSchema

public static void setAvroSchema(HoodieWriteConfig writeConfig,
                                 HoodieTableMetaClient metaClient)
                          throws Exception

Sets up the avro schema string into the HoodieWriteConfig HoodieWriteConfig through reading from the hoodie table metadata.

Parameters:

writeConfig - The HoodieWriteConfig

Throws:

Exception

setPreCombineField

public static void setPreCombineField(org.apache.flink.configuration.Configuration conf,
                                      HoodieTableMetaClient metaClient)

Sets up the preCombine field into the given configuration conf through reading from the hoodie table metadata.

This value is non-null as compaction can only be performed on MOR tables. Of which, MOR tables will have non-null precombine fields.

Parameters:

conf - The configuration

inferChangelogMode

public static void inferChangelogMode(org.apache.flink.configuration.Configuration conf,
                                      HoodieTableMetaClient metaClient)

Infers the changelog mode based on the data file schema(including metadata fields).

We can improve the code if the changelog mode is set up as table config.

Parameters:

conf - The configuration

metaClient - The meta client

inferMetadataConf

public static void inferMetadataConf(org.apache.flink.configuration.Configuration conf,
                                     HoodieTableMetaClient metaClient)

Infers the metadata config based on the existence of metadata folder.

We can improve the code if the metadata config is set up as table config.

Parameters:

conf - The configuration

metaClient - The meta client

rollbackCompaction

public static void rollbackCompaction(HoodieFlinkTable<?> table,
                                      String instantTime)

rollbackCompaction

public static void rollbackCompaction(HoodieFlinkTable<?> table)

Force rolls back all the inflight compaction instants, especially for job failover restart.

Parameters:

table - The hoodie table

rollbackEarliestCompaction

public static void rollbackEarliestCompaction(HoodieFlinkTable<?> table,
                                              org.apache.flink.configuration.Configuration conf)

Rolls back the earliest compaction if there exists.

Makes the strategy not that radical: firstly check whether there exists inflight compaction instants, rolls back the first inflight instant only if it has timed out. That means, if there are multiple timed out instants on the timeline, we only roll back the first one at a time.

isLIFO

public static boolean isLIFO(String seq)

Returns whether the execution sequence is LIFO.