org.apache.hudi.client.timeline.versioning.v2

Class LSMTimelineWriter

java.lang.Object

org.apache.hudi.client.timeline.versioning.v2.LSMTimelineWriter

public class LSMTimelineWriter
extends Object

A timeline writer which organizes the files as an LSM tree.

Field Summary

Fields

Modifier and Type	Field and Description
static int	FILE_LAYER_ZERO
static long	MAX_FILE_SIZE_IN_BYTES

Method Summary

Modifier and Type	Method and Description
void	clean(HoodieEngineContext context, int compactedVersions) Checks whether there is any unfinished compaction operation.
void	compactAndClean(HoodieEngineContext context) Compacts the small parquet files.
static String	compactedFileName(List<String> files) Returns a new file name.
void	compactFiles(List<String> candidateFiles, String compactedFileName)
static LSMTimelineWriter	getInstance(HoodieWriteConfig config, HoodieTable<?,?,?,?> table)
static LSMTimelineWriter	getInstance(HoodieWriteConfig config, HoodieTable<?,?,?,?> table, Option<StoragePath> archivePath)
static LSMTimelineWriter	getInstance(HoodieWriteConfig config, TaskContextSupplier taskContextSupplier, HoodieTableMetaClient metaClient)
void	updateManifest(List<String> filesToRemove, String fileToAdd) Updates a manifest file.
void	updateManifest(String fileToAdd) Updates a manifest file.
void	write(List<ActiveAction> activeActions, Option<Consumer<ActiveAction>> preWriteCallback, Option<Consumer<Exception>> exceptionHandler) Writes the list of active actions into the timeline.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FILE_LAYER_ZERO

public static final int FILE_LAYER_ZERO

See Also:

Constant Field Values

MAX_FILE_SIZE_IN_BYTES

public static final long MAX_FILE_SIZE_IN_BYTES

See Also:

Constant Field Values

Method Detail

getInstance

public static LSMTimelineWriter getInstance(HoodieWriteConfig config,
                                            HoodieTable<?,?,?,?> table)

getInstance

public static LSMTimelineWriter getInstance(HoodieWriteConfig config,
                                            HoodieTable<?,?,?,?> table,
                                            Option<StoragePath> archivePath)

getInstance

public static LSMTimelineWriter getInstance(HoodieWriteConfig config,
                                            TaskContextSupplier taskContextSupplier,
                                            HoodieTableMetaClient metaClient)

write

public void write(List<ActiveAction> activeActions,
                  Option<Consumer<ActiveAction>> preWriteCallback,
                  Option<Consumer<Exception>> exceptionHandler)
           throws HoodieCommitException

Writes the list of active actions into the timeline.

Parameters:

activeActions - The active actions

preWriteCallback - The callback before writing each action

exceptionHandler - The handle for exception

Throws:

HoodieCommitException

updateManifest

public void updateManifest(String fileToAdd)
                    throws IOException

Updates a manifest file.

3 steps:

1. read the latest manifest version file;
2. read the latest manifest file for valid files;
3. add this new file to the existing file list from step2.

Parameters:

fileToAdd - New file name to add

Throws:

IOException

updateManifest

public void updateManifest(List<String> filesToRemove,
                           String fileToAdd)
                    throws IOException

Updates a manifest file.

4 steps:

1. read the latest manifest version file;
2. read the latest manifest file for valid files;
3. remove files to the existing file list from step2;
4. add this new file to the existing file list from step2.

Parameters:

filesToRemove - File names to remove

fileToAdd - New file name to add

Throws:

IOException

compactAndClean

public void compactAndClean(HoodieEngineContext context)
                     throws IOException

Compacts the small parquet files.

The parquet naming convention is:

${min_instant}_${max_instant}_${level}.parquet

The 'min_instant' and 'max_instant' represent the instant time range of the parquet file. The 'level' represents the number of the level where the file is located, currently we have no limit for the number of layers.

These parquet files composite as an LSM tree layout, one parquet file contains instant metadata entries with consecutive timestamp. Different parquet files may have overlapping with the instant time ranges.

   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
                          \            /
                             \     /
                                |
                                V
                          t3_t6_1.parquet                      L1 layer
 

Compaction and cleaning: once the files number exceed a threshold(now constant 10) N, the oldest N files are then replaced with a compacted file in the next layer. A cleaning action is triggered right after the compaction.

Parameters:

context - HoodieEngineContext

Throws:

IOException

compactFiles

public void compactFiles(List<String> candidateFiles,
                         String compactedFileName)

clean

public void clean(HoodieEngineContext context,
                  int compactedVersions)
           throws IOException

Checks whether there is any unfinished compaction operation.

Parameters:

context - HoodieEngineContext used for parallelize to delete obsolete files if necessary.

Throws:

IOException

compactedFileName

public static String compactedFileName(List<String> files)

Returns a new file name.