org.apache.hudi.sink.partitioner.profile

Class WriteProfiles

java.lang.Object

org.apache.hudi.sink.partitioner.profile.WriteProfiles

public class WriteProfiles
extends Object

Factory for WriteProfile.

Method Summary

Modifier and Type	Method and Description
static void	clean(String path)
static HoodieCommitMetadata	getCommitMetadata(String tableName, org.apache.flink.core.fs.Path basePath, HoodieInstant instant, HoodieTimeline timeline) Returns the commit metadata of the given instant.
static Option<HoodieCommitMetadata>	getCommitMetadataSafely(String tableName, org.apache.flink.core.fs.Path basePath, HoodieInstant instant, HoodieTimeline timeline) Returns the commit metadata of the given instant safely.
static org.apache.hadoop.fs.FileStatus[]	getFilesFromMetadata(org.apache.flink.core.fs.Path basePath, org.apache.hadoop.conf.Configuration hadoopConf, List<HoodieCommitMetadata> metadataList, HoodieTableType tableType) Returns all the incremental write file statuses with the given commits metadata.
static org.apache.hadoop.fs.FileStatus[]	getFilesFromMetadata(org.apache.flink.core.fs.Path basePath, org.apache.hadoop.conf.Configuration hadoopConf, List<HoodieCommitMetadata> metadataList, HoodieTableType tableType, boolean ignoreMissingFiles) Returns all the incremental write file statuses with the given commits metadata.
static WriteProfile	singleton(boolean ignoreSmallFiles, boolean delta, HoodieWriteConfig config, HoodieFlinkEngineContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

singleton

public static WriteProfile singleton(boolean ignoreSmallFiles,
                                     boolean delta,
                                     HoodieWriteConfig config,
                                     HoodieFlinkEngineContext context)

clean

public static void clean(String path)

getFilesFromMetadata

public static org.apache.hadoop.fs.FileStatus[] getFilesFromMetadata(org.apache.flink.core.fs.Path basePath,
                                                                     org.apache.hadoop.conf.Configuration hadoopConf,
                                                                     List<HoodieCommitMetadata> metadataList,
                                                                     HoodieTableType tableType)

Returns all the incremental write file statuses with the given commits metadata. Only existing files are included.

Parameters:

basePath - Table base path

hadoopConf - The hadoop conf

metadataList - The commit metadata list (should in ascending order)

tableType - The table type

Returns:

the file status array

getFilesFromMetadata

@Nullable
public static org.apache.hadoop.fs.FileStatus[] getFilesFromMetadata(org.apache.flink.core.fs.Path basePath,
                                                                               org.apache.hadoop.conf.Configuration hadoopConf,
                                                                               List<HoodieCommitMetadata> metadataList,
                                                                               HoodieTableType tableType,
                                                                               boolean ignoreMissingFiles)

Returns all the incremental write file statuses with the given commits metadata.

Parameters:

basePath - Table base path

hadoopConf - The hadoop conf

metadataList - The commit metadata list (should in ascending order)

tableType - The table type

ignoreMissingFiles - Whether to ignore the missing files from filesystem

Returns:

the file status array or null if any file is missing with ignoreMissingFiles as false

getCommitMetadataSafely

public static Option<HoodieCommitMetadata> getCommitMetadataSafely(String tableName,
                                                                   org.apache.flink.core.fs.Path basePath,
                                                                   HoodieInstant instant,
                                                                   HoodieTimeline timeline)

Returns the commit metadata of the given instant safely.

Parameters:

tableName - The table name

basePath - The table base path

instant - The hoodie instant

timeline - The timeline

Returns:

the commit metadata or empty if any error occurs

getCommitMetadata

public static HoodieCommitMetadata getCommitMetadata(String tableName,
                                                     org.apache.flink.core.fs.Path basePath,
                                                     HoodieInstant instant,
                                                     HoodieTimeline timeline)

Returns the commit metadata of the given instant.

Parameters:

tableName - The table name

basePath - The table base path

instant - The hoodie instant

timeline - The timeline

Returns:

the commit metadata