org.apache.hudi.sink.partitioner.profile

Class DeltaWriteProfile

java.lang.Object

org.apache.hudi.sink.partitioner.profile.WriteProfile

org.apache.hudi.sink.partitioner.profile.DeltaWriteProfile

public class DeltaWriteProfile
extends WriteProfile

WriteProfile for MERGE_ON_READ table type, this allows auto correction of small parquet files to larger ones without the need for an index in the logFile.

Note: assumes the index can always index log files for Flink write.

Field Summary

Fields inherited from class org.apache.hudi.sink.partitioner.profile.WriteProfile

config, fsView, metaClient

Constructor Summary

Constructors

Constructor and Description
DeltaWriteProfile(HoodieWriteConfig config, HoodieFlinkEngineContext context)

Method Summary

Modifier and Type	Method and Description
long	convertLogFilesSizeToExpectedParquetSize(List<HoodieLogFile> hoodieLogFiles)
protected SyncableFileSystemView	getFileSystemView()
protected List<SmallFile>	smallFilesProfile(String partitionPath) Returns a list of small files in the given partition path from the latest filesystem view.

Methods inherited from class org.apache.hudi.sink.partitioner.profile.WriteProfile

getAvgSize, getMetaClient, getMetadataCache, getRecordsPerBucket, getSmallFiles, getTable, reload

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeltaWriteProfile

public DeltaWriteProfile(HoodieWriteConfig config,
                         HoodieFlinkEngineContext context)

Method Detail

smallFilesProfile

protected List<SmallFile> smallFilesProfile(String partitionPath)

Description copied from class: WriteProfile

Returns a list of small files in the given partition path from the latest filesystem view.

Overrides:

smallFilesProfile in class WriteProfile

getFileSystemView

protected SyncableFileSystemView getFileSystemView()

Overrides:

getFileSystemView in class WriteProfile

convertLogFilesSizeToExpectedParquetSize

public long convertLogFilesSizeToExpectedParquetSize(List<HoodieLogFile> hoodieLogFiles)