org.apache.hudi.sink.partitioner.profile

Class WriteProfile

java.lang.Object

org.apache.hudi.sink.partitioner.profile.WriteProfile

Direct Known Subclasses:

DeltaWriteProfile, EmptyWriteProfile

public class WriteProfile
extends Object

Profiling of write statistics for BucketAssigner, such as the average record size and small files.

The profile is re-constructed when there are new commits on the timeline.

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieWriteConfig	config The write config.
protected SyncableFileSystemView	fsView The file system view cache for one checkpoint interval.
protected HoodieTableMetaClient	metaClient The meta client.

Constructor Summary

Constructors

Constructor and Description
WriteProfile(HoodieWriteConfig config, HoodieFlinkEngineContext context)

Method Summary

Modifier and Type	Method and Description
long	getAvgSize()
protected SyncableFileSystemView	getFileSystemView()
HoodieTableMetaClient	getMetaClient()
Map<String,HoodieCommitMetadata>	getMetadataCache()
long	getRecordsPerBucket()
List<SmallFile>	getSmallFiles(String partitionPath) Returns a list of small files in the given partition path.
protected HoodieTable<?,?,?,?>	getTable()
void	reload(long checkpointId) Reload the write profile, should do once for each checkpoint.
protected List<SmallFile>	smallFilesProfile(String partitionPath) Returns a list of small files in the given partition path from the latest filesystem view.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected final HoodieWriteConfig config

The write config.

metaClient

protected final HoodieTableMetaClient metaClient

The meta client.

fsView

protected SyncableFileSystemView fsView

The file system view cache for one checkpoint interval.

Constructor Detail

WriteProfile

public WriteProfile(HoodieWriteConfig config,
                    HoodieFlinkEngineContext context)

Method Detail

getAvgSize

public long getAvgSize()

getRecordsPerBucket

public long getRecordsPerBucket()

getMetaClient

public HoodieTableMetaClient getMetaClient()

getTable

protected HoodieTable<?,?,?,?> getTable()

getSmallFiles

public List<SmallFile> getSmallFiles(String partitionPath)

Returns a list of small files in the given partition path.

Note: This method should be thread safe.

smallFilesProfile

protected List<SmallFile> smallFilesProfile(String partitionPath)

Returns a list of small files in the given partition path from the latest filesystem view.

getFileSystemView

protected SyncableFileSystemView getFileSystemView()

reload

public void reload(long checkpointId)

Reload the write profile, should do once for each checkpoint.

We do these things: i). reload the timeline; ii). re-construct the record profile; iii) clean the small files cache.

Note: This method should be thread safe.

getMetadataCache

@VisibleForTesting
public Map<String,HoodieCommitMetadata> getMetadataCache()