org.apache.hudi.sink.partitioner

Class BucketAssigner

java.lang.Object

org.apache.hudi.sink.partitioner.BucketAssigner

All Implemented Interfaces:

AutoCloseable

public class BucketAssigner
extends Object
implements AutoCloseable

Bucket assigner that assigns the data buffer of one checkpoint into buckets.

This assigner assigns the record one by one. If the record is an update, checks and reuse existing UPDATE bucket or generates a new one; If the record is an insert, checks the record partition for small files first, try to find a small file that has space to append new records and reuse the small file's data bucket, if there is no small file(or no left space for new records), generates an INSERT bucket.

Use {partition}_{fileId} as the bucket identifier, so that the bucket is unique within and among partitions.

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieWriteConfig	config The write config.

Constructor Summary

Constructors

Constructor and Description
BucketAssigner(int taskID, int maxParallelism, int numTasks, WriteProfile profile, HoodieWriteConfig config)

Method Summary

Modifier and Type	Method and Description
BucketInfo	addInsert(String partitionPath)
BucketInfo	addUpdate(String partitionPath, String fileIdHint)
void	close()
String	createFileIdOfThisTask()
void	reload(long checkpointId) Refresh the table state like TableFileSystemView and HoodieTimeline.
void	reset() Reset the states of this assigner, should do once for each checkpoint, all the states are accumulated within one checkpoint interval.
List<SmallFile>	smallFilesOfThisTask(List<SmallFile> smallFiles)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected final HoodieWriteConfig config

The write config.

Constructor Detail

BucketAssigner

public BucketAssigner(int taskID,
                      int maxParallelism,
                      int numTasks,
                      WriteProfile profile,
                      HoodieWriteConfig config)

Method Detail

reset

public void reset()

Reset the states of this assigner, should do once for each checkpoint, all the states are accumulated within one checkpoint interval.

addUpdate

public BucketInfo addUpdate(String partitionPath,
                            String fileIdHint)

addInsert

public BucketInfo addInsert(String partitionPath)

reload

public void reload(long checkpointId)

Refresh the table state like TableFileSystemView and HoodieTimeline.

createFileIdOfThisTask

@VisibleForTesting
public String createFileIdOfThisTask()

smallFilesOfThisTask

@VisibleForTesting
public List<SmallFile> smallFilesOfThisTask(List<SmallFile> smallFiles)

close

public void close()

Specified by:

close in interface AutoCloseable