org.apache.hudi.sink.partitioner

Class BucketAssignFunction<K,I,O extends HoodieRecord<?>>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.KeyedProcessFunction<K,I,O>

org.apache.hudi.sink.partitioner.BucketAssignFunction<K,I,O>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

public class BucketAssignFunction<K,I,O extends HoodieRecord<?>>
extends org.apache.flink.streaming.api.functions.KeyedProcessFunction<K,I,O>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.api.common.state.CheckpointListener

The function to build the write profile incrementally for records within a checkpoint, it then assigns the bucket with ID using the BucketAssigner.

All the records are tagged with HoodieRecordLocation, instead of real instant time, INSERT record uses "I" and UPSERT record uses "U" as instant time. There is no need to keep the "real" instant time for each record, the bucket ID (partition path & fileID) actually decides where the record should write to. The "I" and "U" tags are only used for downstream to decide whether the data bucket is an INSERT or an UPSERT, we should factor the tags out when the underneath writer supports specifying the bucket type explicitly.

The output records should then shuffle by the bucket ID and thus do scalable write.

See Also:

BucketAssigner, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.KeyedProcessFunction

org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context, org.apache.flink.streaming.api.functions.KeyedProcessFunction.OnTimerContext

Constructor Summary

Constructors

Constructor and Description
BucketAssignFunction(org.apache.flink.configuration.Configuration conf)

Method Summary

Modifier and Type	Method and Description
void	close()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration parameters)
void	processElement(I value, org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context ctx, org.apache.flink.util.Collector<O> out)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.streaming.api.functions.KeyedProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

BucketAssignFunction

public BucketAssignFunction(org.apache.flink.configuration.Configuration conf)

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

processElement

public void processElement(I value,
                           org.apache.flink.streaming.api.functions.KeyedProcessFunction.Context ctx,
                           org.apache.flink.util.Collector<O> out)
                    throws Exception

Specified by:

processElement in class org.apache.flink.streaming.api.functions.KeyedProcessFunction<K,I,O extends HoodieRecord<?>>

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception