org.apache.hudi.sink.bucket

Class BucketStreamWriteFunction<I>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.common.AbstractStreamWriteFunction<I>

org.apache.hudi.sink.StreamWriteFunction<I>

org.apache.hudi.sink.bucket.BucketStreamWriteFunction<I>

Type Parameters:

I - the input type

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

public class BucketStreamWriteFunction<I>
extends StreamWriteFunction<I>

A stream write function with simple bucket hash index.

The task holds a fresh new local index: {(partition + bucket number) &rarr fileId} mapping, this index is used for deciding whether the incoming records in an UPDATE or INSERT. The index is local because different partition paths have separate items in the index.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.sink.StreamWriteFunction

StreamWriteFunction.DataBucket

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields inherited from class org.apache.hudi.sink.StreamWriteFunction

writeFunction

Fields inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

config, confirming, currentInstant, eventGateway, metaClient, taskID, writeClient, writeStatuses

Constructor Summary

Constructors

Constructor and Description
BucketStreamWriteFunction(org.apache.flink.configuration.Configuration config) Constructs a BucketStreamWriteFunction.

Method Summary

Modifier and Type	Method and Description
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
boolean	isBucketToLoad(int bucketNumber, String partition) Determine whether the current fileID belongs to the current task.
void	open(org.apache.flink.configuration.Configuration parameters)
void	processElement(I i, org.apache.flink.streaming.api.functions.ProcessFunction.Context context, org.apache.flink.util.Collector<Object> collector)
void	snapshotState()

Methods inherited from class org.apache.hudi.sink.StreamWriteFunction

bufferRecord, close, endInput, getDataBuffer, writeBucket

Methods inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

handleOperatorEvent, instantToWrite, isConfirming, lastPendingInstant, sendBootstrapEvent, setOperatorEventGateway, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BucketStreamWriteFunction

public BucketStreamWriteFunction(org.apache.flink.configuration.Configuration config)

Constructs a BucketStreamWriteFunction.

Parameters:

config - The config options

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class StreamWriteFunction<I>

Throws:

IOException

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Overrides:

initializeState in class AbstractStreamWriteFunction<I>

Throws:

Exception

snapshotState

public void snapshotState()

Overrides:

snapshotState in class StreamWriteFunction<I>

processElement

public void processElement(I i,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context context,
                           org.apache.flink.util.Collector<Object> collector)
                    throws Exception

Overrides:

processElement in class StreamWriteFunction<I>

Throws:

Exception

isBucketToLoad

public boolean isBucketToLoad(int bucketNumber,
                              String partition)

Determine whether the current fileID belongs to the current task. (partition + curBucket) % numPartitions == this taskID belongs to this task.