org.apache.hudi.sink

Class StreamWriteFunction<I>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.common.AbstractStreamWriteFunction<I>

org.apache.hudi.sink.StreamWriteFunction<I>

Type Parameters:

I - Type of the input record

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

Direct Known Subclasses:

BucketStreamWriteFunction, ConsistentBucketStreamWriteFunction

public class StreamWriteFunction<I>
extends AbstractStreamWriteFunction<I>

Sink function to write the data to the underneath filesystem.

Work Flow

The function firstly buffers the data as a batch of HoodieRecords, It flushes(write) the records batch when the batch size exceeds the configured size FlinkOptions.WRITE_BATCH_SIZE or the total buffer size exceeds the configured size FlinkOptions.WRITE_TASK_MAX_SIZE or a Flink checkpoint starts. After a batch has been written successfully, the function notifies its operator coordinator StreamWriteOperatorCoordinator to mark a successful write.

The Semantics

The task implements exactly-once semantics by buffering the data between checkpoints. The operator coordinator starts a new instant on the timeline when a checkpoint triggers, the coordinator checkpoints always start before its operator, so when this function starts a checkpoint, a REQUESTED instant already exists.

The function process thread blocks data buffering after the checkpoint thread finishes flushing the existing data buffer until the current checkpoint succeed and the coordinator starts a new instant. Any error triggers the job failure during the metadata committing, when the job recovers from a failure, the write function re-send the write metadata to the coordinator to see if these metadata can re-commit, thus if unexpected error happens during the instant committing, the coordinator would retry to commit when the job recovers.

Fault Tolerance

The operator coordinator checks and commits the last instant then starts a new one after a checkpoint finished successfully. It rolls back any inflight instant before it starts a new instant, this means one hoodie instant only span one checkpoint, the write function blocks data buffer flushing for the configured checkpoint timeout before it throws exception, any checkpoint failure would finally trigger the job failure.

Note: The function task requires the input stream be shuffled by the file IDs.

See Also:

StreamWriteOperatorCoordinator, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	StreamWriteFunction.DataBucket Data bucket.

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields

Modifier and Type	Field and Description
protected BiFunction<List<HoodieRecord>,String,List<WriteStatus>>	writeFunction

Fields inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

config, confirming, currentInstant, eventGateway, metaClient, taskID, writeClient, writeStatuses

Constructor Summary

Constructors

Constructor and Description
StreamWriteFunction(org.apache.flink.configuration.Configuration config) Constructs a StreamingSinkFunction.

Method Summary

Modifier and Type	Method and Description
protected void	bufferRecord(HoodieRecord<?> value) Buffers the given record.
void	close()
void	endInput() End input action for batch source.
Map<String,List<HoodieRecord>>	getDataBuffer()
void	open(org.apache.flink.configuration.Configuration parameters)
void	processElement(I value, org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx, org.apache.flink.util.Collector<Object> out)
void	snapshotState()
protected List<WriteStatus>	writeBucket(String instant, StreamWriteFunction.DataBucket bucket, List<HoodieRecord> records)

Methods inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

handleOperatorEvent, initializeState, instantToWrite, isConfirming, lastPendingInstant, sendBootstrapEvent, setOperatorEventGateway, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

writeFunction

protected transient BiFunction<List<HoodieRecord>,String,List<WriteStatus>> writeFunction

Constructor Detail

StreamWriteFunction

public StreamWriteFunction(org.apache.flink.configuration.Configuration config)

Constructs a StreamingSinkFunction.

Parameters:

config - The config options

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

IOException

snapshotState

public void snapshotState()

Specified by:

snapshotState in class AbstractStreamWriteFunction<I>

processElement

public void processElement(I value,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx,
                           org.apache.flink.util.Collector<Object> out)
                    throws Exception

Specified by:

processElement in class org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

Throws:

Exception

close

public void close()

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

endInput

public void endInput()

End input action for batch source.

Specified by:

endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput

Overrides:

endInput in class AbstractStreamWriteFunction<I>

getDataBuffer

@VisibleForTesting
public Map<String,List<HoodieRecord>> getDataBuffer()

bufferRecord

protected void bufferRecord(HoodieRecord<?> value)

Buffers the given record.

Flush the data bucket first if the bucket records size is greater than the configured value FlinkOptions.WRITE_BATCH_SIZE.

Flush the max size data bucket if the total buffer size exceeds the configured threshold FlinkOptions.WRITE_TASK_MAX_SIZE.

Parameters:

value - HoodieRecord

writeBucket

protected List<WriteStatus> writeBucket(String instant,
                                        StreamWriteFunction.DataBucket bucket,
                                        List<HoodieRecord> records)