org.apache.hudi.sink.append

Class AppendWriteFunction<I>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.common.AbstractStreamWriteFunction<I>

org.apache.hudi.sink.append.AppendWriteFunction<I>

Type Parameters:

I - Type of the input record

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

public class AppendWriteFunction<I>
extends AbstractStreamWriteFunction<I>

Sink function to write the data to the underneath filesystem.

The function writes base files directly for each checkpoint, the file may roll over when it’s size hits the configured threshold.

See Also:

StreamWriteOperatorCoordinator, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

config, confirming, currentInstant, eventGateway, metaClient, taskID, writeClient, writeStatuses

Constructor Summary

Constructors

Constructor and Description
AppendWriteFunction(org.apache.flink.configuration.Configuration config, org.apache.flink.table.types.logical.RowType rowType) Constructs an AppendWriteFunction.

Method Summary

Modifier and Type	Method and Description
void	endInput() End input action for batch source.
BulkInsertWriterHelper	getWriterHelper()
void	processElement(I value, org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx, org.apache.flink.util.Collector<Object> out)
protected void	sendBootstrapEvent()
void	snapshotState()

Methods inherited from class org.apache.hudi.sink.common.AbstractStreamWriteFunction

handleOperatorEvent, initializeState, instantToWrite, isConfirming, lastPendingInstant, setOperatorEventGateway, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AppendWriteFunction

public AppendWriteFunction(org.apache.flink.configuration.Configuration config,
                           org.apache.flink.table.types.logical.RowType rowType)

Constructs an AppendWriteFunction.

Parameters:

config - The config options

Method Detail

snapshotState

public void snapshotState()

Specified by:

snapshotState in class AbstractStreamWriteFunction<I>

processElement

public void processElement(I value,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx,
                           org.apache.flink.util.Collector<Object> out)
                    throws Exception

Specified by:

processElement in class org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

Throws:

Exception

endInput

public void endInput()

End input action for batch source.

Specified by:

endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput

Overrides:

endInput in class AbstractStreamWriteFunction<I>

sendBootstrapEvent

protected void sendBootstrapEvent()

Overrides:

sendBootstrapEvent in class AbstractStreamWriteFunction<I>

getWriterHelper

@VisibleForTesting
public BulkInsertWriterHelper getWriterHelper()