org.apache.hudi.sink.bulk

Class BulkInsertWriteFunction<I>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.bulk.BulkInsertWriteFunction<I>

Type Parameters:

I - Type of the input record

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

public class BulkInsertWriteFunction<I>
extends AbstractWriteFunction<I>

Sink function to write the data to the underneath filesystem.

The function should only be used in operation type WriteOperationType.BULK_INSERT.

Note: The function task requires the input stream be shuffled by partition path.

See Also:

StreamWriteOperatorCoordinator, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Constructor Summary

Constructors

Constructor and Description
BulkInsertWriteFunction(org.apache.flink.configuration.Configuration config, org.apache.flink.table.types.logical.RowType rowType) Constructs a StreamingSinkFunction.

Method Summary

Modifier and Type	Method and Description
void	close()
void	endInput() End input action for batch source.
void	handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event) Handles the operator event sent by the coordinator.
protected String	lastPendingInstant() Returns the last pending instant time.
void	open(org.apache.flink.configuration.Configuration parameters)
void	processElement(I value, org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx, org.apache.flink.util.Collector<Object> out)
void	setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway) Sets up the event gateway.

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkInsertWriteFunction

public BulkInsertWriteFunction(org.apache.flink.configuration.Configuration config,
                               org.apache.flink.table.types.logical.RowType rowType)

Constructs a StreamingSinkFunction.

Parameters:

config - The config options

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

IOException

processElement

public void processElement(I value,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context ctx,
                           org.apache.flink.util.Collector<Object> out)
                    throws IOException

Specified by:

processElement in class org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

Throws:

IOException

close

public void close()

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

endInput

public void endInput()

End input action for batch source.

Specified by:

endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput

Specified by:

endInput in class AbstractWriteFunction<I>

handleOperatorEvent

public void handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)

Description copied from class: AbstractWriteFunction

Handles the operator event sent by the coordinator.

Specified by:

handleOperatorEvent in class AbstractWriteFunction<I>

Parameters:

event - The event

setOperatorEventGateway

public void setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway)

Description copied from class: AbstractWriteFunction

Sets up the event gateway.

Specified by:

setOperatorEventGateway in class AbstractWriteFunction<I>

lastPendingInstant

protected String lastPendingInstant()

Returns the last pending instant time.