org.apache.hudi.sink.common

Class AbstractStreamWriteFunction<I>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<I,Object>

org.apache.hudi.sink.common.AbstractWriteFunction<I>

org.apache.hudi.sink.common.AbstractStreamWriteFunction<I>

Type Parameters:

I - Type of the input record

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.operators.BoundedOneInput

Direct Known Subclasses:

AppendWriteFunction, StreamWriteFunction

public abstract class AbstractStreamWriteFunction<I>
extends AbstractWriteFunction<I>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Base infrastructures for streaming writer function.

See Also:

StreamWriteOperatorCoordinator, Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.configuration.Configuration	config Config options.
protected boolean	confirming Flag saying whether the write task is waiting for the checkpoint success notification after it finished a checkpoint.
protected String	currentInstant The REQUESTED instant we write the data.
protected org.apache.flink.runtime.operators.coordination.OperatorEventGateway	eventGateway Gateway to send operator events to the operator coordinator.
protected HoodieTableMetaClient	metaClient Meta Client.
protected int	taskID Id of current subtask.
protected HoodieFlinkWriteClient	writeClient Write Client.
protected List<WriteStatus>	writeStatuses Write status list for the current checkpoint.

Constructor Summary

Constructors

Constructor and Description
AbstractStreamWriteFunction(org.apache.flink.configuration.Configuration config) Constructs a StreamWriteFunctionBase.

Method Summary

Modifier and Type	Method and Description
void	endInput() Invoked when bounded source ends up.
void	handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event) Handles the operator event sent by the coordinator.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
protected String	instantToWrite(boolean hasData) Prepares the instant time to write with for next checkpoint.
boolean	isConfirming()
protected String	lastPendingInstant() Returns the last pending instant time.
void	setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway) Sets up the event gateway.
abstract void	snapshotState()
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer, processElement

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected final org.apache.flink.configuration.Configuration config

Config options.

taskID

protected int taskID

Id of current subtask.

metaClient

protected transient HoodieTableMetaClient metaClient

Meta Client.

writeClient

protected transient HoodieFlinkWriteClient writeClient

Write Client.

currentInstant

protected volatile String currentInstant

The REQUESTED instant we write the data.

eventGateway

protected transient org.apache.flink.runtime.operators.coordination.OperatorEventGateway eventGateway

Gateway to send operator events to the operator coordinator.

confirming

protected volatile boolean confirming

Flag saying whether the write task is waiting for the checkpoint success notification after it finished a checkpoint.

The flag is needed because the write task does not block during the waiting time interval, some data buckets still flush out with old instant time. There are two cases that the flush may produce corrupted files if the old instant is committed successfully: 1) the write handle was writing data but interrupted, left a corrupted parquet file; 2) the write handle finished the write but was not closed, left an empty parquet file.

To solve, when this flag was set to true, we block the data flushing thus the #processElement method, the flag was reset to false if the task receives the checkpoint success event or the latest inflight instant time changed(the last instant committed successfully).

writeStatuses

protected List<WriteStatus> writeStatuses

Write status list for the current checkpoint.

Constructor Detail

AbstractStreamWriteFunction

public AbstractStreamWriteFunction(org.apache.flink.configuration.Configuration config)

Constructs a StreamWriteFunctionBase.

Parameters:

config - The config options

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public abstract void snapshotState()

endInput

public void endInput()

Description copied from class: AbstractWriteFunction

Invoked when bounded source ends up.

Specified by:

endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput

Specified by:

endInput in class AbstractWriteFunction<I>

isConfirming

@VisibleForTesting
public boolean isConfirming()

setOperatorEventGateway

public void setOperatorEventGateway(org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway)

Description copied from class: AbstractWriteFunction

Sets up the event gateway.

Specified by:

setOperatorEventGateway in class AbstractWriteFunction<I>

handleOperatorEvent

public void handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)

Description copied from class: AbstractWriteFunction

Handles the operator event sent by the coordinator.

Specified by:

handleOperatorEvent in class AbstractWriteFunction<I>

Parameters:

event - The event

lastPendingInstant

protected String lastPendingInstant()

Returns the last pending instant time.

instantToWrite

protected String instantToWrite(boolean hasData)

Prepares the instant time to write with for next checkpoint.

Parameters:

hasData - Whether the task has buffering data

Returns:

The instant time