org.apache.hudi.sink

Class StreamWriteOperatorCoordinator

java.lang.Object

org.apache.hudi.sink.StreamWriteOperatorCoordinator

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.operators.coordination.OperatorCoordinator

public class StreamWriteOperatorCoordinator
extends Object
implements org.apache.flink.runtime.operators.coordination.OperatorCoordinator

OperatorCoordinator for StreamWriteFunction.

This coordinator starts a new instant when a new checkpoint starts. It commits the instant when all the operator tasks write the buffer successfully for a round of checkpoint.

If there is no data for a round of checkpointing, it resets the events buffer and returns early.

See Also:

for the work flow and semantics

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamWriteOperatorCoordinator.Provider Provider for StreamWriteOperatorCoordinator.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context, org.apache.flink.runtime.operators.coordination.OperatorCoordinator.SubtaskGateway

Field Summary

Fields inherited from interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

NO_CHECKPOINT

Constructor Summary

Constructors

Constructor and Description
StreamWriteOperatorCoordinator(org.apache.flink.configuration.Configuration conf, org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context context) Constructs a StreamingSinkOperatorCoordinator.

Method Summary

Modifier and Type	Method and Description
void	checkpointCoordinator(long checkpointId, CompletableFuture<byte[]> result)
void	close()
void	doSyncHive() Sync hoodie table metadata to Hive metastore.
org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context	getContext()
WriteMetadataEvent[]	getEventBuffer()
String	getInstant()
void	handleEventFromOperator(int i, org.apache.flink.runtime.operators.coordination.OperatorEvent operatorEvent)
void	notifyCheckpointComplete(long checkpointId)
void	resetToCheckpoint(long checkpointID, byte[] checkpointData)
void	setExecutor(NonThrownExecutor executor)
void	start()
void	subtaskFailed(int i, @Nullable Throwable throwable)
void	subtaskReady(int i, org.apache.flink.runtime.operators.coordination.OperatorCoordinator.SubtaskGateway subtaskGateway)
void	subtaskReset(int i, long l)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

notifyCheckpointAborted

Constructor Detail

StreamWriteOperatorCoordinator

public StreamWriteOperatorCoordinator(org.apache.flink.configuration.Configuration conf,
                                      org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context context)

Constructs a StreamingSinkOperatorCoordinator.

Parameters:

conf - The config options

context - The coordinator context

Method Detail

start

public void start()
           throws Exception

Specified by:

start in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

Throws:

Exception

checkpointCoordinator

public void checkpointCoordinator(long checkpointId,
                                  CompletableFuture<byte[]> result)

Specified by:

checkpointCoordinator in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

resetToCheckpoint

public void resetToCheckpoint(long checkpointID,
                              byte[] checkpointData)

Specified by:

resetToCheckpoint in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

handleEventFromOperator

public void handleEventFromOperator(int i,
                                    org.apache.flink.runtime.operators.coordination.OperatorEvent operatorEvent)

Specified by:

handleEventFromOperator in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

subtaskFailed

public void subtaskFailed(int i,
                          @Nullable
                          @Nullable Throwable throwable)

Specified by:

subtaskFailed in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

subtaskReset

public void subtaskReset(int i,
                         long l)

Specified by:

subtaskReset in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

subtaskReady

public void subtaskReady(int i,
                         org.apache.flink.runtime.operators.coordination.OperatorCoordinator.SubtaskGateway subtaskGateway)

Specified by:

subtaskReady in interface org.apache.flink.runtime.operators.coordination.OperatorCoordinator

doSyncHive

public void doSyncHive()

Sync hoodie table metadata to Hive metastore.

getEventBuffer

@VisibleForTesting
public WriteMetadataEvent[] getEventBuffer()

getInstant

@VisibleForTesting
public String getInstant()

getContext

@VisibleForTesting
public org.apache.flink.runtime.operators.coordination.OperatorCoordinator.Context getContext()

setExecutor

@VisibleForTesting
public void setExecutor(NonThrownExecutor executor)
                                    throws Exception

Throws:

Exception