org.apache.hudi.sink.meta

Class CkpMetadata

java.lang.Object

org.apache.hudi.sink.meta.CkpMetadata

All Implemented Interfaces:

Serializable

public class CkpMetadata
extends Object
implements Serializable

The checkpoint metadata for bookkeeping the checkpoint messages.

Each time the driver starts a new instant, it writes a commit message into the metadata, the write tasks then consume the message and unblock the data flushing.

Why we use the DFS based message queue instead of sending the OperatorEvent ? The write task handles the operator event using the main mailbox executor which has the lowest priority for mails, it is also used to process the inputs. When the write task blocks and waits for the operator event to ack the valid instant to write, it actually blocks all the subsequent events in the mailbox, the operator event would never be consumed then it causes deadlock.

The checkpoint metadata is also more lightweight than the active timeline.

NOTE: should be removed in the future if we have good manner to handle the async notifications from driver.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	MAX_RETAIN_CKP_NUM
protected org.apache.hadoop.fs.Path	path

Method Summary

Modifier and Type	Method and Description
void	abortInstant(String instant) Add an aborted checkpoint message.
void	bootstrap(HoodieTableMetaClient metaClient) Initialize the message bus, would clean all the messages and publish the last pending instant.
protected static String	ckpMetaPath(String basePath)
void	close()
void	commitInstant(String instant) Add a checkpoint commit message.
static CkpMetadata	getInstance(org.apache.flink.configuration.Configuration config)
static CkpMetadata	getInstance(org.apache.hadoop.fs.FileSystem fs, String basePath)
List<CkpMessage>	getMessages()
boolean	isAborted(String instant)
String	lastPendingInstant()
void	startInstant(String instant)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_RETAIN_CKP_NUM

protected static final int MAX_RETAIN_CKP_NUM

See Also:

Constant Field Values

path

protected final org.apache.hadoop.fs.Path path

Method Detail

close

public void close()

bootstrap

public void bootstrap(HoodieTableMetaClient metaClient)
               throws IOException

Initialize the message bus, would clean all the messages and publish the last pending instant.

This expects to be called by the driver.

Throws:

IOException

startInstant

public void startInstant(String instant)

commitInstant

public void commitInstant(String instant)

Add a checkpoint commit message.

Parameters:

instant - The committed instant

abortInstant

public void abortInstant(String instant)

Add an aborted checkpoint message.

lastPendingInstant

@Nullable
public String lastPendingInstant()

getMessages

public List<CkpMessage> getMessages()

isAborted

public boolean isAborted(String instant)

getInstance

public static CkpMetadata getInstance(org.apache.flink.configuration.Configuration config)

getInstance

public static CkpMetadata getInstance(org.apache.hadoop.fs.FileSystem fs,
                                      String basePath)

ckpMetaPath

protected static String ckpMetaPath(String basePath)