org.apache.hudi.sink.meta

Class CkpMetadata

java.lang.Object

org.apache.hudi.sink.meta.CkpMetadata

All Implemented Interfaces:

Serializable, AutoCloseable

public class CkpMetadata
extends Object
implements Serializable, AutoCloseable

The checkpoint metadata for bookkeeping the checkpoint messages.

Each time the driver starts a new instant, it writes a commit message into the metadata, the write tasks then consume the message and unblock the data flushing.

Why we use the DFS based message queue instead of sending the OperatorEvent ? The writer task thread handles the operator event using the main mailbox executor which has the lowest priority for mails, it is also used to process the inputs. When the writer task blocks and waits for the operator event to ack the valid instant to write, it actually blocks all the subsequent events in the mailbox, the operator event would never be consumed then it causes deadlock.

The checkpoint metadata is also more lightweight than the active timeline.

NOTE: should be removed in the future if we have good manner to handle the async notifications from driver.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.fs.Path	path

Method Summary

Modifier and Type	Method and Description
void	abortInstant(String instant) Add an aborted checkpoint message.
void	bootstrap() Initialize the message bus, would clean all the messages
protected static String	ckpMetaPath(String basePath, String uniqueId)
void	close()
void	commitInstant(String instant) Add a checkpoint commit message.
static CkpMetadata	getInstance(org.apache.flink.configuration.Configuration config)
static CkpMetadata	getInstance(org.apache.hadoop.fs.FileSystem fs, String basePath, String uniqueId)
static CkpMetadata	getInstance(HoodieTableMetaClient metaClient, String uniqueId)
List<String>	getInstantCache()
List<CkpMessage>	getMessages()
boolean	isAborted(String instant)
String	lastPendingInstant()
void	startInstant(String instant)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

path

protected final org.apache.hadoop.fs.Path path

Method Detail

close

public void close()

Specified by:

close in interface AutoCloseable

bootstrap

public void bootstrap()
               throws IOException

Initialize the message bus, would clean all the messages

This expects to be called by the driver.

Throws:

IOException

startInstant

public void startInstant(String instant)

commitInstant

public void commitInstant(String instant)

Add a checkpoint commit message.

Parameters:

instant - The committed instant

abortInstant

public void abortInstant(String instant)

Add an aborted checkpoint message.

lastPendingInstant

@Nullable
public String lastPendingInstant()

getMessages

public List<CkpMessage> getMessages()

isAborted

public boolean isAborted(String instant)

getInstantCache

public List<String> getInstantCache()

getInstance

public static CkpMetadata getInstance(org.apache.flink.configuration.Configuration config)

getInstance

public static CkpMetadata getInstance(HoodieTableMetaClient metaClient,
                                      String uniqueId)

getInstance

public static CkpMetadata getInstance(org.apache.hadoop.fs.FileSystem fs,
                                      String basePath,
                                      String uniqueId)

ckpMetaPath

protected static String ckpMetaPath(String basePath,
                                    String uniqueId)