org.apache.flink.runtime.checkpoint

Class ZooKeeperCompletedCheckpointStore

java.lang.Object

org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore

All Implemented Interfaces:

CompletedCheckpointStore

public class ZooKeeperCompletedCheckpointStore
extends Object
implements CompletedCheckpointStore

CompletedCheckpointStore for JobManagers running in RecoveryMode.ZOOKEEPER.

Checkpoints are added under a ZNode per job:

 +----O /flink/checkpoints/<job-id>  [persistent]
 .    |
 .    +----O /flink/checkpoints/<job-id>/1 [persistent]
 .    .                                  .
 .    .                                  .
 .    .                                  .
 .    +----O /flink/checkpoints/<job-id>/N [persistent]
 

During recovery, the latest checkpoint is read from ZooKeeper. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).

If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.

Constructor Summary

Constructors

Constructor and Description
ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, ClassLoader userClassLoader, org.apache.curator.framework.CuratorFramework client, String checkpointsPath, StateStorageHelper<CompletedCheckpoint> stateStorage) Creates a ZooKeeperCompletedCheckpointStore instance.

Method Summary

Modifier and Type	Method and Description
void	addCheckpoint(CompletedCheckpoint checkpoint) Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
void	discardAllCheckpoints() Discards all added CompletedCheckpoint instances via CompletedCheckpoint.discard(ClassLoader).
List<CompletedCheckpoint>	getAllCheckpoints() Returns all CompletedCheckpoint instances.
CompletedCheckpoint	getLatestCheckpoint() Returns the latest CompletedCheckpoint instance or null if none was added.
int	getNumberOfRetainedCheckpoints() Returns the current number of retained checkpoints.
void	recover() Gets the latest checkpoint from ZooKeeper and removes all others.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ZooKeeperCompletedCheckpointStore

public ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
                                         ClassLoader userClassLoader,
                                         org.apache.curator.framework.CuratorFramework client,
                                         String checkpointsPath,
                                         StateStorageHelper<CompletedCheckpoint> stateStorage)
                                  throws Exception

Creates a ZooKeeperCompletedCheckpointStore instance.

Parameters:

maxNumberOfCheckpointsToRetain - The maximum number of checkpoints to retain (at least 1). Adding more checkpoints than this results in older checkpoints being discarded. On recovery, we will only start with a single checkpoint.

userClassLoader - The user class loader used to discard checkpoints

client - The Curator ZooKeeper client

checkpointsPath - The ZooKeeper path for the checkpoints (needs to start with a '/')

stateStorage - State storage to be used to persist the completed checkpoint

Throws:

Exception

Method Detail

recover

public void recover()
             throws Exception

Gets the latest checkpoint from ZooKeeper and removes all others.

Important: Even if there are more than one checkpoint in ZooKeeper, this will only recover the latest and discard the others. Otherwise, there is no guarantee that the history of checkpoints is consistent.

Specified by:

recover in interface CompletedCheckpointStore

Throws:

Exception

addCheckpoint

public void addCheckpoint(CompletedCheckpoint checkpoint)
                   throws Exception

Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.

Specified by:

addCheckpoint in interface CompletedCheckpointStore

Parameters:

checkpoint - Completed checkpoint to add.

Throws:

Exception

getLatestCheckpoint

public CompletedCheckpoint getLatestCheckpoint()
                                        throws Exception

Description copied from interface: CompletedCheckpointStore

Returns the latest CompletedCheckpoint instance or null if none was added.

Specified by:

getLatestCheckpoint in interface CompletedCheckpointStore

Throws:

Exception

getAllCheckpoints

public List<CompletedCheckpoint> getAllCheckpoints()
                                            throws Exception

Description copied from interface: CompletedCheckpointStore

Returns all CompletedCheckpoint instances.

Returns an empty list if no checkpoint has been added yet.

Specified by:

getAllCheckpoints in interface CompletedCheckpointStore

Throws:

Exception

getNumberOfRetainedCheckpoints

public int getNumberOfRetainedCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns the current number of retained checkpoints.

Specified by:

getNumberOfRetainedCheckpoints in interface CompletedCheckpointStore

discardAllCheckpoints

public void discardAllCheckpoints()
                           throws Exception

Description copied from interface: CompletedCheckpointStore

Discards all added CompletedCheckpoint instances via CompletedCheckpoint.discard(ClassLoader).

Specified by:

discardAllCheckpoints in interface CompletedCheckpointStore

Throws:

Exception