org.apache.flink.runtime.checkpoint

Class ZooKeeperCompletedCheckpointStore

java.lang.Object

org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore

All Implemented Interfaces:

CompletedCheckpointStore

public class ZooKeeperCompletedCheckpointStore
extends Object
implements CompletedCheckpointStore

CompletedCheckpointStore for JobManagers running in HighAvailabilityMode.ZOOKEEPER.

Checkpoints are added under a ZNode per job:

 +----O /flink/checkpoints/<job-id>  [persistent]
 .    |
 .    +----O /flink/checkpoints/<job-id>/1 [persistent]
 .    .                                  .
 .    .                                  .
 .    .                                  .
 .    +----O /flink/checkpoints/<job-id>/N [persistent]
 

During recovery, the latest checkpoint is read from ZooKeeper. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).

If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.

Constructor Summary

Constructors

Constructor and Description
ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, ZooKeeperStateHandleStore<CompletedCheckpoint> checkpointsInZooKeeper, Executor executor) Creates a ZooKeeperCompletedCheckpointStore instance.

Method Summary

Modifier and Type	Method and Description
void	addCheckpoint(CompletedCheckpoint checkpoint) Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
static String	checkpointIdToPath(long checkpointId) Convert a checkpoint id into a ZooKeeper path.
List<CompletedCheckpoint>	getAllCheckpoints() Returns all CompletedCheckpoint instances.
int	getMaxNumberOfRetainedCheckpoints() Returns the max number of retained checkpoints.
int	getNumberOfRetainedCheckpoints() Returns the current number of retained checkpoints.
static long	pathToCheckpointId(String path) Converts a path to the checkpoint id.
void	recover() Gets the latest checkpoint from ZooKeeper and removes all others.
boolean	requiresExternalizedCheckpoints() This method returns whether the completed checkpoint store requires checkpoints to be externalized.
void	shutdown(org.apache.flink.api.common.JobStatus jobStatus) Shuts down the store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.checkpoint.CompletedCheckpointStore

getLatestCheckpoint

Constructor Detail

ZooKeeperCompletedCheckpointStore

public ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
                                         ZooKeeperStateHandleStore<CompletedCheckpoint> checkpointsInZooKeeper,
                                         Executor executor)

Creates a ZooKeeperCompletedCheckpointStore instance.

Parameters:

maxNumberOfCheckpointsToRetain - The maximum number of checkpoints to retain (at least 1). Adding more checkpoints than this results in older checkpoints being discarded. On recovery, we will only start with a single checkpoint.

checkpointsInZooKeeper - Completed checkpoints in ZooKeeper

executor - to execute blocking calls

Method Detail

requiresExternalizedCheckpoints

public boolean requiresExternalizedCheckpoints()

Description copied from interface: CompletedCheckpointStore

This method returns whether the completed checkpoint store requires checkpoints to be externalized. Externalized checkpoints have their meta data persisted, which the checkpoint store can exploit (for example by simply pointing the persisted metadata).

Specified by:

requiresExternalizedCheckpoints in interface CompletedCheckpointStore

Returns:

True, if the store requires that checkpoints are externalized before being added, false if the store stores the metadata itself.

recover

public void recover()
             throws Exception

Gets the latest checkpoint from ZooKeeper and removes all others.

Important: Even if there are more than one checkpoint in ZooKeeper, this will only recover the latest and discard the others. Otherwise, there is no guarantee that the history of checkpoints is consistent.

Specified by:

recover in interface CompletedCheckpointStore

Throws:

Exception

addCheckpoint

public void addCheckpoint(CompletedCheckpoint checkpoint)
                   throws Exception

Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.

Specified by:

addCheckpoint in interface CompletedCheckpointStore

Parameters:

checkpoint - Completed checkpoint to add.

Throws:

Exception

getAllCheckpoints

public List<CompletedCheckpoint> getAllCheckpoints()
                                            throws Exception

Description copied from interface: CompletedCheckpointStore

Returns all CompletedCheckpoint instances.

Returns an empty list if no checkpoint has been added yet.

Specified by:

getAllCheckpoints in interface CompletedCheckpointStore

Throws:

Exception

getNumberOfRetainedCheckpoints

public int getNumberOfRetainedCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns the current number of retained checkpoints.

Specified by:

getNumberOfRetainedCheckpoints in interface CompletedCheckpointStore

getMaxNumberOfRetainedCheckpoints

public int getMaxNumberOfRetainedCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns the max number of retained checkpoints.

Specified by:

getMaxNumberOfRetainedCheckpoints in interface CompletedCheckpointStore

shutdown

public void shutdown(org.apache.flink.api.common.JobStatus jobStatus)
              throws Exception

Description copied from interface: CompletedCheckpointStore

Shuts down the store.

The job status is forwarded and used to decide whether state should actually be discarded or kept.

Specified by:

shutdown in interface CompletedCheckpointStore

Parameters:

jobStatus - Job state on shut down

Throws:

Exception

checkpointIdToPath

public static String checkpointIdToPath(long checkpointId)

Convert a checkpoint id into a ZooKeeper path.

Parameters:

checkpointId - to convert to the path

Returns:

Path created from the given checkpoint id

pathToCheckpointId

public static long pathToCheckpointId(String path)

Converts a path to the checkpoint id.

Parameters:

path - in ZooKeeper

Returns:

Checkpoint id parsed from the path