org.apache.gobblin.service.modules.orchestration

Interface DagStateStore

All Known Implementing Classes:

FSDagStateStore, MysqlDagStateStore

@Alpha
public interface DagStateStore

An interface for storing and retrieving currently running Dags. In case of a leadership change in the GobblinServiceManager, the corresponding DagManager loads the running Dags from the DagStateStore to resume their execution.

Method Summary

Modifier and Type	Method and Description
void	cleanUp(Dag<JobExecutionPlan> dag) Delete the Dag from the backing store, typically upon completion of execution.
List<Dag<JobExecutionPlan>>	getDags() Load all currently running Dags from the underlying store.
void	writeCheckpoint(Dag<JobExecutionPlan> dag) Persist the Dag to the backing store.

Method Detail

writeCheckpoint

void writeCheckpoint(Dag<JobExecutionPlan> dag)
              throws IOException

Persist the Dag to the backing store. This is not an actual checkpoint but more like a Write-ahead log, where uncommitted job will be persisted and be picked up again when leader transition happens.

Parameters:

dag - The dag submitted to DagManager

Throws:

IOException

cleanUp

void cleanUp(Dag<JobExecutionPlan> dag)
      throws IOException

Delete the Dag from the backing store, typically upon completion of execution.

Parameters:

dag - The dag completed/cancelled from execution on SpecExecutor.

Throws:

IOException

getDags

List<Dag<JobExecutionPlan>> getDags()
                             throws IOException

Load all currently running Dags from the underlying store. Typically, invoked when a new DagManager takes over or on restart of service.

Returns:

a List of currently running Dags.

Throws:

IOException