org.apache.gobblin.service.modules.orchestration

Class FSDagStateStore

java.lang.Object

org.apache.gobblin.service.modules.orchestration.FSDagStateStore

All Implemented Interfaces:

DagStateStore

@Alpha
public class FSDagStateStore
extends Object
implements DagStateStore

Field Summary

Fields

Modifier and Type	Field and Description
static String	DAG_FILE_EXTENSION

Constructor Summary

Constructors

Constructor and Description
FSDagStateStore(com.typesafe.config.Config config, Map<URI,TopologySpec> topologySpecMap)

Method Summary

Modifier and Type	Method and Description
void	cleanUp(Dag<JobExecutionPlan> dag) Delete the Dag from the backing store, typically upon completion of execution.
Dag<JobExecutionPlan>	getDag(File dagFile) Return a Dag given a file name.
List<Dag<JobExecutionPlan>>	getDags() Load all currently running Dags from the underlying store.
void	writeCheckpoint(Dag<JobExecutionPlan> dag) Persist the Dag to the backing store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DAG_FILE_EXTENSION

public static final String DAG_FILE_EXTENSION

See Also:

Constant Field Values

Constructor Detail

FSDagStateStore

public FSDagStateStore(com.typesafe.config.Config config,
                       Map<URI,TopologySpec> topologySpecMap)
                throws IOException

Throws:

IOException

Method Detail

writeCheckpoint

public void writeCheckpoint(Dag<JobExecutionPlan> dag)
                     throws IOException

Persist the Dag to the backing store. This is not an actual checkpoint but more like a Write-ahead log, where uncommitted job will be persisted and be picked up again when leader transition happens.

Specified by:

writeCheckpoint in interface DagStateStore

Parameters:

dag - The dag submitted to DagManager

Throws:

IOException

cleanUp

public void cleanUp(Dag<JobExecutionPlan> dag)

Delete the Dag from the backing store, typically upon completion of execution.

Specified by:

cleanUp in interface DagStateStore

Parameters:

dag - The dag completed/cancelled from execution on SpecExecutor.

getDags

public List<Dag<JobExecutionPlan>> getDags()
                                    throws IOException

Load all currently running Dags from the underlying store. Typically, invoked when a new DagManager takes over or on restart of service.

Specified by:

getDags in interface DagStateStore

Returns:

a List of currently running Dags.

Throws:

IOException

getDag

public Dag<JobExecutionPlan> getDag(File dagFile)
                             throws IOException

Return a Dag given a file name.

Parameters:

dagFile -

Returns:

the Dag associated with the dagFile.

Throws:

IOException