org.apache.gobblin.service.modules.orchestration

Class MysqlDagStateStore

java.lang.Object

org.apache.gobblin.service.modules.orchestration.MysqlDagStateStore

All Implemented Interfaces:

DagStateStore

public class MysqlDagStateStore
extends Object
implements DagStateStore

A implementation of DagStateStore using MySQL as a backup, leverage MysqlStateStore. It implements interfaces of DagStateStore but delegating responsibilities to methods provided in MysqlStateStore. It also implements conversion between Dag to State. The schema of this will simply be: | storeName | tableName | State | where storeName represents FlowId, a combination of FlowGroup and FlowName, and tableName represents FlowExecutionId. State is a pocket for serialized Dag object.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONFIG_PREFIX
static String	DAG_KEY_IN_STATE

Constructor Summary

Constructors

Constructor and Description
MysqlDagStateStore(com.typesafe.config.Config config, Map<URI,TopologySpec> topologySpecMap)

Method Summary

Modifier and Type	Method and Description
void	cleanUp(Dag<JobExecutionPlan> dag) Delete the Dag from the backing store, typically upon completion of execution.
protected StateStore<State>	createStateStore(com.typesafe.config.Config config) Creating an instance of StateStore.
List<Dag<JobExecutionPlan>>	getDags() Load all currently running Dags from the underlying store.
void	writeCheckpoint(Dag<JobExecutionPlan> dag) Persist the Dag to the backing store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONFIG_PREFIX

public static final String CONFIG_PREFIX

See Also:

Constant Field Values

DAG_KEY_IN_STATE

public static final String DAG_KEY_IN_STATE

See Also:

Constant Field Values

Constructor Detail

MysqlDagStateStore

public MysqlDagStateStore(com.typesafe.config.Config config,
                          Map<URI,TopologySpec> topologySpecMap)

Method Detail

createStateStore

protected StateStore<State> createStateStore(com.typesafe.config.Config config)

Creating an instance of StateStore.

writeCheckpoint

public void writeCheckpoint(Dag<JobExecutionPlan> dag)
                     throws IOException

Description copied from interface: DagStateStore

Persist the Dag to the backing store. This is not an actual checkpoint but more like a Write-ahead log, where uncommitted job will be persisted and be picked up again when leader transition happens.

Specified by:

writeCheckpoint in interface DagStateStore

Parameters:

dag - The dag submitted to DagManager

Throws:

IOException

cleanUp

public void cleanUp(Dag<JobExecutionPlan> dag)
             throws IOException

Description copied from interface: DagStateStore

Delete the Dag from the backing store, typically upon completion of execution.

Specified by:

cleanUp in interface DagStateStore

Parameters:

dag - The dag completed/cancelled from execution on SpecExecutor.

Throws:

IOException

getDags

public List<Dag<JobExecutionPlan>> getDags()
                                    throws IOException

Description copied from interface: DagStateStore

Load all currently running Dags from the underlying store. Typically, invoked when a new DagManager takes over or on restart of service.

Specified by:

getDags in interface DagStateStore

Returns:

a List of currently running Dags.

Throws:

IOException