org.apache.gobblin.service.modules.flow

Interface SpecCompiler

All Superinterfaces:

Instrumentable, SpecCatalogListener

All Known Implementing Classes:

BaseFlowToJobSpecCompiler, IdentityFlowToJobSpecCompiler, MockedSpecCompiler, MultiHopFlowCompiler

public interface SpecCompiler
extends SpecCatalogListener, Instrumentable

Take in a logical Spec and compile corresponding materialized Specs and the mapping to SpecExecutor that they can be run on.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

SpecCatalogListener.AddSpecCallback, SpecCatalogListener.DeleteSpecCallback, SpecCatalogListener.UpdateSpecCallback

Method Summary

Modifier and Type	Method and Description
void	awaitHealthy() Waits for the SpecCompiler to become healthy.
Dag<JobExecutionPlan>	compileFlow(Spec spec) Take in a logical Spec and compile corresponding materialized Specs and the mapping to SpecExecutor that they can be run on.
Map<URI,TopologySpec>	getTopologySpecMap() Map of Spec URI and TopologySpec the SpecCompiler is aware about.
void	setActive(boolean active) Mark the SpecCompiler active/inactive.

Methods inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

getName, onAddSpec, onDeleteSpec, onUpdateSpec

Methods inherited from interface org.apache.gobblin.instrumented.Instrumentable

generateTags, getMetricContext, isInstrumentationEnabled, switchMetricContext, switchMetricContext

Method Detail

compileFlow

Dag<JobExecutionPlan> compileFlow(Spec spec)

Take in a logical Spec and compile corresponding materialized Specs and the mapping to SpecExecutor that they can be run on. All the specs generated from the compileFlow must have a "flow.executionId"

Parameters:

spec - Spec to compile.

Returns:

Map of materialized physical Spec and SpecExecutor.

getTopologySpecMap

Map<URI,TopologySpec> getTopologySpecMap()

Map of Spec URI and TopologySpec the SpecCompiler is aware about.

Returns:

Map of Spec URI and TopologySpec

setActive

void setActive(boolean active)

Mark the SpecCompiler active/inactive. Useful to trigger the initialization of SpecCompiler, if necessary, before it can start compiling FlowSpecs.

Parameters:

active -

awaitHealthy

void awaitHealthy()
           throws InterruptedException

Waits for the SpecCompiler to become healthy. A SpecCompiler is healthy when all the component services it depends on have been successfully initialized. For instance, the MultiHopFlowCompiler is healthy when the DataNodes and FlowEdges can be added to the FlowGraph. The FlowEdge instantiation in turn depends on the successful initialization of TopologyCatalog, which instantiates all the configured SpecExecutors.

Throws:

InterruptedException