org.apache.gobblin.service.modules.flow

Class MultiHopFlowCompiler

java.lang.Object

org.apache.gobblin.service.modules.flow.BaseFlowToJobSpecCompiler

org.apache.gobblin.service.modules.flow.MultiHopFlowCompiler

All Implemented Interfaces:

Instrumentable, SpecCatalogListener, SpecCompiler

@Alpha
public class MultiHopFlowCompiler
extends BaseFlowToJobSpecCompiler

Take in a logical Spec ie flow and compile corresponding materialized job Spec and its mapping to SpecExecutor.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

SpecCatalogListener.AddSpecCallback, SpecCatalogListener.DeleteSpecCallback, SpecCatalogListener.UpdateSpecCallback

Field Summary

Fields inherited from class org.apache.gobblin.service.modules.flow.BaseFlowToJobSpecCompiler

active, config, dataAuthorizationTimer, flowCompilationFailedMeter, flowCompilationSuccessFulMeter, flowCompilationTimer, log, metricContext, templateCatalog, topologySpecMap

Constructor Summary

Constructors

Constructor and Description
MultiHopFlowCompiler(com.typesafe.config.Config config)
MultiHopFlowCompiler(com.typesafe.config.Config config, boolean instrumentationEnabled)
MultiHopFlowCompiler(com.typesafe.config.Config config, com.google.common.base.Optional<org.slf4j.Logger> log)
MultiHopFlowCompiler(com.typesafe.config.Config config, com.google.common.base.Optional<org.slf4j.Logger> log, boolean instrumentationEnabled)

Method Summary

Modifier and Type	Method and Description
void	awaitHealthy() Waits for the SpecCompiler to become healthy.
Dag<JobExecutionPlan>	compileFlow(Spec spec) j
void	setActive(boolean active) Mark the SpecCompiler as active.

Methods inherited from class org.apache.gobblin.service.modules.flow.BaseFlowToJobSpecCompiler

generateTags, getMetricContext, getTopologySpecMap, isInstrumentationEnabled, jobSpecGenerator, jobSpecTemplateURIGenerator, jobSpecURIGenerator, onAddSpec, onDeleteSpec, onDeleteSpec, onUpdateSpec, switchMetricContext, switchMetricContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

getName

Constructor Detail

MultiHopFlowCompiler

public MultiHopFlowCompiler(com.typesafe.config.Config config)

MultiHopFlowCompiler

public MultiHopFlowCompiler(com.typesafe.config.Config config,
                            boolean instrumentationEnabled)

MultiHopFlowCompiler

public MultiHopFlowCompiler(com.typesafe.config.Config config,
                            com.google.common.base.Optional<org.slf4j.Logger> log)

MultiHopFlowCompiler

public MultiHopFlowCompiler(com.typesafe.config.Config config,
                            com.google.common.base.Optional<org.slf4j.Logger> log,
                            boolean instrumentationEnabled)

Method Detail

setActive

public void setActive(boolean active)

Mark the SpecCompiler as active. This in turn activates the GitFlowGraphMonitor, allowing to start polling and processing changes

Parameters:

active -

awaitHealthy

public void awaitHealthy()
                  throws InterruptedException

Description copied from interface: SpecCompiler

Waits for the SpecCompiler to become healthy. A SpecCompiler is healthy when all the component services it depends on have been successfully initialized. For instance, the MultiHopFlowCompiler is healthy when the DataNodes and FlowEdges can be added to the FlowGraph. The FlowEdge instantiation in turn depends on the successful initialization of TopologyCatalog, which instantiates all the configured SpecExecutors.

Specified by:

awaitHealthy in interface SpecCompiler

Overrides:

awaitHealthy in class BaseFlowToJobSpecCompiler

Throws:

InterruptedException

compileFlow

public Dag<JobExecutionPlan> compileFlow(Spec spec)

j

Specified by:

compileFlow in interface SpecCompiler

Specified by:

compileFlow in class BaseFlowToJobSpecCompiler

Parameters:

spec - an instance of FlowSpec.

Returns:

A DAG of JobExecutionPlans, which encapsulates the compiled JobSpecs together with the SpecExecutor where the job can be executed.