org.apache.gobblin.service.modules.flow

Class BaseFlowToJobSpecCompiler

java.lang.Object

org.apache.gobblin.service.modules.flow.BaseFlowToJobSpecCompiler

All Implemented Interfaces:

Instrumentable, SpecCatalogListener, SpecCompiler

Direct Known Subclasses:

IdentityFlowToJobSpecCompiler, MultiHopFlowCompiler

@Alpha
public abstract class BaseFlowToJobSpecCompiler
extends Object
implements SpecCompiler

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

SpecCatalogListener.AddSpecCallback, SpecCatalogListener.DeleteSpecCallback, SpecCatalogListener.UpdateSpecCallback

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	active
protected com.typesafe.config.Config	config
protected com.google.common.base.Optional<com.codahale.metrics.Timer>	dataAuthorizationTimer
protected com.google.common.base.Optional<com.codahale.metrics.Meter>	flowCompilationFailedMeter
protected com.google.common.base.Optional<com.codahale.metrics.Meter>	flowCompilationSuccessFulMeter
protected com.google.common.base.Optional<com.codahale.metrics.Timer>	flowCompilationTimer
protected org.slf4j.Logger	log
protected org.apache.gobblin.metrics.MetricContext	metricContext
protected com.google.common.base.Optional<FSJobCatalog>	templateCatalog
protected Map<URI,TopologySpec>	topologySpecMap

Constructor Summary

Constructors

Constructor and Description
BaseFlowToJobSpecCompiler(com.typesafe.config.Config config)
BaseFlowToJobSpecCompiler(com.typesafe.config.Config config, boolean instrumentationEnabled)
BaseFlowToJobSpecCompiler(com.typesafe.config.Config config, com.google.common.base.Optional<org.slf4j.Logger> log)
BaseFlowToJobSpecCompiler(com.typesafe.config.Config config, com.google.common.base.Optional<org.slf4j.Logger> log, boolean instrumentationEnabled)

Method Summary

Modifier and Type	Method and Description
void	awaitHealthy() Waits for the SpecCompiler to become healthy.
abstract Dag<JobExecutionPlan>	compileFlow(Spec spec) Take in a logical Spec and compile corresponding materialized Specs and the mapping to SpecExecutor that they can be run on.
List<org.apache.gobblin.metrics.Tag<?>>	generateTags(State state)
org.apache.gobblin.metrics.MetricContext	getMetricContext()
Map<URI,TopologySpec>	getTopologySpecMap() Map of Spec URI and TopologySpec the SpecCompiler is aware about.
boolean	isInstrumentationEnabled()
protected JobSpec	jobSpecGenerator(FlowSpec flowSpec) Naive implementation of generating jobSpec, which fetch the first available template, in an exemplified single-hop FlowCompiler implementation.
protected URI	jobSpecTemplateURIGenerator(FlowSpec flowSpec) It returns the template uri for job.
URI	jobSpecURIGenerator(Object... objects) It can receive multiple number of parameters, needed to generate a unique URI.
AddSpecResponse	onAddSpec(Spec addedSpec)
void	onDeleteSpec(URI deletedSpecURI, String deletedSpecVersion)
void	onDeleteSpec(URI deletedSpecURI, String deletedSpecVersion, Properties headers)
void	onUpdateSpec(Spec updatedSpec)
void	switchMetricContext(List<org.apache.gobblin.metrics.Tag<?>> tags)
void	switchMetricContext(org.apache.gobblin.metrics.MetricContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.gobblin.service.modules.flow.SpecCompiler

setActive

Methods inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

getName

Field Detail

topologySpecMap

protected final Map<URI,TopologySpec> topologySpecMap

config

protected final com.typesafe.config.Config config

log

protected final org.slf4j.Logger log

templateCatalog

protected final com.google.common.base.Optional<FSJobCatalog> templateCatalog

metricContext

protected final org.apache.gobblin.metrics.MetricContext metricContext

flowCompilationSuccessFulMeter

protected com.google.common.base.Optional<com.codahale.metrics.Meter> flowCompilationSuccessFulMeter

flowCompilationFailedMeter

protected com.google.common.base.Optional<com.codahale.metrics.Meter> flowCompilationFailedMeter

flowCompilationTimer

protected com.google.common.base.Optional<com.codahale.metrics.Timer> flowCompilationTimer

dataAuthorizationTimer

protected com.google.common.base.Optional<com.codahale.metrics.Timer> dataAuthorizationTimer

active

protected boolean active

Constructor Detail

BaseFlowToJobSpecCompiler

public BaseFlowToJobSpecCompiler(com.typesafe.config.Config config)

BaseFlowToJobSpecCompiler

public BaseFlowToJobSpecCompiler(com.typesafe.config.Config config,
                                 boolean instrumentationEnabled)

BaseFlowToJobSpecCompiler

public BaseFlowToJobSpecCompiler(com.typesafe.config.Config config,
                                 com.google.common.base.Optional<org.slf4j.Logger> log)

BaseFlowToJobSpecCompiler

public BaseFlowToJobSpecCompiler(com.typesafe.config.Config config,
                                 com.google.common.base.Optional<org.slf4j.Logger> log,
                                 boolean instrumentationEnabled)

Method Detail

awaitHealthy

public void awaitHealthy()
                  throws InterruptedException

Description copied from interface: SpecCompiler

Waits for the SpecCompiler to become healthy. A SpecCompiler is healthy when all the component services it depends on have been successfully initialized. For instance, the MultiHopFlowCompiler is healthy when the DataNodes and FlowEdges can be added to the FlowGraph. The FlowEdge instantiation in turn depends on the successful initialization of TopologyCatalog, which instantiates all the configured SpecExecutors.

Specified by:

awaitHealthy in interface SpecCompiler

Throws:

InterruptedException

onAddSpec

public AddSpecResponse onAddSpec(Spec addedSpec)

Specified by:

onAddSpec in interface SpecCatalogListener

onDeleteSpec

public void onDeleteSpec(URI deletedSpecURI,
                         String deletedSpecVersion)

onDeleteSpec

public void onDeleteSpec(URI deletedSpecURI,
                         String deletedSpecVersion,
                         Properties headers)

Specified by:

onDeleteSpec in interface SpecCatalogListener

onUpdateSpec

public void onUpdateSpec(Spec updatedSpec)

Specified by:

onUpdateSpec in interface SpecCatalogListener

getMetricContext

@Nonnull
public org.apache.gobblin.metrics.MetricContext getMetricContext()

Specified by:

getMetricContext in interface Instrumentable

isInstrumentationEnabled

public boolean isInstrumentationEnabled()

Specified by:

isInstrumentationEnabled in interface Instrumentable

generateTags

public List<org.apache.gobblin.metrics.Tag<?>> generateTags(State state)

Specified by:

generateTags in interface Instrumentable

switchMetricContext

public void switchMetricContext(List<org.apache.gobblin.metrics.Tag<?>> tags)

Specified by:

switchMetricContext in interface Instrumentable

switchMetricContext

public void switchMetricContext(org.apache.gobblin.metrics.MetricContext context)

Specified by:

switchMetricContext in interface Instrumentable

getTopologySpecMap

public Map<URI,TopologySpec> getTopologySpecMap()

Description copied from interface: SpecCompiler

Map of Spec URI and TopologySpec the SpecCompiler is aware about.

Specified by:

getTopologySpecMap in interface SpecCompiler

Returns:

Map of Spec URI and TopologySpec

compileFlow

public abstract Dag<JobExecutionPlan> compileFlow(Spec spec)

Description copied from interface: SpecCompiler

Take in a logical Spec and compile corresponding materialized Specs and the mapping to SpecExecutor that they can be run on. All the specs generated from the compileFlow must have a "flow.executionId"

Specified by:

compileFlow in interface SpecCompiler

Parameters:

spec - Spec to compile.

Returns:

Map of materialized physical Spec and SpecExecutor.

jobSpecGenerator

protected JobSpec jobSpecGenerator(FlowSpec flowSpec)

Naive implementation of generating jobSpec, which fetch the first available template, in an exemplified single-hop FlowCompiler implementation.

Parameters:

flowSpec -

Returns:

jobSpecURIGenerator

public URI jobSpecURIGenerator(Object... objects)

It can receive multiple number of parameters, needed to generate a unique URI. Implementation is flowSpecCompiler dependent. This method should return URI which has job name at third place, when split by "/" e.g. /flowGroup/flowName /flowGroup/flowName/sourceNode-targetNode SafeDatasetCommit creates state store using this name and KafkaJobMonitor extract job name to find the state store path.

Parameters:

objects -

Returns:

jobSpecTemplateURIGenerator

protected URI jobSpecTemplateURIGenerator(FlowSpec flowSpec)

It returns the template uri for job. This method can be overridden by derived FlowToJobSpecCompiler classes.

Parameters:

flowSpec -

Returns:

template URI