org.apache.gobblin.service.modules.template_catalog

Class FSFlowTemplateCatalog

java.lang.Object

com.google.common.util.concurrent.AbstractIdleService

org.apache.gobblin.runtime.job_catalog.JobCatalogBase

org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

org.apache.gobblin.runtime.job_catalog.FSJobCatalog

org.apache.gobblin.service.modules.template_catalog.FSFlowTemplateCatalog

All Implemented Interfaces:

com.google.common.util.concurrent.Service, Instrumentable, StandardMetricsBridge, JobCatalog, JobCatalogListenersContainer, JobCatalogWithTemplates, MutableJobCatalog, FlowCatalogWithTemplates

Direct Known Subclasses:

ObservingFSFlowEdgeTemplateCatalog

@Alpha
public class FSFlowTemplateCatalog
extends FSJobCatalog
implements FlowCatalogWithTemplates

An implementation of a catalog for FlowTemplates. Provides basic API for retrieving a FlowTemplate from the catalog and for retrieving JobTemplates that are part of a FlowTemplate. The flow and job configuration files are assumed to have the following path structure:

/path/to/template/catalog/flowName/flow.conf

/path/to/template/catalog/flowName/jobs/job1.(job|template)

/path/to/template/catalog/flowName/jobs/job2.(job|template)

Avoid confusing with FlowCatalog which is a catalog for FlowSpec. Note that any exceptions thrown here should be propagated into called level for handling, since the handling of exceptions while loading/resolving template is subject to caller logic.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

ImmutableFSJobCatalog.ConfigAccessor, ImmutableFSJobCatalog.JobSpecConverter

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.MutableJobCatalog

MutableJobCatalog.MutableStandardMetrics

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.JobCatalog

JobCatalog.StandardMetrics

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Field Summary

Fields

Modifier and Type	Field and Description
static String	FLOW_CONF_FILE_NAME
protected static String	FS_SCHEME
static List<String>	JOB_FILE_EXTENSIONS
static String	JOBS_DIR_NAME

Fields inherited from class org.apache.gobblin.runtime.job_catalog.FSJobCatalog

CONF_EXTENSION, mutableMetrics

Fields inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

DESCRIPTION_KEY_IN_JOBSPEC, fs, FS_CATALOG_KEY_PREFIX, jobConfDirPath, loader, pathAlterationDetector, sysConfig, VERSION_KEY_IN_JOBSPEC

Fields inherited from class org.apache.gobblin.runtime.job_catalog.JobCatalogBase

listeners, log, metricContext, metrics

Constructor Summary

Constructors

Constructor and Description
FSFlowTemplateCatalog(com.typesafe.config.Config sysConfig) Initialize the FlowCatalog

Method Summary

Modifier and Type	Method and Description
boolean	getAndSetShouldRefreshFlowGraph(boolean value)
FlowTemplate	getFlowTemplate(URI flowTemplateDirURI) Get FlowTemplate with given URI.
List<JobTemplate>	getJobTemplatesForFlow(URI flowTemplateDirURI)

Methods inherited from class org.apache.gobblin.runtime.job_catalog.FSJobCatalog

createStandardMetrics, getAllTemplates, getInjectedExtension, getPathForURI, getTemplate, put, remove, shouldLoadGlobalConf

Methods inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

getJobs, getJobSpec, getJobSpecIterator, getListener, shutDown, startUp

Methods inherited from class org.apache.gobblin.runtime.job_catalog.JobCatalogBase

addListener, generateTags, getMetricContext, getMetrics, isInstrumentationEnabled, notifyAllListeners, registerWeakJobCatalogListener, removeListener, switchMetricContext, switchMetricContext

Methods inherited from class com.google.common.util.concurrent.AbstractIdleService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, start, startAndWait, startAsync, state, stop, stopAndWait, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.gobblin.runtime.api.JobCatalog

getJobs, getJobSpec, getJobSpecIterator, getMetrics, getStandardMetricsCollection

Methods inherited from interface org.apache.gobblin.runtime.api.JobCatalogListenersContainer

addListener, registerWeakJobCatalogListener, removeListener

Methods inherited from interface org.apache.gobblin.instrumented.Instrumentable

generateTags, getMetricContext, isInstrumentationEnabled, switchMetricContext, switchMetricContext

Field Detail

JOBS_DIR_NAME

public static final String JOBS_DIR_NAME

See Also:

Constant Field Values

FLOW_CONF_FILE_NAME

public static final String FLOW_CONF_FILE_NAME

See Also:

Constant Field Values

JOB_FILE_EXTENSIONS

public static final List<String> JOB_FILE_EXTENSIONS

FS_SCHEME

protected static final String FS_SCHEME

See Also:

Constant Field Values

Constructor Detail

FSFlowTemplateCatalog

public FSFlowTemplateCatalog(com.typesafe.config.Config sysConfig)
                      throws IOException

Initialize the FlowCatalog

Parameters:

sysConfig - that must contain the fully qualified path of the flow template catalog

Throws:

IOException

Method Detail

getFlowTemplate

public FlowTemplate getFlowTemplate(URI flowTemplateDirURI)
                             throws SpecNotFoundException,
                                    JobTemplate.TemplateException,
                                    IOException,
                                    URISyntaxException

Description copied from interface: FlowCatalogWithTemplates

Get FlowTemplate with given URI.

Specified by:

getFlowTemplate in interface FlowCatalogWithTemplates

Parameters:

flowTemplateDirURI - URI of the flow template directory

Returns:

a FlowTemplate

Throws:

SpecNotFoundException

JobTemplate.TemplateException

IOException

URISyntaxException

getJobTemplatesForFlow

public List<JobTemplate> getJobTemplatesForFlow(URI flowTemplateDirURI)
                                         throws IOException,
                                                SpecNotFoundException,
                                                JobTemplate.TemplateException,
                                                URISyntaxException

Specified by:

getJobTemplatesForFlow in interface FlowCatalogWithTemplates

Parameters:

flowTemplateDirURI - Relative URI of the flow template directory

Returns:

a list of JobTemplates for a given flow identified by its URI.

Throws:

IOException

SpecNotFoundException

JobTemplate.TemplateException

URISyntaxException

getAndSetShouldRefreshFlowGraph

public boolean getAndSetShouldRefreshFlowGraph(boolean value)