org.apache.gobblin.service.modules.template_catalog

Class ObservingFSFlowEdgeTemplateCatalog

java.lang.Object

com.google.common.util.concurrent.AbstractIdleService

org.apache.gobblin.runtime.job_catalog.JobCatalogBase

org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

org.apache.gobblin.runtime.job_catalog.FSJobCatalog

org.apache.gobblin.service.modules.template_catalog.FSFlowTemplateCatalog

org.apache.gobblin.service.modules.template_catalog.ObservingFSFlowEdgeTemplateCatalog

All Implemented Interfaces:

com.google.common.util.concurrent.Service, Instrumentable, StandardMetricsBridge, JobCatalog, JobCatalogListenersContainer, JobCatalogWithTemplates, MutableJobCatalog, FlowCatalogWithTemplates

public class ObservingFSFlowEdgeTemplateCatalog
extends FSFlowTemplateCatalog

FSFlowTemplateCatalog that keeps a cache of flow and job templates. It has a PathAlterationListener on the root path, and clears the cache when a change is detected so that the next call to getFlowTemplate(URI) will use the updated files.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

ImmutableFSJobCatalog.ConfigAccessor, ImmutableFSJobCatalog.JobSpecConverter

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.MutableJobCatalog

MutableJobCatalog.MutableStandardMetrics

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.JobCatalog

JobCatalog.StandardMetrics

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Field Summary

Fields inherited from class org.apache.gobblin.service.modules.template_catalog.FSFlowTemplateCatalog

FLOW_CONF_FILE_NAME, FS_SCHEME, JOB_FILE_EXTENSIONS, JOBS_DIR_NAME

Fields inherited from class org.apache.gobblin.runtime.job_catalog.FSJobCatalog

CONF_EXTENSION, mutableMetrics

Fields inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

DESCRIPTION_KEY_IN_JOBSPEC, fs, FS_CATALOG_KEY_PREFIX, jobConfDirPath, loader, pathAlterationDetector, sysConfig, VERSION_KEY_IN_JOBSPEC

Fields inherited from class org.apache.gobblin.runtime.job_catalog.JobCatalogBase

listeners, log, metricContext, metrics

Constructor Summary

Constructors

Constructor and Description
ObservingFSFlowEdgeTemplateCatalog(com.typesafe.config.Config sysConfig, ReadWriteLock rwLock)

Method Summary

Modifier and Type	Method and Description
boolean	getAndSetShouldRefreshFlowGraph(boolean value)
FlowTemplate	getFlowTemplate(URI flowTemplateDirURI) Get FlowTemplate with given URI.
List<JobTemplate>	getJobTemplatesForFlow(URI flowTemplateDirURI)
protected org.apache.gobblin.util.filesystem.PathAlterationListener	getListener()
protected void	startUp()

Methods inherited from class org.apache.gobblin.runtime.job_catalog.FSJobCatalog

createStandardMetrics, getAllTemplates, getInjectedExtension, getPathForURI, getTemplate, put, remove, shouldLoadGlobalConf

Methods inherited from class org.apache.gobblin.runtime.job_catalog.ImmutableFSJobCatalog

getJobs, getJobSpec, getJobSpecIterator, shutDown

Methods inherited from class org.apache.gobblin.runtime.job_catalog.JobCatalogBase

addListener, generateTags, getMetricContext, getMetrics, isInstrumentationEnabled, notifyAllListeners, registerWeakJobCatalogListener, removeListener, switchMetricContext, switchMetricContext

Methods inherited from class com.google.common.util.concurrent.AbstractIdleService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, start, startAndWait, startAsync, state, stop, stopAndWait, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.gobblin.runtime.api.JobCatalog

getJobs, getJobSpec, getJobSpecIterator, getMetrics, getStandardMetricsCollection

Methods inherited from interface org.apache.gobblin.runtime.api.JobCatalogListenersContainer

addListener, registerWeakJobCatalogListener, removeListener

Methods inherited from interface org.apache.gobblin.instrumented.Instrumentable

generateTags, getMetricContext, isInstrumentationEnabled, switchMetricContext, switchMetricContext

Constructor Detail

ObservingFSFlowEdgeTemplateCatalog

public ObservingFSFlowEdgeTemplateCatalog(com.typesafe.config.Config sysConfig,
                                          ReadWriteLock rwLock)
                                   throws IOException

Throws:

IOException

Method Detail

getFlowTemplate

public FlowTemplate getFlowTemplate(URI flowTemplateDirURI)
                             throws SpecNotFoundException,
                                    JobTemplate.TemplateException,
                                    IOException,
                                    URISyntaxException

Description copied from interface: FlowCatalogWithTemplates

Get FlowTemplate with given URI.

Specified by:

getFlowTemplate in interface FlowCatalogWithTemplates

Overrides:

getFlowTemplate in class FSFlowTemplateCatalog

Parameters:

flowTemplateDirURI - URI of the flow template directory

Returns:

a FlowTemplate

Throws:

SpecNotFoundException - if a JobTemplate with given URI cannot be found.

JobTemplate.TemplateException

IOException

URISyntaxException

getJobTemplatesForFlow

public List<JobTemplate> getJobTemplatesForFlow(URI flowTemplateDirURI)
                                         throws IOException,
                                                SpecNotFoundException,
                                                JobTemplate.TemplateException,
                                                URISyntaxException

Specified by:

getJobTemplatesForFlow in interface FlowCatalogWithTemplates

Overrides:

getJobTemplatesForFlow in class FSFlowTemplateCatalog

Parameters:

flowTemplateDirURI - Relative URI of the flow template directory

Returns:

a list of JobTemplates for a given flow identified by its URI.

Throws:

IOException

SpecNotFoundException

JobTemplate.TemplateException

URISyntaxException

getListener

protected org.apache.gobblin.util.filesystem.PathAlterationListener getListener()

Overrides:

getListener in class ImmutableFSJobCatalog

startUp

protected void startUp()
                throws IOException

Overrides:

startUp in class ImmutableFSJobCatalog

Throws:

IOException

getAndSetShouldRefreshFlowGraph

public boolean getAndSetShouldRefreshFlowGraph(boolean value)

Overrides:

getAndSetShouldRefreshFlowGraph in class FSFlowTemplateCatalog