org.apache.gobblin.service.modules.scheduler

Class GobblinServiceJobScheduler

java.lang.Object

com.google.common.util.concurrent.AbstractIdleService

org.apache.gobblin.scheduler.JobScheduler

org.apache.gobblin.service.modules.scheduler.GobblinServiceJobScheduler

All Implemented Interfaces:

com.google.common.util.concurrent.Service, SpecCatalogListener

@Alpha
public class GobblinServiceJobScheduler
extends JobScheduler
implements SpecCatalogListener

An extension to JobScheduler that is also a SpecCatalogListener. GobblinServiceJobScheduler listens for new / updated FlowSpec and schedules and runs them via Orchestrator.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GobblinServiceJobScheduler.GobblinServiceJob A Gobblin job to be scheduled.

Nested classes/interfaces inherited from class org.apache.gobblin.scheduler.JobScheduler

JobScheduler.Action, JobScheduler.GobblinJob

Nested classes/interfaces inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

SpecCatalogListener.AddSpecCallback, SpecCatalogListener.DeleteSpecCallback, SpecCatalogListener.UpdateSpecCallback

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Field Summary

Fields

Modifier and Type	Field and Description
protected org.slf4j.Logger	_log
static String	DR_FILTER_TAG Use this to tag all DR-applicable FlowSpec entries in SpecStore so only they would be loaded during DR handling.
protected com.google.common.base.Optional<FlowCatalog>	flowCatalog
static String	GOBBLIN_SERVICE_SCHEDULER_DR_NOMINATED
protected com.google.common.base.Optional<org.apache.helix.HelixManager>	helixManager
protected Orchestrator	orchestrator
protected Map<String,Spec>	scheduledFlowSpecs

Fields inherited from class org.apache.gobblin.scheduler.JobScheduler

JOB_LISTENER_KEY, JOB_SCHEDULER_KEY, jobConfigFileDirPath, jobConfigFileExtensions, jobExecutor, listener, pathAlterationDetector, properties, PROPERTIES_KEY

Constructor Summary

Constructors

Constructor and Description
GobblinServiceJobScheduler(String serviceName, com.typesafe.config.Config config, com.google.common.base.Optional<org.apache.helix.HelixManager> helixManager, com.google.common.base.Optional<FlowCatalog> flowCatalog, com.google.common.base.Optional<TopologyCatalog> topologyCatalog, com.google.common.base.Optional<DagManager> dagManager, SchedulerService schedulerService, com.google.common.base.Optional<org.slf4j.Logger> log)
GobblinServiceJobScheduler(String serviceName, com.typesafe.config.Config config, com.google.common.base.Optional<org.apache.helix.HelixManager> helixManager, com.google.common.base.Optional<FlowCatalog> flowCatalog, com.google.common.base.Optional<TopologyCatalog> topologyCatalog, Orchestrator orchestrator, SchedulerService schedulerService, com.google.common.base.Optional<org.slf4j.Logger> log)

Method Summary

Modifier and Type	Method and Description
protected static Spec	disableFlowRunImmediatelyOnStart(FlowSpec spec)
AddSpecResponse	onAddSpec(Spec addedSpec)
void	onDeleteSpec(URI deletedSpecURI, String deletedSpecVersion)
void	onDeleteSpec(URI deletedSpecURI, String deletedSpecVersion, Properties headers)
void	onUpdateSpec(Spec updatedSpec)
void	runJob(Properties jobProps, JobListener jobListener)
void	scheduleJob(Properties jobProps, JobListener jobListener) Synchronize the job scheduling because the same flowSpec can be scheduled by different threads.
void	setActive(boolean isActive)
protected void	startUp()

Methods inherited from class org.apache.gobblin.scheduler.JobScheduler

buildJobLauncher, getJobSpecResolver, getScheduledJobs, isCancelRequested, runJob, scheduleJob, scheduleJobImmediately, scheduleJobImmediately, shutDown, startServices, submitRunnableToExecutor, unscheduleJob

Methods inherited from class com.google.common.util.concurrent.AbstractIdleService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, start, startAndWait, startAsync, state, stop, stopAndWait, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.gobblin.runtime.api.SpecCatalogListener

getName

Field Detail

GOBBLIN_SERVICE_SCHEDULER_DR_NOMINATED

public static final String GOBBLIN_SERVICE_SCHEDULER_DR_NOMINATED

See Also:

Constant Field Values

_log

protected final org.slf4j.Logger _log

flowCatalog

protected final com.google.common.base.Optional<FlowCatalog> flowCatalog

helixManager

protected final com.google.common.base.Optional<org.apache.helix.HelixManager> helixManager

orchestrator

protected final Orchestrator orchestrator

scheduledFlowSpecs

protected final Map<String,Spec> scheduledFlowSpecs

DR_FILTER_TAG

public static final String DR_FILTER_TAG

Use this to tag all DR-applicable FlowSpec entries in SpecStore so only they would be loaded during DR handling.

See Also:

Constant Field Values

Constructor Detail

GobblinServiceJobScheduler

public GobblinServiceJobScheduler(String serviceName,
                                  com.typesafe.config.Config config,
                                  com.google.common.base.Optional<org.apache.helix.HelixManager> helixManager,
                                  com.google.common.base.Optional<FlowCatalog> flowCatalog,
                                  com.google.common.base.Optional<TopologyCatalog> topologyCatalog,
                                  Orchestrator orchestrator,
                                  SchedulerService schedulerService,
                                  com.google.common.base.Optional<org.slf4j.Logger> log)
                           throws Exception

Throws:

Exception

GobblinServiceJobScheduler

public GobblinServiceJobScheduler(String serviceName,
                                  com.typesafe.config.Config config,
                                  com.google.common.base.Optional<org.apache.helix.HelixManager> helixManager,
                                  com.google.common.base.Optional<FlowCatalog> flowCatalog,
                                  com.google.common.base.Optional<TopologyCatalog> topologyCatalog,
                                  com.google.common.base.Optional<DagManager> dagManager,
                                  SchedulerService schedulerService,
                                  com.google.common.base.Optional<org.slf4j.Logger> log)
                           throws Exception

Throws:

Exception

Method Detail

setActive

public void setActive(boolean isActive)

disableFlowRunImmediatelyOnStart

protected static Spec disableFlowRunImmediatelyOnStart(FlowSpec spec)

startUp

protected void startUp()
                throws Exception

Overrides:

startUp in class JobScheduler

Throws:

Exception

scheduleJob

public void scheduleJob(Properties jobProps,
                        JobListener jobListener)
                 throws JobException

Synchronize the job scheduling because the same flowSpec can be scheduled by different threads.

Overrides:

scheduleJob in class JobScheduler

Throws:

JobException

runJob

public void runJob(Properties jobProps,
                   JobListener jobListener)
            throws JobException

Overrides:

runJob in class JobScheduler

Throws:

JobException

onAddSpec

public AddSpecResponse onAddSpec(Spec addedSpec)

Specified by:

onAddSpec in interface SpecCatalogListener

Parameters:

addedSpec - spec to be added

Returns:

add spec response

onDeleteSpec

public void onDeleteSpec(URI deletedSpecURI,
                         String deletedSpecVersion)

onDeleteSpec

public void onDeleteSpec(URI deletedSpecURI,
                         String deletedSpecVersion,
                         Properties headers)

Specified by:

onDeleteSpec in interface SpecCatalogListener

onUpdateSpec

public void onUpdateSpec(Spec updatedSpec)

Specified by:

onUpdateSpec in interface SpecCatalogListener