org.apache.flink.runtime.resourcemanager.active

Class ActiveResourceManager<WorkerType extends ResourceIDRetrievable>

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<ResourceManagerId>

org.apache.flink.runtime.resourcemanager.ResourceManager<WorkerType>

org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager<WorkerType>

All Implemented Interfaces:

AutoCloseable, ClusterPartitionManager, LeaderContender, ResourceEventHandler<WorkerType>, ResourceManagerGateway, FencedRpcGateway<ResourceManagerId>, RpcGateway, org.apache.flink.util.AutoCloseableAsync

public class ActiveResourceManager<WorkerType extends ResourceIDRetrievable>
extends ResourceManager<WorkerType>
implements ResourceEventHandler<WorkerType>

An active implementation of ResourceManager.

This resource manager actively requests and releases resources from/to the external resource management frameworks. With different ResourceManagerDriver provided, this resource manager can work with various frameworks.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.configuration.Configuration	flinkConfig

Fields inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

ioExecutor, RESOURCE_MANAGER_NAME, resourceManagerMetricGroup

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

ActiveResourceManager(ResourceManagerDriver<WorkerType> resourceManagerDriver, org.apache.flink.configuration.Configuration flinkConfig, RpcService rpcService, ResourceID resourceId, HighAvailabilityServices highAvailabilityServices, HeartbeatServices heartbeatServices, SlotManager slotManager, ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory, JobLeaderIdService jobLeaderIdService, ClusterInformation clusterInformation, FatalErrorHandler fatalErrorHandler, ResourceManagerMetricGroup resourceManagerMetricGroup, ThresholdMeter startWorkerFailureRater, java.time.Duration retryInterval, java.time.Duration workerRegistrationTimeout, Executor ioExecutor)

Method Summary

Modifier and Type	Method and Description
protected CompletableFuture<Void>	clearStateAsync() This method can be overridden to add a (non-blocking) state clearing routine to the ResourceManager that will be called when leadership is revoked.
protected void	initialize() Initializes the framework specific components.
protected void	internalDeregisterApplication(ApplicationStatus finalStatus, String optionalDiagnostics) The framework specific code to deregister the application.
void	onError(Throwable exception) Notifies that an error has occurred that the process cannot proceed.
void	onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWorkers) Notifies that workers of previous attempt have been recovered from the external resource manager.
protected void	onWorkerRegistered(WorkerType worker)
void	onWorkerTerminated(ResourceID resourceId, String diagnostics) Notifies that the worker has been terminated.
protected CompletableFuture<Void>	prepareLeadershipAsync() This method can be overridden to add a (non-blocking) initialization routine to the ResourceManager that will be called when leadership is granted but before leadership is confirmed.
protected void	registerMetrics()
boolean	startNewWorker(WorkerResourceSpec workerResourceSpec) Allocates a resource using the worker resource specification.
boolean	stopWorker(WorkerType worker) Stops the given worker.
protected void	terminate() Terminates the framework specific components.
protected WorkerType	workerStarted(ResourceID resourceID) Callback when a worker was started.

Methods inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

cancelSlotRequest, closeJobManagerConnection, closeTaskManagerConnection, declareRequiredResources, deregisterApplication, disconnectJobManager, disconnectTaskManager, getNumberOfRegisteredTaskManagers, getNumberRequiredTaskManagers, getRequiredResources, grantLeadership, handleError, hasLeadership, heartbeatFromJobManager, heartbeatFromTaskManager, jobLeaderLostLeadership, listDataSets, notifySlotAvailable, onFatalError, onLeadership, onStart, onStop, registerJobManager, registerTaskExecutor, releaseClusterPartitions, releaseResource, removeJob, requestResourceOverview, requestSlot, requestTaskExecutorThreadInfoGateway, requestTaskManagerDetailsInfo, requestTaskManagerFileUploadByName, requestTaskManagerFileUploadByType, requestTaskManagerInfo, requestTaskManagerLogList, requestTaskManagerMetricQueryServiceAddresses, requestThreadDump, revokeLeadership, sendSlotReport, setFailUnfulfillableRequest

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

callAsyncWithoutFencing, getFencingToken, getMainThreadExecutor, getUnfencedMainThreadExecutor, runAsyncWithoutFencing, setFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, closeAsync, getAddress, getEndpointId, getHostname, getRpcService, getSelfGateway, getTerminationFuture, internalCallOnStart, internalCallOnStop, isRunning, runAsync, scheduleRunAsync, scheduleRunAsync, start, stop, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.runtime.leaderelection.LeaderContender

getDescription

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Field Detail

flinkConfig

protected final org.apache.flink.configuration.Configuration flinkConfig

Constructor Detail

ActiveResourceManager

public ActiveResourceManager(ResourceManagerDriver<WorkerType> resourceManagerDriver,
                             org.apache.flink.configuration.Configuration flinkConfig,
                             RpcService rpcService,
                             ResourceID resourceId,
                             HighAvailabilityServices highAvailabilityServices,
                             HeartbeatServices heartbeatServices,
                             SlotManager slotManager,
                             ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory,
                             JobLeaderIdService jobLeaderIdService,
                             ClusterInformation clusterInformation,
                             FatalErrorHandler fatalErrorHandler,
                             ResourceManagerMetricGroup resourceManagerMetricGroup,
                             ThresholdMeter startWorkerFailureRater,
                             java.time.Duration retryInterval,
                             java.time.Duration workerRegistrationTimeout,
                             Executor ioExecutor)

Method Detail

initialize

protected void initialize()
                   throws ResourceManagerException

Description copied from class: ResourceManager

Initializes the framework specific components.

Specified by:

initialize in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Throws:

ResourceManagerException - which occurs during initialization and causes the resource manager to fail.

terminate

protected void terminate()
                  throws ResourceManagerException

Description copied from class: ResourceManager

Terminates the framework specific components.

Specified by:

terminate in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Throws:

ResourceManagerException

prepareLeadershipAsync

protected CompletableFuture<Void> prepareLeadershipAsync()

Description copied from class: ResourceManager

This method can be overridden to add a (non-blocking) initialization routine to the ResourceManager that will be called when leadership is granted but before leadership is confirmed.

Overrides:

prepareLeadershipAsync in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Returns:

Returns a CompletableFuture that completes when the computation is finished.

clearStateAsync

protected CompletableFuture<Void> clearStateAsync()

Description copied from class: ResourceManager

This method can be overridden to add a (non-blocking) state clearing routine to the ResourceManager that will be called when leadership is revoked.

Overrides:

clearStateAsync in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Returns:

Returns a CompletableFuture that completes when the state clearing routine is finished.

internalDeregisterApplication

protected void internalDeregisterApplication(ApplicationStatus finalStatus,
                                             @Nullable
                                             String optionalDiagnostics)
                                      throws ResourceManagerException

Description copied from class: ResourceManager

The framework specific code to deregister the application. This should report the application's final status and shut down the resource manager cleanly.

This method also needs to make sure all pending containers that are not registered yet are returned.

Specified by:

internalDeregisterApplication in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Parameters:

finalStatus - The application status to report.

optionalDiagnostics - A diagnostics message or null.

Throws:

ResourceManagerException - if the application could not be shut down.

startNewWorker

public boolean startNewWorker(WorkerResourceSpec workerResourceSpec)

Description copied from class: ResourceManager

Allocates a resource using the worker resource specification.

Specified by:

startNewWorker in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Parameters:

workerResourceSpec - workerResourceSpec specifies the size of the to be allocated resource

Returns:

whether the resource can be allocated

workerStarted

protected WorkerType workerStarted(ResourceID resourceID)

Description copied from class: ResourceManager

Callback when a worker was started.

Specified by:

workerStarted in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Parameters:

resourceID - The worker resource id

stopWorker

public boolean stopWorker(WorkerType worker)

Description copied from class: ResourceManager

Stops the given worker.

Specified by:

stopWorker in class ResourceManager<WorkerType extends ResourceIDRetrievable>

Parameters:

worker - The worker.

Returns:

True if the worker was stopped, otherwise false

onWorkerRegistered

protected void onWorkerRegistered(WorkerType worker)

Overrides:

onWorkerRegistered in class ResourceManager<WorkerType extends ResourceIDRetrievable>

registerMetrics

protected void registerMetrics()

Overrides:

registerMetrics in class ResourceManager<WorkerType extends ResourceIDRetrievable>

onPreviousAttemptWorkersRecovered

public void onPreviousAttemptWorkersRecovered(Collection<WorkerType> recoveredWorkers)

Description copied from interface: ResourceEventHandler

Notifies that workers of previous attempt have been recovered from the external resource manager.

Specified by:

onPreviousAttemptWorkersRecovered in interface ResourceEventHandler<WorkerType extends ResourceIDRetrievable>

Parameters:

recoveredWorkers - Collection of worker nodes, in the deployment specific type.

onWorkerTerminated

public void onWorkerTerminated(ResourceID resourceId,
                               String diagnostics)

Description copied from interface: ResourceEventHandler

Notifies that the worker has been terminated.

See also ResourceManagerDriver.requestResource(org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec).

Specified by:

onWorkerTerminated in interface ResourceEventHandler<WorkerType extends ResourceIDRetrievable>

Parameters:

resourceId - Identifier of the terminated worker.

diagnostics - Diagnostic message about the worker termination.

onError

public void onError(Throwable exception)

Description copied from interface: ResourceEventHandler

Notifies that an error has occurred that the process cannot proceed.

Specified by:

onError in interface ResourceEventHandler<WorkerType extends ResourceIDRetrievable>

Parameters:

exception - Exception that describes the error.