org.apache.uima.ducc.rm.scheduler
Class RmJob

java.lang.Object
  org.apache.uima.ducc.rm.scheduler.RmJob

All Implemented Interfaces:

IRmJob, SchedConstants

public class RmJob
extends Object
implements SchedConstants, IRmJob

id

protected org.apache.uima.ducc.common.utils.id.DuccId id

ducc_type

protected org.apache.uima.ducc.transport.event.common.IDuccTypes.DuccType ducc_type

name

protected String name

resource_class_name

protected String resource_class_name

resource_class

protected ResourceClass resource_class

user_priority

protected int user_priority

n_machines

protected int n_machines

min_shares

protected int min_shares

max_shares

protected int max_shares

is_reservation

protected boolean is_reservation

threads

protected int threads

memory

protected int memory

nquestions

protected int nquestions

nquestions_remaining

protected int nquestions_remaining

time_per_item

protected double time_per_item

share_order

protected int share_order

share_cap

protected int share_cap

job_cap

protected int job_cap

pure_fair_share

protected int pure_fair_share

submit_time

protected long submit_time

username

protected String username

user

protected User user

assignedShares

protected HashMap<Share,Share> assignedShares

pendingShares

protected HashMap<Share,Share> pendingShares

pendingRemoves

protected HashMap<Share,Share> pendingRemoves

recoveredShares

protected HashMap<Share,Share> recoveredShares

init_wait

protected boolean init_wait

completed

protected boolean completed

orchestrator_epoch

protected int orchestrator_epoch

rm_rate

protected int rm_rate

ducc_epoch

protected int ducc_epoch

jobprops

protected Properties jobprops

refusalReason

protected String refusalReason

RmJob

protected RmJob()

RmJob

public RmJob(org.apache.uima.ducc.common.utils.id.DuccId id)

init

public void init()

read the props file and set inital values for internal fields

Specified by:

init in interface IRmJob

getId

public org.apache.uima.ducc.common.utils.id.DuccId getId()

Specified by:

getId in interface IRmJob

setId

public void setId(org.apache.uima.ducc.common.utils.id.DuccId id)

getFriendlyId

public long getFriendlyId()

Specified by:

getFriendlyId in interface IRmJob

getName

public String getName()

Specified by:

getName in interface IRmJob

setJobName

public void setJobName(String name)

Specified by:

setJobName in interface IRmJob

setReservation

public void setReservation()

Specified by:

setReservation in interface IRmJob

isReservation

public boolean isReservation()

Specified by:

isReservation in interface IRmJob

markComplete

public void markComplete()

For non-preemptable, remember max alloc has been reached, so we don't try to expand if used for a job-job. For preemptable, must remember the job completed for defrag, because it could stick around a while after completion.

Specified by:

markComplete in interface IRmJob

isCompleted

public boolean isCompleted()

Specified by:

isCompleted in interface IRmJob

setInitWait

public boolean setInitWait(boolean is_running)

The matrix: IW = am I in initializaton wait RU = am I runnable Resched = do we need to reschedule Reset = set IW to this x = don't care IW RU Resched Reset T T T F T F F T F T F F F F x x So resched = IR & RU IW = !RU We return resched so caller knows to tickle the scheduler

Specified by:

setInitWait in interface IRmJob

setResourceClass

public void setResourceClass(ResourceClass cl)

Save ref to the class we are in, and init class-based structures.

Specified by:

setResourceClass in interface IRmJob

nQuestions

public int nQuestions()

Number of questions submitted, all must be answered. This is used by job manager to know when they've all been dealt with.

Specified by:

nQuestions in interface IRmJob

setNQuestions

public void setNQuestions(int allq,
                          int remainingq,
                          double time_per_item)

Specified by:

setNQuestions in interface IRmJob

nQuestionsRemaining

public int nQuestionsRemaining()

Number of questions still to be answered. Used by scheduler to determing current machine requirement.

Specified by:

nQuestionsRemaining in interface IRmJob

getSharesByMachine

public Map<Machine,Map<Share,Share>> getSharesByMachine()

Description copied from interface: IRmJob

For defrag, we want to know what machines a job is running on.

Specified by:

getSharesByMachine in interface IRmJob

getMachines

public Map<Machine,Machine> getMachines()

Specified by:

getMachines in interface IRmJob

clearShares

public void clearShares()

Before each scheduling epoch, clear the counting from the last time.

Specified by:

clearShares in interface IRmJob

setPureFairShare

public void setPureFairShare(int pfs)

public void addQShares(int s) { this.shares_given += ( s / share_order ) ; // convert to N-shares == processes }

Specified by:

setPureFairShare in interface IRmJob

getPureFairShare

public int getPureFairShare()

Specified by:

getPureFairShare in interface IRmJob

getGivenByOrder

public int[] getGivenByOrder()

setGivenByOrder

public void setGivenByOrder(int[] gbo)

getShareWeight

public int getShareWeight()

initWantedByOrder

public void initWantedByOrder(ResourceClass unused)

getWantedByOrder

public int[] getWantedByOrder()

calculateCap

public int calculateCap(int order,
                        int total)

countNSharesGiven

public int countNSharesGiven()

Specified by:

countNSharesGiven in interface IRmJob

countQSharesGiven

public int countQSharesGiven()

Specified by:

countQSharesGiven in interface IRmJob

countNSharesLost

public int countNSharesLost()

Number of N-shares I'm losing.

Specified by:

countNSharesLost in interface IRmJob

getAssignedShares

public HashMap<Share,Share> getAssignedShares()

Officially allocated shares assigned to this job which are known to be in use.

Specified by:

getAssignedShares in interface IRmJob

getRecoveredShares

public HashMap<Share,Share> getRecoveredShares()

Shares recovered from the OR during job recovery.

Specified by:

getRecoveredShares in interface IRmJob

getPendingShares

public HashMap<Share,Share> getPendingShares()

Newly allocated shares that have not been dispatched. They're unavailable for scheduling but job manager doesn't know about them yet. When we tell job manager we'll "promote" them to the assignedShares list.

Specified by:

getPendingShares in interface IRmJob

promoteShares

public HashMap<Share,Share> promoteShares()

We're dispatching, move machines to active list, and clear pending list. Tell caller which machines are affected so it can dispatch them.

Specified by:

promoteShares in interface IRmJob

cancelPending

public void cancelPending(Share s)

This share is being donated to someone more needy than I - see defrag code in NodepoolScheduler

Specified by:

cancelPending in interface IRmJob

assignShare

public void assignShare(Share s)

Scheduler found us a new toy, make it pending until it's given to job manager.

Specified by:

assignShare in interface IRmJob

recoverShare

public void recoverShare(Share s)

Job recovery: OR reports this share as one it already knew about.

Specified by:

recoverShare in interface IRmJob

isPendingShare

public boolean isPendingShare(Share s)

Specified by:

isPendingShare in interface IRmJob

isExpanded

public boolean isExpanded()

Do I have un-dispatched shares?

Specified by:

isExpanded in interface IRmJob

isShrunken

public boolean isShrunken()

Specified by:

isShrunken in interface IRmJob

isStable

public boolean isStable()

Specified by:

isStable in interface IRmJob

isDormant

public boolean isDormant()

Specified by:

isDormant in interface IRmJob

removeAllShares

public void removeAllShares()

Description copied from interface: IRmJob

Clear the job, it has no resources of any form.

Specified by:

removeAllShares in interface IRmJob

removeShare

public void removeShare(Share share)

I've shrunk or this share has nothing left to do. Remove this specific share.

Specified by:

removeShare in interface IRmJob

shrinkByOne

public void shrinkByOne(Share share)

Used by Fixed Share and by Reservations

Specified by:

shrinkByOne in interface IRmJob

shrinkByOrderByMachine

public int shrinkByOrderByMachine(int shares,
                                  int order,
                                  boolean force,
                                  org.apache.uima.ducc.rm.scheduler.NodePool nodepool)

Shrink by 'shares' from machines of largest order starting from 'order' and decreasing. Investment is not used, this is to shrink-from-largest-machine to MINIMIZE FRAGMENTATIOPN. This implementation is simplest, we just vacate. This is called in the sequence of highest order wanted so we're leaving the largest holes in the largest machines first.

Specified by:

shrinkByOrderByMachine in interface IRmJob

Parameters:

shares - - number of N-shares that are wanted

order - - N - try to free up space for shares of this size.

force - - When evicting for non-preemptables, we may need to free the requested shares even if it puts us over our "fair" count. If this happens we'll end up "sliding" onto other machines (eventually).

nodepool - - only interested in shares from this nodepool. So when this is called, somebody needs (shares*order) shares, given out in chunks of (order).

shrinkByInvestment

public int shrinkByInvestment(int shares,
                              int order,
                              boolean force,
                              org.apache.uima.ducc.rm.scheduler.NodePool nodepool)

Shrink by 'shares' from machines of largest order starting from 'order' and decreasing. Investment is not used, this is a shrink-from-largest-machine to MINIMIZE LOST WORK at the possibl eexpense of fragmentation.

Specified by:

shrinkByInvestment in interface IRmJob

Parameters:

shares - - number of N-shares that are wanted

order - - N - try to free up space for shares of this size.

force - - When evicting for non-preemptables, we may need to free the requested shares even if it puts us over our "fair" count. If this happens we'll end up "sliding" onto other machines (eventually).

nodepool - - only interested in shares from this nodepool. So when this is called, somebody needs (shares*order) shares, given out in chunks of (order).

shrinkTo

public void shrinkTo(int k)

I'm shrinking by this many. Put into pendingRemoves but don't take them off the assigned list until we know job manager has been told. TODO: this will likely change when the Scheduler starts deciding - either we'll put decision logic into Job proper, or defer it to the IScheduler implementation (probably the latter).

isShrunk

public boolean isShrunk()

Waiting for somebody to deal with my shrinkage?

getPendingRemoves

public HashMap<Share,Share> getPendingRemoves()

Return the reclaimed shares.

Specified by:

getPendingRemoves in interface IRmJob

clearPendingRemoves

public void clearPendingRemoves()

And finally, dump the pending shrinkage.

Specified by:

clearPendingRemoves in interface IRmJob

clearRecoveredShares

public void clearRecoveredShares()

Recovery complete, clear the share map

Specified by:

clearRecoveredShares in interface IRmJob

countNShares

public int countNShares()

Find number of nShares (virtual shares) this machine has assigned already. If things are slow in the cluster the pending removes might be non-zero. This is an extreme corner case it's best to be safe.

Specified by:

countNShares in interface IRmJob

refuse

public void refuse(String refusal)

Description copied from interface: IRmJob

Can't schedudle this nohow. Here's why.

Specified by:

refuse in interface IRmJob

isRefused

public boolean isRefused()

Specified by:

isRefused in interface IRmJob

getRefusalReason

public String getRefusalReason()

Specified by:

getRefusalReason in interface IRmJob

setShareOrder

public void setShareOrder(int s)

Description copied from interface: IRmJob

Scheduler looks at job memory and decides what its share order is.

Specified by:

setShareOrder in interface IRmJob

getShareOrder

public int getShareOrder()

Specified by:

getShareOrder in interface IRmJob

setShareCap

public void setShareCap(int cap)

During the scheduling algorithm we want to track some things by userid. The "share cap" stuff is used to keep track of max shares that I can actually use or want during scheduling but is generally just nonsense.

getShareCap

public int getShareCap()

initJobCap

public void initJobCap()

This returns the largest number that can actually be used, which will be either the share cap itself, or nProcess / nThreads, in N shares.

Specified by:

initJobCap in interface IRmJob

getJobCap

public int getJobCap()

Description copied from interface: IRmJob

This returns the largest number that can actually be used, which will be either the share cap itself, or nProcess / nThreads, in quantum shares.

Specified by:

getJobCap in interface IRmJob

getMaxShares

public int getMaxShares()

Specified by:

getMaxShares in interface IRmJob

getMinShares

public int getMinShares()

Specified by:

getMinShares in interface IRmJob

setMaxShares

public void setMaxShares(int s)

Specified by:

setMaxShares in interface IRmJob

setMinShares

public void setMinShares(int s)

Specified by:

setMinShares in interface IRmJob

isRunning

public boolean isRunning()

getUserName

public String getUserName()

Specified by:

getUserName in interface IRmJob

setUserName

public void setUserName(String n)

Specified by:

setUserName in interface IRmJob

getUser

public User getUser()

Specified by:

getUser in interface IRmJob

setUser

public void setUser(User u)

Specified by:

setUser in interface IRmJob

getTimestamp

public long getTimestamp()

Specified by:

getTimestamp in interface IRmJob

setTimestamp

public void setTimestamp(long t)

Specified by:

setTimestamp in interface IRmJob

getUserPriority

public int getUserPriority()

Specified by:

getUserPriority in interface IRmJob

setUserPriority

public void setUserPriority(int p)

Specified by:

setUserPriority in interface IRmJob

getClassName

public String getClassName()

Specified by:

getClassName in interface IRmJob

setClassName

public void setClassName(String class_name)

Specified by:

setClassName in interface IRmJob

getSchedulingPriority

public int getSchedulingPriority()

Specified by:

getSchedulingPriority in interface IRmJob

getSchedulingPolicy

public SchedConstants.Policy getSchedulingPolicy()

Specified by:

getSchedulingPolicy in interface IRmJob

getResourceClass

public ResourceClass getResourceClass()

Specified by:

getResourceClass in interface IRmJob

countInstances

public int countInstances()

Specified by:

countInstances in interface IRmJob

setNInstances

public void setNInstances(int m)

Specified by:

setNInstances in interface IRmJob

getMaxMachines

public int getMaxMachines()

nThreads

public int nThreads()

Specified by:

nThreads in interface IRmJob

setThreads

public void setThreads(int th)

Specified by:

setThreads in interface IRmJob

getMemory

public int getMemory()

Specified by:

getMemory in interface IRmJob

setMemory

public void setMemory(int memory)

Specified by:

setMemory in interface IRmJob

setDuccType

public void setDuccType(org.apache.uima.ducc.transport.event.common.IDuccTypes.DuccType type)

Specified by:

setDuccType in interface IRmJob

getDuccType

public org.apache.uima.ducc.transport.event.common.IDuccTypes.DuccType getDuccType()

Specified by:

getDuccType in interface IRmJob

isInitialized

public boolean isInitialized()

Is at least one of my current shares initialized?

Specified by:

isInitialized in interface IRmJob

printShares

public String printShares()

Logging and debugging - nice to know what my assigned shares are.

Specified by:

printShares in interface IRmJob

getHeader

public static String getHeader()

toString

public String toString()

Overrides:

toString in class Object

toStringWithHeader

public String toStringWithHeader()

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

getApportionmentSorter

public Comparator<org.apache.uima.ducc.rm.scheduler.IEntity> getApportionmentSorter()