org.apache.drill.exec.planner.fragment

Class SimpleParallelizer

java.lang.Object

org.apache.drill.exec.planner.fragment.SimpleParallelizer

All Implemented Interfaces:

ParallelizationParameters

Direct Known Subclasses:

SplittingParallelizer

public class SimpleParallelizer
extends Object
implements ParallelizationParameters

The simple parallelizer determines the level of parallelization of a plan based on the cost of the underlying operations. It doesn't take into account system load or other factors. Based on the cost of the query, the parallelization for each major fragment will be determined. Once the amount of parallelization is done, assignment is done based on round robin assignment ordered by operator affinity (locality) to available execution Drillbits.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SimpleParallelizer.CountRequiredFragments Designed to setup initial values for arriving fragment accounting.

Constructor Summary

Constructors

Constructor and Description
SimpleParallelizer(long parallelizationThreshold, int maxWidthPerNode, int maxGlobalWidth, double affinityFactor)
SimpleParallelizer(QueryContext context)

Method Summary

Methods

Modifier and Type	Method and Description
protected QueryWorkUnit	generateWorkUnit(OptionList options, CoordinationProtos.DrillbitEndpoint foremanNode, UserBitShared.QueryId queryId, Fragment rootNode, PlanningSet planningSet, UserSession session, BitControl.QueryContextInformation queryContextInfo)
double	getAffinityFactor()
QueryWorkUnit	getFragments(OptionList options, CoordinationProtos.DrillbitEndpoint foremanNode, UserBitShared.QueryId queryId, Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints, Fragment rootFragment, UserSession session, BitControl.QueryContextInformation queryContextInfo) Generate a set of assigned fragments based on the provided fragment tree.
protected PlanningSet	getFragmentsHelper(Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints, Fragment rootFragment) Helper method to reuse the code for QueryWorkUnit(s) generation
int	getMaxGlobalWidth()
int	getMaxWidthPerNode()
long	getSliceTarget()
List<QueryWorkUnit>	getSplitFragments(OptionList options, CoordinationProtos.DrillbitEndpoint foremanNode, UserBitShared.QueryId queryId, Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints, PhysicalPlanReader reader, Fragment rootFragment, UserSession session, BitControl.QueryContextInformation queryContextInfo) Create multiple physical plans from original query planning, it will allow execute them eventually independently
void	initFragmentWrappers(Fragment rootFragment, PlanningSet planningSet)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleParallelizer

public SimpleParallelizer(QueryContext context)

SimpleParallelizer

public SimpleParallelizer(long parallelizationThreshold,
                  int maxWidthPerNode,
                  int maxGlobalWidth,
                  double affinityFactor)

Method Detail

getSliceTarget

public long getSliceTarget()

Specified by:

getSliceTarget in interface ParallelizationParameters

Returns:

Configured max width per slice of work.

getMaxWidthPerNode

public int getMaxWidthPerNode()

Specified by:

getMaxWidthPerNode in interface ParallelizationParameters

Returns:

Configured maximum allowed number of parallelization units per node.

getMaxGlobalWidth

public int getMaxGlobalWidth()

Specified by:

getMaxGlobalWidth in interface ParallelizationParameters

Returns:

Configured maximum allowed number of parallelization units per all nodes in the cluster.

getAffinityFactor

public double getAffinityFactor()

Specified by:

getAffinityFactor in interface ParallelizationParameters

Returns:

Factor by which a node with endpoint affinity will be favored while creating assignment.

getFragments

public QueryWorkUnit getFragments(OptionList options,
                         CoordinationProtos.DrillbitEndpoint foremanNode,
                         UserBitShared.QueryId queryId,
                         Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints,
                         Fragment rootFragment,
                         UserSession session,
                         BitControl.QueryContextInformation queryContextInfo)
                           throws ExecutionSetupException

Generate a set of assigned fragments based on the provided fragment tree. Do not allow parallelization stages to go beyond the global max width.

Parameters:

options - Option list

foremanNode - The driving/foreman node for this query. (this node)

queryId - The queryId for this query.

activeEndpoints - The list of endpoints to consider for inclusion in planning this query.

reader - Tool used to read JSON plans

rootFragment - The root node of the PhysicalPlan that we will be parallelizing.

session - UserSession of user who launched this query.

queryContextInfo - Info related to the context when query has started.

Returns:

The list of generated PlanFragment protobuf objects to be assigned out to the individual nodes.

Throws:

ExecutionSetupException

getSplitFragments

public List<QueryWorkUnit> getSplitFragments(OptionList options,
                                    CoordinationProtos.DrillbitEndpoint foremanNode,
                                    UserBitShared.QueryId queryId,
                                    Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints,
                                    PhysicalPlanReader reader,
                                    Fragment rootFragment,
                                    UserSession session,
                                    BitControl.QueryContextInformation queryContextInfo)
                                      throws ExecutionSetupException

Create multiple physical plans from original query planning, it will allow execute them eventually independently

Parameters:

options -

foremanNode -

queryId -

activeEndpoints -

reader -

rootFragment -

session -

queryContextInfo -

Returns:

Throws:

ExecutionSetupException

getFragmentsHelper

protected PlanningSet getFragmentsHelper(Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints,
                             Fragment rootFragment)
                                  throws ExecutionSetupException

Helper method to reuse the code for QueryWorkUnit(s) generation

Parameters:

activeEndpoints -

rootFragment -

Returns:

Throws:

ExecutionSetupException

initFragmentWrappers

public void initFragmentWrappers(Fragment rootFragment,
                        PlanningSet planningSet)

generateWorkUnit

protected QueryWorkUnit generateWorkUnit(OptionList options,
                             CoordinationProtos.DrillbitEndpoint foremanNode,
                             UserBitShared.QueryId queryId,
                             Fragment rootNode,
                             PlanningSet planningSet,
                             UserSession session,
                             BitControl.QueryContextInformation queryContextInfo)
                                  throws ExecutionSetupException

Throws:

ExecutionSetupException