PactCompiler (flink-compiler 0.7.0-incubating API)

java.lang.Object
- org.apache.flink.compiler.PactCompiler

```
public class PactCompiler
extends Object
```
The optimizer that takes the user specified program plan and creates an optimized plan that contains exact descriptions about how the physical execution will take place. It first translates the user program into an internal optimizer representation and then chooses between different alternatives for shipping strategies and local strategies.
The basic principle is taken from optimizer works in systems such as Volcano/Cascades and Selinger/System-R/DB2. The optimizer walks from the sinks down, generating interesting properties, and ascends from the sources generating alternative plans, pruning against the interesting properties.
The optimizer also assigns the memory to the individual tasks. This is currently done in a very simple fashion: All sub-tasks that need memory (e.g. reduce or join) are given an equal share of memory.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class PactCompiler.InterestingPropertyVisitor
Visitor that computes the interesting properties for each node in the plan.

Nested Classes
Modifier and Type	Class and Description
`static class`	`PactCompiler.InterestingPropertyVisitor` Visitor that computes the interesting properties for each node in the plan.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`HINT_LOCAL_STRATEGY` Compiler hint key for the operator's local strategy.
`static String`	`HINT_LOCAL_STRATEGY_COMBINING_SORT` Value for the local strategy compiler hint that enforces a sort based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_HASH_BUILD_FIRST` Value for the local strategy compiler hint that enforces a hash based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_HASH_BUILD_SECOND` Value for the local strategy compiler hint that enforces a hash based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_MERGE` Value for the local strategy compiler hint that enforces a merge based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_FIRST` Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy.
`static String`	`HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_SECOND` Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy.
`static String`	`HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_FIRST` Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy.
`static String`	`HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_SECOND` Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy.
`static String`	`HINT_LOCAL_STRATEGY_SORT` Value for the local strategy compiler hint that enforces a sort based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_SORT_BOTH_MERGE` Value for the local strategy compiler hint that enforces a sort merge based local strategy on both inputs with subsequent merging of inputs.
`static String`	`HINT_LOCAL_STRATEGY_SORT_FIRST_MERGE` Value for the local strategy compiler hint that enforces a sort merge based local strategy.
`static String`	`HINT_LOCAL_STRATEGY_SORT_SECOND_MERGE` Value for the local strategy compiler hint that enforces a sort merge based local strategy.
`static String`	`HINT_SHIP_STRATEGY` Compiler hint key for the input channel's shipping strategy.
`static String`	`HINT_SHIP_STRATEGY_BROADCAST` Value for the shipping strategy compiler hint that enforces a broadcast strategy on the input channel.
`static String`	`HINT_SHIP_STRATEGY_FIRST_INPUT` Compiler hint key for the first input channel's shipping strategy.
`static String`	`HINT_SHIP_STRATEGY_FORWARD` Value for the shipping strategy compiler hint that enforces a Forward strategy on the input channel, i.e.
`static String`	`HINT_SHIP_STRATEGY_REPARTITION` Value for the shipping strategy compiler hint that enforces a random repartition strategy.
`static String`	`HINT_SHIP_STRATEGY_REPARTITION_HASH` Value for the shipping strategy compiler hint that enforces a hash-partition strategy.
`static String`	`HINT_SHIP_STRATEGY_REPARTITION_RANGE` Value for the shipping strategy compiler hint that enforces a range-partition strategy.
`static String`	`HINT_SHIP_STRATEGY_SECOND_INPUT` Compiler hint key for the second input channel's shipping strategy.
`static org.slf4j.Logger`	`LOG` The log handle that is used by the compiler to log messages.

Constructor Summary

Constructors
Constructor and Description
`PactCompiler()` Creates a new compiler instance.
`PactCompiler(CostEstimator estimator)` Creates a new compiler instance.
`PactCompiler(DataStatistics stats)` Creates a new compiler instance that uses the statistics object to determine properties about the input.
`PactCompiler(DataStatistics stats, CostEstimator estimator)` Creates a new compiler instance that uses the statistics object to determine properties about the input.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`OptimizedPlan`	`compile(Plan program)` Translates the given plan in to an OptimizedPlan, where all nodes have their local strategy assigned and all channels have a shipping strategy assigned.
`static List<DataSinkNode>`	`createPreOptimizedPlan(Plan program)` This function performs only the first step to the compilation process - the creation of the optimizer representation of the plan.
`int`	`getDefaultDegreeOfParallelism()`
`void`	`setDefaultDegreeOfParallelism(int defaultDegreeOfParallelism)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - HINT_SHIP_STRATEGY
```
public static final String HINT_SHIP_STRATEGY
```
    Compiler hint key for the input channel's shipping strategy. This String is a key to the operator's stub parameters. The corresponding value tells the compiler which shipping strategy to use for the input channel. If the operator has two input channels, the shipping strategy is applied to both input channels.
    
    See Also:
    
    Constant Field Values
  - HINT_SHIP_STRATEGY_FIRST_INPUT
```
public static final String HINT_SHIP_STRATEGY_FIRST_INPUT
```
    Compiler hint key for the first input channel's shipping strategy. This String is a key to the operator's stub parameters. The corresponding value tells the compiler which shipping strategy to use for the first input channel. Only applicable to operators with two inputs.
    
    See Also:
    
    Constant Field Values
  - HINT_SHIP_STRATEGY_SECOND_INPUT
```
public static final String HINT_SHIP_STRATEGY_SECOND_INPUT
```
    Compiler hint key for the second input channel's shipping strategy. This String is a key to the operator's stub parameters. The corresponding value tells the compiler which shipping strategy to use for the second input channel. Only applicable to operators with two inputs.
    
    See Also:
    
    Constant Field Values
  - HINT_SHIP_STRATEGY_FORWARD
```
public static final String HINT_SHIP_STRATEGY_FORWARD
```
    Value for the shipping strategy compiler hint that enforces a Forward strategy on the input channel, i.e. no redistribution of any kind.
    
    See Also:
    
    HINT_SHIP_STRATEGY, HINT_SHIP_STRATEGY_FIRST_INPUT, HINT_SHIP_STRATEGY_SECOND_INPUT, Constant Field Values
  - HINT_SHIP_STRATEGY_REPARTITION
```
public static final String HINT_SHIP_STRATEGY_REPARTITION
```
    Value for the shipping strategy compiler hint that enforces a random repartition strategy.
    
    See Also:
    
    HINT_SHIP_STRATEGY, HINT_SHIP_STRATEGY_FIRST_INPUT, HINT_SHIP_STRATEGY_SECOND_INPUT, Constant Field Values
  - HINT_SHIP_STRATEGY_REPARTITION_HASH
```
public static final String HINT_SHIP_STRATEGY_REPARTITION_HASH
```
    Value for the shipping strategy compiler hint that enforces a hash-partition strategy.
    
    See Also:
    
    HINT_SHIP_STRATEGY, HINT_SHIP_STRATEGY_FIRST_INPUT, HINT_SHIP_STRATEGY_SECOND_INPUT, Constant Field Values
  - HINT_SHIP_STRATEGY_REPARTITION_RANGE
```
public static final String HINT_SHIP_STRATEGY_REPARTITION_RANGE
```
    Value for the shipping strategy compiler hint that enforces a range-partition strategy.
    
    See Also:
    
    HINT_SHIP_STRATEGY, HINT_SHIP_STRATEGY_FIRST_INPUT, HINT_SHIP_STRATEGY_SECOND_INPUT, Constant Field Values
  - HINT_SHIP_STRATEGY_BROADCAST
```
public static final String HINT_SHIP_STRATEGY_BROADCAST
```
    Value for the shipping strategy compiler hint that enforces a broadcast strategy on the input channel.
    
    See Also:
    
    HINT_SHIP_STRATEGY, HINT_SHIP_STRATEGY_FIRST_INPUT, HINT_SHIP_STRATEGY_SECOND_INPUT, Constant Field Values
  - HINT_LOCAL_STRATEGY
```
public static final String HINT_LOCAL_STRATEGY
```
    Compiler hint key for the operator's local strategy. This String is a key to the operator's stub parameters. The corresponding value tells the compiler which local strategy to use to process the data inside one partition.
    This hint is ignored by operators that do not have a local strategy (such as Map), or by operators that have no choice in their local strategy (such as Cross).
    
    See Also:
    
    Constant Field Values
  - HINT_LOCAL_STRATEGY_SORT
```
public static final String HINT_LOCAL_STRATEGY_SORT
```
    Value for the local strategy compiler hint that enforces a sort based local strategy. For example, a Reduce operator will sort the data to group it.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_COMBINING_SORT
```
public static final String HINT_LOCAL_STRATEGY_COMBINING_SORT
```
    Value for the local strategy compiler hint that enforces a sort based local strategy. During sorting a combine method is repeatedly applied to reduce the data volume. For example, a Reduce operator will sort the data to group it.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_SORT_BOTH_MERGE
```
public static final String HINT_LOCAL_STRATEGY_SORT_BOTH_MERGE
```
    Value for the local strategy compiler hint that enforces a sort merge based local strategy on both inputs with subsequent merging of inputs. For example, a Match or CoGroup operator will use a sort-merge strategy to find pairs of matching keys.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_SORT_FIRST_MERGE
```
public static final String HINT_LOCAL_STRATEGY_SORT_FIRST_MERGE
```
    Value for the local strategy compiler hint that enforces a sort merge based local strategy. The the first input is sorted, the second input is assumed to be sorted. After sorting both inputs are merged. For example, a Match or CoGroup operator will use a sort-merge strategy to find pairs of matching keys.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_SORT_SECOND_MERGE
```
public static final String HINT_LOCAL_STRATEGY_SORT_SECOND_MERGE
```
    Value for the local strategy compiler hint that enforces a sort merge based local strategy. The the second input is sorted, the first input is assumed to be sorted. After sorting both inputs are merged. For example, a Match or CoGroup operator will use a sort-merge strategy to find pairs of matching keys.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_MERGE
```
public static final String HINT_LOCAL_STRATEGY_MERGE
```
    Value for the local strategy compiler hint that enforces a merge based local strategy. Both inputs are assumed to be sorted and are merged. For example, a Match or CoGroup operator will use a merge strategy to find pairs of matching keys.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_HASH_BUILD_FIRST
```
public static final String HINT_LOCAL_STRATEGY_HASH_BUILD_FIRST
```
    Value for the local strategy compiler hint that enforces a hash based local strategy. For example, a Match operator will use a hybrid-hash-join strategy to find pairs of matching keys. The first input will be used to build the hash table, the second input will be used to probe the table.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_HASH_BUILD_SECOND
```
public static final String HINT_LOCAL_STRATEGY_HASH_BUILD_SECOND
```
    Value for the local strategy compiler hint that enforces a hash based local strategy. For example, a Match operator will use a hybrid-hash-join strategy to find pairs of matching keys. The second input will be used to build the hash table, the first input will be used to probe the table.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_FIRST
```
public static final String HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_FIRST
```
    Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy. A Cross operator will process the data of the first input in the outer-loop of the nested loops. Hence, the data of the first input will be is streamed though, while the data of the second input is stored on disk and repeatedly read.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_SECOND
```
public static final String HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_SECOND
```
    Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy. A Cross operator will process the data of the second input in the outer-loop of the nested loops. Hence, the data of the second input will be is streamed though, while the data of the first input is stored on disk and repeatedly read.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_FIRST
```
public static final String HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_FIRST
```
    Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy. A Cross operator will process the data of the first input in the outer-loop of the nested loops. Further more, the first input, being the outer side, will be processed in blocks, and for each block, the second input, being the inner side, will read repeatedly from disk.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_SECOND
```
public static final String HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_SECOND
```
    Value for the local strategy compiler hint that chooses the outer side of the nested-loop local strategy. A Cross operator will process the data of the second input in the outer-loop of the nested loops. Further more, the second input, being the outer side, will be processed in blocks, and for each block, the first input, being the inner side, will read repeatedly from disk.
    
    See Also:
    
    HINT_LOCAL_STRATEGY, Constant Field Values
  - LOG
```
public static final org.slf4j.Logger LOG
```
    The log handle that is used by the compiler to log messages.
- Constructor Detail
  - PactCompiler
```
public PactCompiler()
```
    Creates a new compiler instance. The compiler has no access to statistics about the inputs and can hence not determine any properties. It will perform all optimization with unknown sizes and default to the most robust execution strategies. The compiler also uses conservative default estimates for the operator costs, since it has no access to another cost estimator.
    The address of the job manager (to obtain system characteristics) is determined via the global configuration.
  - PactCompiler
```
public PactCompiler(DataStatistics stats)
```
    Creates a new compiler instance that uses the statistics object to determine properties about the input. Given those statistics, the compiler can make better choices for the execution strategies. as if no filesystem was given. The compiler uses conservative default estimates for the operator costs, since it has no access to another cost estimator.
    The address of the job manager (to obtain system characteristics) is determined via the global configuration.
    
    Parameters:
    
    stats - The statistics to be used to determine the input properties.
  - PactCompiler
```
public PactCompiler(CostEstimator estimator)
```
    Creates a new compiler instance. The compiler has no access to statistics about the inputs and can hence not determine any properties. It will perform all optimization with unknown sizes and default to the most robust execution strategies. It uses however the given cost estimator to compute the costs of the individual operations.
    The address of the job manager (to obtain system characteristics) is determined via the global configuration.
    
    Parameters:
    
    estimator - The CostEstimator to use to cost the individual operations.
  - PactCompiler
```
public PactCompiler(DataStatistics stats,
                    CostEstimator estimator)
```
    Creates a new compiler instance that uses the statistics object to determine properties about the input. Given those statistics, the compiler can make better choices for the execution strategies. as if no filesystem was given. It uses the given cost estimator to compute the costs of the individual operations.
    The address of the job manager (to obtain system characteristics) is determined via the global configuration.
    
    Parameters:
    
    stats - The statistics to be used to determine the input properties.
    
    estimator - The CostEstimator to use to cost the individual operations.
- Method Detail
  - getDefaultDegreeOfParallelism
```
public int getDefaultDegreeOfParallelism()
```
  - setDefaultDegreeOfParallelism
```
public void setDefaultDegreeOfParallelism(int defaultDegreeOfParallelism)
```
  - compile
```
public OptimizedPlan compile(Plan program)
                      throws CompilerException
```
    Translates the given plan in to an OptimizedPlan, where all nodes have their local strategy assigned and all channels have a shipping strategy assigned. The compiler connects to the job manager to obtain information about the available instances and their memory and then chooses an instance type to schedule the execution on.
    The compilation process itself goes through several phases:
    1. Create an optimizer data flow representation of the program, assign parallelism and compute size estimates.
    2. Compute interesting properties and auxiliary structures.
    3. Enumerate plan alternatives. This cannot be done in the same step as the interesting property computation (as opposed to the Database approaches), because we support plans that are not trees.
    Parameters:
    
    program - The program to be translated.
    
    Returns:
    
    The optimized plan.
    
    Throws:
    
    CompilerException - Thrown, if the plan is invalid or the optimizer encountered an inconsistent situation during the compilation process.
  - createPreOptimizedPlan
```
public static List<DataSinkNode> createPreOptimizedPlan(Plan program)
```
    This function performs only the first step to the compilation process - the creation of the optimizer representation of the plan. No estimations or enumerations of alternatives are done here.
    
    Parameters:
    
    program - The plan to generate the optimizer representation for.
    
    Returns:
    
    The optimizer representation of the plan, as a collection of all data sinks from the plan can be traversed.

Class PactCompiler

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

HINT_SHIP_STRATEGY

HINT_SHIP_STRATEGY_FIRST_INPUT

HINT_SHIP_STRATEGY_SECOND_INPUT

HINT_SHIP_STRATEGY_FORWARD

HINT_SHIP_STRATEGY_REPARTITION

HINT_SHIP_STRATEGY_REPARTITION_HASH

HINT_SHIP_STRATEGY_REPARTITION_RANGE

HINT_SHIP_STRATEGY_BROADCAST

HINT_LOCAL_STRATEGY

HINT_LOCAL_STRATEGY_SORT

HINT_LOCAL_STRATEGY_COMBINING_SORT

HINT_LOCAL_STRATEGY_SORT_BOTH_MERGE

HINT_LOCAL_STRATEGY_SORT_FIRST_MERGE

HINT_LOCAL_STRATEGY_SORT_SECOND_MERGE

HINT_LOCAL_STRATEGY_MERGE

HINT_LOCAL_STRATEGY_HASH_BUILD_FIRST

HINT_LOCAL_STRATEGY_HASH_BUILD_SECOND

HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_FIRST

HINT_LOCAL_STRATEGY_NESTEDLOOP_STREAMED_OUTER_SECOND

HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_FIRST

HINT_LOCAL_STRATEGY_NESTEDLOOP_BLOCKED_OUTER_SECOND

LOG

Constructor Detail

PactCompiler

PactCompiler

PactCompiler

PactCompiler

Method Detail

getDefaultDegreeOfParallelism

setDefaultDegreeOfParallelism

compile

createPreOptimizedPlan