org.apache.flink.api.java.operators

Class TwoInputUdfOperator<IN1,IN2,OUT,O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

java.lang.Object

org.apache.flink.api.java.DataSet<OUT>

org.apache.flink.api.java.operators.Operator<OUT,O>

org.apache.flink.api.java.operators.TwoInputOperator<IN1,IN2,OUT,O>

org.apache.flink.api.java.operators.TwoInputUdfOperator<IN1,IN2,OUT,O>

Type Parameters:

IN1 - The data type of the first input data set.

IN2 - The data type of the second input data set.

OUT - The data type of the returned data set.

All Implemented Interfaces:

UdfOperator<O>

Direct Known Subclasses:

CoGroupOperator, CrossOperator, JoinOperator

public abstract class TwoInputUdfOperator<IN1,IN2,OUT,O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>
extends TwoInputOperator<IN1,IN2,OUT,O>
implements UdfOperator<O>

The TwoInputUdfOperator is the base class of all binary operators that execute user-defined functions (UDFs). The UDFs encapsulated by this operator are naturally UDFs that have two inputs (such as RichJoinFunction or RichCoGroupFunction).

This class encapsulates utilities for the UDFs, such as broadcast variables, parameterization through configuration objects, and semantic properties.

Field Summary

Fields inherited from class org.apache.flink.api.java.operators.Operator

dop, name

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	TwoInputUdfOperator(DataSet<IN1> input1, DataSet<IN2> input2, TypeInformation<OUT> resultType) Creates a new operators with the two given data sets as inputs.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	extractSemanticAnnotationsFromUdf(Class<?> udfClass)
Map<String,DataSet<?>>	getBroadcastSets() Gets the broadcast sets (name and data set) that have been added to context of the UDF.
Configuration	getParameters() Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration).
DualInputSemanticProperties	getSemanticProperties() Gets the semantic properties that have been set for the user-defined functions (UDF).
void	setSemanticProperties(DualInputSemanticProperties properties) Sets the semantic properties for the user-defined function (UDF).
O	withBroadcastSet(DataSet<?> data, String name) Adds a certain data set as a broadcast set to this operator.
O	withConstantSetFirst(String... constantSetFirst) Adds a constant-set annotation for the first input of the UDF.
O	withConstantSetSecond(String... constantSetSecond) Adds a constant-set annotation for the second input of the UDF.
O	withParameters(Configuration parameters) Sets the configuration parameters for the UDF.

Methods inherited from class org.apache.flink.api.java.operators.TwoInputOperator

getInput1, getInput1Type, getInput2, getInput2Type, translateToDataFlow

Methods inherited from class org.apache.flink.api.java.operators.Operator

getName, getParallelism, getResultType, name, setParallelism

Methods inherited from class org.apache.flink.api.java.DataSet

aggregate, checkSameExecutionContext, clean, coGroup, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, filter, first, flatMap, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionCustom, partitionCustom, partitionCustom, print, printToErr, project, rebalance, reduce, reduceGroup, runOperation, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TwoInputUdfOperator

protected TwoInputUdfOperator(DataSet<IN1> input1,
                   DataSet<IN2> input2,
                   TypeInformation<OUT> resultType)

Creates a new operators with the two given data sets as inputs. The given result type describes the data type of the elements in the data set produced by the operator.

Parameters:

input1 - The data set for the first input.

input2 - The data set for the second input.

resultType - The type of the elements in the resulting data set.

Method Detail

extractSemanticAnnotationsFromUdf

protected void extractSemanticAnnotationsFromUdf(Class<?> udfClass)

withParameters

public O withParameters(Configuration parameters)

Description copied from interface: UdfOperator

Sets the configuration parameters for the UDF. These are optional parameters that are passed to the UDF in the AbstractRichFunction.open(Configuration) method.

Specified by:

withParameters in interface UdfOperator<O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

Parameters:

parameters - The configuration parameters for the UDF.

Returns:

The operator itself, to allow chaining function calls.

withBroadcastSet

public O withBroadcastSet(DataSet<?> data,
                 String name)

Description copied from interface: UdfOperator

Adds a certain data set as a broadcast set to this operator. Broadcasted data sets are available at all parallel instances of this operator. A broadcast data set is registered under a certain name, and can be retrieved under that name from the operators runtime context via RuntimeContext.getBroadcastVariable(String). The runtime context itself is available in all UDFs via AbstractRichFunction.getRuntimeContext().

Specified by:

withBroadcastSet in interface UdfOperator<O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

Parameters:

data - The data set to be broadcasted.

name - The name under which the broadcast data set retrieved.

Returns:

The operator itself, to allow chaining function calls.

withConstantSetFirst

public O withConstantSetFirst(String... constantSetFirst)

Adds a constant-set annotation for the first input of the UDF.

Constant set annotations are used by the optimizer to infer the existence of data properties (sorted, partitioned, grouped). In certain cases, these annotations allow the optimizer to generate a more efficient execution plan which can lead to improved performance. Constant set annotations can only be specified if the first input and the output type of the UDF are of Tuple data types.

A constant-set annotation is a set of constant field specifications. The constant field specification String "4->3" specifies, that this UDF copies the fourth field of an input tuple to the third field of the output tuple. Field references are zero-indexed.

NOTICE: Constant set annotations are optional, but if given need to be correct. Otherwise, the program might produce wrong results!

Parameters:

constantSetFirst - A list of constant field specification Strings for the first input.

Returns:

This operator with an annotated constant field set for the first input.

withConstantSetSecond

public O withConstantSetSecond(String... constantSetSecond)

Adds a constant-set annotation for the second input of the UDF.

Constant set annotations are used by the optimizer to infer the existence of data properties (sorted, partitioned, grouped). In certain cases, these annotations allow the optimizer to generate a more efficient execution plan which can lead to improved performance. Constant set annotations can only be specified if the second input and the output type of the UDF are of Tuple data types.

A constant-set annotation is a set of constant field specifications. The constant field specification String "4->3" specifies, that this UDF copies the fourth field of an input tuple to the third field of the output tuple. Field references are zero-indexed.

NOTICE: Constant set annotations are optional, but if given need to be correct. Otherwise, the program might produce wrong results!

Parameters:

constantSetSecond - A list of constant field specification Strings for the second input.

Returns:

This operator with an annotated constant field set for the second input.

getBroadcastSets

public Map<String,DataSet<?>> getBroadcastSets()

Description copied from interface: UdfOperator

Gets the broadcast sets (name and data set) that have been added to context of the UDF. Broadcast sets are added to a UDF via the method UdfOperator.withBroadcastSet(DataSet, String).

Specified by:

getBroadcastSets in interface UdfOperator<O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

Returns:

The broadcast data sets that have been added to this UDF.

getParameters

public Configuration getParameters()

Description copied from interface: UdfOperator

Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration). The configuration is set via the UdfOperator.withParameters(Configuration) method.

Specified by:

getParameters in interface UdfOperator<O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

Returns:

The configuration parameters for the UDF.

getSemanticProperties

public DualInputSemanticProperties getSemanticProperties()

Description copied from interface: UdfOperator

Gets the semantic properties that have been set for the user-defined functions (UDF). This method may return null, if no semantic properties have been set so far.

Specified by:

getSemanticProperties in interface UdfOperator<O extends TwoInputUdfOperator<IN1,IN2,OUT,O>>

Returns:

The semantic properties of the UDF.

setSemanticProperties

public void setSemanticProperties(DualInputSemanticProperties properties)

Sets the semantic properties for the user-defined function (UDF). The semantic properties define how fields of tuples and other objects are modified or preserved through this UDF. The configured properties can be retrieved via UdfOperator.getSemanticProperties().

Parameters:

properties - The semantic properties for the UDF.

See Also:

UdfOperator.getSemanticProperties()