org.apache.flink.api.java.operators

Class SingleInputUdfOperator<IN,OUT,O extends SingleInputUdfOperator<IN,OUT,O>>

java.lang.Object

org.apache.flink.api.java.DataSet<OUT>

org.apache.flink.api.java.operators.Operator<OUT,O>

org.apache.flink.api.java.operators.SingleInputOperator<IN,OUT,O>

org.apache.flink.api.java.operators.SingleInputUdfOperator<IN,OUT,O>

Type Parameters:

IN - The data type of the input data set.

OUT - The data type of the returned data set.

All Implemented Interfaces:

UdfOperator<O>

Direct Known Subclasses:

FilterOperator, FlatMapOperator, GroupReduceOperator, MapOperator, MapPartitionOperator, PartitionOperator, ReduceOperator

public abstract class SingleInputUdfOperator<IN,OUT,O extends SingleInputUdfOperator<IN,OUT,O>>
extends SingleInputOperator<IN,OUT,O>
implements UdfOperator<O>

The SingleInputUdfOperator is the base class of all unary operators that execute user-defined functions (UDFs). The UDFs encapsulated by this operator are naturally UDFs that have one input (such as RichMapFunction or RichReduceFunction).

This class encapsulates utilities for the UDFs, such as broadcast variables, parameterization through configuration objects, and semantic properties.

Field Summary

Fields inherited from class org.apache.flink.api.java.operators.Operator

dop, name

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SingleInputUdfOperator(DataSet<IN> input, TypeInformation<OUT> resultType) Creates a new operators with the given data set as input.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	extractSemanticAnnotationsFromUdf(Class<?> udfClass)
Map<String,DataSet<?>>	getBroadcastSets() Gets the broadcast sets (name and data set) that have been added to context of the UDF.
Configuration	getParameters() Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration).
SingleInputSemanticProperties	getSemanticProperties() Gets the semantic properties that have been set for the user-defined functions (UDF).
void	setSemanticProperties(SingleInputSemanticProperties properties) Sets the semantic properties for the user-defined function (UDF).
O	withBroadcastSet(DataSet<?> data, String name) Adds a certain data set as a broadcast set to this operator.
O	withConstantSet(String... constantSet) Adds a constant-set annotation for the UDF.
O	withParameters(Configuration parameters) Sets the configuration parameters for the UDF.

Methods inherited from class org.apache.flink.api.java.operators.SingleInputOperator

getInput, getInputType, translateToDataFlow

Methods inherited from class org.apache.flink.api.java.operators.Operator

getName, getParallelism, getResultType, name, setParallelism

Methods inherited from class org.apache.flink.api.java.DataSet

aggregate, checkSameExecutionContext, clean, coGroup, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, filter, first, flatMap, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionCustom, partitionCustom, partitionCustom, print, printToErr, project, rebalance, reduce, reduceGroup, runOperation, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleInputUdfOperator

protected SingleInputUdfOperator(DataSet<IN> input,
                      TypeInformation<OUT> resultType)

Creates a new operators with the given data set as input. The given result type describes the data type of the elements in the data set produced by the operator.

Parameters:

input - The data set that is the input to the operator.

resultType - The type of the elements in the resulting data set.

Method Detail

extractSemanticAnnotationsFromUdf

protected void extractSemanticAnnotationsFromUdf(Class<?> udfClass)

withParameters

public O withParameters(Configuration parameters)

Description copied from interface: UdfOperator

Sets the configuration parameters for the UDF. These are optional parameters that are passed to the UDF in the AbstractRichFunction.open(Configuration) method.

Specified by:

withParameters in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Parameters:

parameters - The configuration parameters for the UDF.

Returns:

The operator itself, to allow chaining function calls.

withBroadcastSet

public O withBroadcastSet(DataSet<?> data,
                 String name)

Description copied from interface: UdfOperator

Adds a certain data set as a broadcast set to this operator. Broadcasted data sets are available at all parallel instances of this operator. A broadcast data set is registered under a certain name, and can be retrieved under that name from the operators runtime context via RuntimeContext.getBroadcastVariable(String). The runtime context itself is available in all UDFs via AbstractRichFunction.getRuntimeContext().

Specified by:

withBroadcastSet in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Parameters:

data - The data set to be broadcasted.

name - The name under which the broadcast data set retrieved.

Returns:

The operator itself, to allow chaining function calls.

withConstantSet

public O withConstantSet(String... constantSet)

Adds a constant-set annotation for the UDF.

Constant set annotations are used by the optimizer to infer the existence of data properties (sorted, partitioned, grouped). In certain cases, these annotations allow the optimizer to generate a more efficient execution plan which can lead to improved performance. Constant set annotations can only be specified if the second input and the output type of the UDF are of Tuple data types.

A constant-set annotation is a set of constant field specifications. The constant field specification String "4->3" specifies, that this UDF copies the fourth field of an input tuple to the third field of the output tuple. Field references are zero-indexed.

NOTICE: Constant set annotations are optional, but if given need to be correct. Otherwise, the program might produce wrong results!

Parameters:

constantSet - A list of constant field specification Strings.

Returns:

This operator with an annotated constant field set.

getBroadcastSets

public Map<String,DataSet<?>> getBroadcastSets()

Description copied from interface: UdfOperator

Gets the broadcast sets (name and data set) that have been added to context of the UDF. Broadcast sets are added to a UDF via the method UdfOperator.withBroadcastSet(DataSet, String).

Specified by:

getBroadcastSets in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The broadcast data sets that have been added to this UDF.

getParameters

public Configuration getParameters()

Description copied from interface: UdfOperator

Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration). The configuration is set via the UdfOperator.withParameters(Configuration) method.

Specified by:

getParameters in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The configuration parameters for the UDF.

getSemanticProperties

public SingleInputSemanticProperties getSemanticProperties()

Description copied from interface: UdfOperator

Gets the semantic properties that have been set for the user-defined functions (UDF). This method may return null, if no semantic properties have been set so far.

Specified by:

getSemanticProperties in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The semantic properties of the UDF.

setSemanticProperties

public void setSemanticProperties(SingleInputSemanticProperties properties)

Sets the semantic properties for the user-defined function (UDF). The semantic properties define how fields of tuples and other objects are modified or preserved through this UDF. The configured properties can be retrieved via UdfOperator.getSemanticProperties().

Parameters:

properties - The semantic properties for the UDF.

See Also:

UdfOperator.getSemanticProperties()