org.apache.flink.api.java.operators

Class SingleInputUdfOperator<IN,OUT,O extends SingleInputUdfOperator<IN,OUT,O>>

java.lang.Object

org.apache.flink.api.java.DataSet<OUT>

org.apache.flink.api.java.operators.Operator<OUT,O>

org.apache.flink.api.java.operators.SingleInputOperator<IN,OUT,O>

org.apache.flink.api.java.operators.SingleInputUdfOperator<IN,OUT,O>

Type Parameters:

IN - The data type of the input data set.

OUT - The data type of the returned data set.

All Implemented Interfaces:

UdfOperator<O>

Direct Known Subclasses:

FilterOperator, FlatMapOperator, GroupCombineOperator, GroupReduceOperator, MapOperator, MapPartitionOperator, ReduceOperator

public abstract class SingleInputUdfOperator<IN,OUT,O extends SingleInputUdfOperator<IN,OUT,O>>
extends SingleInputOperator<IN,OUT,O>
implements UdfOperator<O>

The SingleInputUdfOperator is the base class of all unary operators that execute user-defined functions (UDFs). The UDFs encapsulated by this operator are naturally UDFs that have one input (such as RichMapFunction or RichReduceFunction).

This class encapsulates utilities for the UDFs, such as broadcast variables, parameterization through configuration objects, and semantic properties.

Field Summary

Fields inherited from class org.apache.flink.api.java.operators.Operator

name, parallelism

Fields inherited from class org.apache.flink.api.java.DataSet

context

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SingleInputUdfOperator(DataSet<IN> input, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType) Creates a new operators with the given data set as input.

Method Summary

Modifier and Type	Method and Description
protected org.apache.flink.api.common.operators.SingleInputSemanticProperties	extractSemanticAnnotations(Class<?> udfClass)
protected boolean	getAnalyzedUdfSemanticsFlag()
Map<String,DataSet<?>>	getBroadcastSets() Gets the broadcast sets (name and data set) that have been added to context of the UDF.
protected abstract org.apache.flink.api.common.functions.Function	getFunction()
org.apache.flink.configuration.Configuration	getParameters() Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration).
org.apache.flink.api.common.operators.SingleInputSemanticProperties	getSemanticProperties() Gets the semantic properties that have been set for the user-defined functions (UDF).
O	returns(Class<OUT> typeClass) Adds a type information hint about the return type of this operator.
O	returns(String typeInfoString) Adds a type information hint about the return type of this operator.
O	returns(org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo) Adds a type information hint about the return type of this operator.
protected void	setAnalyzedUdfSemanticsFlag()
void	setSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties properties) Sets the semantic properties for the user-defined function (UDF).
protected boolean	udfWithForwardedFieldsAnnotation(Class<?> udfClass)
O	withBroadcastSet(DataSet<?> data, String name) Adds a certain data set as a broadcast set to this operator.
O	withForwardedFields(String... forwardedFields) Adds semantic information about forwarded fields of the user-defined function.
O	withParameters(org.apache.flink.configuration.Configuration parameters) Sets the configuration parameters for the UDF.

Methods inherited from class org.apache.flink.api.java.operators.SingleInputOperator

getInput, getInputType, translateToDataFlow

Methods inherited from class org.apache.flink.api.java.operators.Operator

getName, getParallelism, getResultType, name, setParallelism

Methods inherited from class org.apache.flink.api.java.DataSet

aggregate, checkSameExecutionContext, clean, coGroup, collect, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, fillInType, filter, first, flatMap, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, project, rebalance, reduce, reduceGroup, runOperation, sortPartition, sortPartition, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleInputUdfOperator

protected SingleInputUdfOperator(DataSet<IN> input,
                                 org.apache.flink.api.common.typeinfo.TypeInformation<OUT> resultType)

Creates a new operators with the given data set as input. The given result type describes the data type of the elements in the data set produced by the operator.

Parameters:

input - The data set that is the input to the operator.

resultType - The type of the elements in the resulting data set.

Method Detail

getFunction

protected abstract org.apache.flink.api.common.functions.Function getFunction()

withParameters

public O withParameters(org.apache.flink.configuration.Configuration parameters)

Description copied from interface: UdfOperator

Sets the configuration parameters for the UDF. These are optional parameters that are passed to the UDF in the AbstractRichFunction.open(Configuration) method.

Specified by:

withParameters in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Parameters:

parameters - The configuration parameters for the UDF.

Returns:

The operator itself, to allow chaining function calls.

withBroadcastSet

public O withBroadcastSet(DataSet<?> data,
                          String name)

Description copied from interface: UdfOperator

Adds a certain data set as a broadcast set to this operator. Broadcasted data sets are available at all parallel instances of this operator. A broadcast data set is registered under a certain name, and can be retrieved under that name from the operators runtime context via RuntimeContext.getBroadcastVariable(String). The runtime context itself is available in all UDFs via AbstractRichFunction.getRuntimeContext().

Specified by:

withBroadcastSet in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Parameters:

data - The data set to be broadcasted.

name - The name under which the broadcast data set retrieved.

Returns:

The operator itself, to allow chaining function calls.

withForwardedFields

public O withForwardedFields(String... forwardedFields)

Adds semantic information about forwarded fields of the user-defined function. The forwarded fields information declares fields which are never modified by the function and which are forwarded at the same position to the output or unchanged copied to another position in the output.

Fields that are forwarded at the same position are specified by their position. The specified position must be valid for the input and output data type and have the same type. For example withForwardedFields("f2") declares that the third field of a Java input tuple is copied to the third field of an output tuple.

Fields which are unchanged copied to another position in the output are declared by specifying the source field reference in the input and the target field reference in the output. withForwardedFields("f0->f2") denotes that the first field of the Java input tuple is unchanged copied to the third field of the Java output tuple. When using a wildcard ("*") ensure that the number of declared fields and their types in input and output type match.

Multiple forwarded fields can be annotated in one (withForwardedFields("f2; f3->f0; f4")) or separate Strings (withForwardedFields("f2", "f3->f0", "f4")). Please refer to the JavaDoc of Function or Flink's documentation for details on field references such as nested fields and wildcard.

It is not possible to override existing semantic information about forwarded fields which was for example added by a FunctionAnnotation.ForwardedFields class annotation.

NOTE: Adding semantic information for functions is optional! If used correctly, semantic information can help the Flink optimizer to generate more efficient execution plans. However, incorrect semantic information can cause the optimizer to generate incorrect execution plans which compute wrong results! So be careful when adding semantic information.

Parameters:

forwardedFields - A list of field forward expressions.

Returns:

This operator with annotated forwarded field information.

See Also:

FunctionAnnotation, FunctionAnnotation.ForwardedFields

returns

public O returns(String typeInfoString)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes a type information string that will be parsed. A type information string can contain the following types:

• Basic types such as Integer, String, etc.
• Basic type arrays such as Integer[], String[], etc.
• Tuple types such as Tuple1<TYPE0>, Tuple2<TYPE0, TYPE1>, etc.
• Pojo types such as org.my.MyPojo<myFieldName=TYPE0,myFieldName2=TYPE1>, etc.
• Generic types such as java.lang.Class, etc.
• Custom type arrays such as org.my.CustomClass[], org.my.CustomClass$StaticInnerClass[], etc.
• Value types such as DoubleValue, StringValue, IntegerValue, etc.
• Tuple array types such as Tuple2<TYPE0,TYPE1>[], etc.
• Writable types such as Writable<org.my.CustomWritable>
• Enum types such as Enum<org.my.CustomEnum>

Example: "Tuple2<String,Tuple2<Integer,org.my.MyJob$Pojo<word=String>>>"

Parameters:

typeInfoString - type information string to be parsed

Returns:

This operator with a given return type hint.

returns

public O returns(org.apache.flink.api.common.typeinfo.TypeInformation<OUT> typeInfo)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes an instance of TypeInformation such as:

• BasicTypeInfo
• BasicArrayTypeInfo
• TupleTypeInfo
• PojoTypeInfo
• WritableTypeInfo
• ValueTypeInfo
• etc.

Parameters:

typeInfo - type information as a return type hint

Returns:

This operator with a given return type hint.

returns

public O returns(Class<OUT> typeClass)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes a class that will be analyzed by Flink's type extraction capabilities.

Examples for classes are:

• Basic types such as Integer.class, String.class, etc.
• POJOs such as MyPojo.class
• Classes that extend tuples. Classes like Tuple1.class,Tuple2.class, etc. are not sufficient.
• Arrays such as String[].class, etc.

Parameters:

typeClass - class as a return type hint

Returns:

This operator with a given return type hint.

getBroadcastSets

public Map<String,DataSet<?>> getBroadcastSets()

Description copied from interface: UdfOperator

Gets the broadcast sets (name and data set) that have been added to context of the UDF. Broadcast sets are added to a UDF via the method UdfOperator.withBroadcastSet(DataSet, String).

Specified by:

getBroadcastSets in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The broadcast data sets that have been added to this UDF.

getParameters

public org.apache.flink.configuration.Configuration getParameters()

Description copied from interface: UdfOperator

Gets the configuration parameters that will be passed to the UDF's open method AbstractRichFunction.open(Configuration). The configuration is set via the UdfOperator.withParameters(Configuration) method.

Specified by:

getParameters in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The configuration parameters for the UDF.

getSemanticProperties

public org.apache.flink.api.common.operators.SingleInputSemanticProperties getSemanticProperties()

Description copied from interface: UdfOperator

Gets the semantic properties that have been set for the user-defined functions (UDF).

Specified by:

getSemanticProperties in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>

Returns:

The semantic properties of the UDF.

setSemanticProperties

public void setSemanticProperties(org.apache.flink.api.common.operators.SingleInputSemanticProperties properties)

Sets the semantic properties for the user-defined function (UDF). The semantic properties define how fields of tuples and other objects are modified or preserved through this UDF. The configured properties can be retrieved via UdfOperator.getSemanticProperties().

Parameters:

properties - The semantic properties for the UDF.

See Also:

UdfOperator.getSemanticProperties()

getAnalyzedUdfSemanticsFlag

protected boolean getAnalyzedUdfSemanticsFlag()

setAnalyzedUdfSemanticsFlag

protected void setAnalyzedUdfSemanticsFlag()

extractSemanticAnnotations

protected org.apache.flink.api.common.operators.SingleInputSemanticProperties extractSemanticAnnotations(Class<?> udfClass)

udfWithForwardedFieldsAnnotation

protected boolean udfWithForwardedFieldsAnnotation(Class<?> udfClass)