org.apache.flink.streaming.api.datastream

Class ConnectedDataStream<IN1,IN2>

java.lang.Object

org.apache.flink.streaming.api.datastream.ConnectedDataStream<IN1,IN2>

Type Parameters:

IN1 - Type of the first input data steam.

IN2 - Type of the second input data stream.

public class ConnectedDataStream<IN1,IN2>
extends Object

The ConnectedDataStream represents a stream for two different data types. It can be used to apply transformations like CoMapFunction on two DataStreams

Field Summary

Fields

Modifier and Type	Field and Description
protected DataStream<IN1>	dataStream1
protected DataStream<IN2>	dataStream2
protected StreamExecutionEnvironment	environment
protected boolean	isGrouped
protected StreamGraph	jobGraphBuilder
protected org.apache.flink.api.java.functions.KeySelector<IN1,?>	keySelector1
protected org.apache.flink.api.java.functions.KeySelector<IN2,?>	keySelector2

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ConnectedDataStream(ConnectedDataStream<IN1,IN2> coDataStream)
protected	ConnectedDataStream(DataStream<IN1> input1, DataStream<IN2> input2)

Method Summary

Modifier and Type	Method and Description
<OUT> SingleOutputStreamOperator<OUT,?>	addGeneralWindowCombine(CoWindowFunction<IN1,IN2,OUT> coWindowFunction, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo, long windowSize, long slideInterval, TimestampWrapper<IN1> timestamp1, TimestampWrapper<IN2> timestamp2)
<F> F	clean(F f)
protected ConnectedDataStream<IN1,IN2>	copy()
<OUT> SingleOutputStreamOperator<OUT,?>	flatMap(CoFlatMapFunction<IN1,IN2,OUT> coFlatMapper) Applies a CoFlatMap transformation on a ConnectedDataStream and maps the output to a common type.
StreamExecutionEnvironment	getExecutionEnvironment()
DataStream<IN1>	getFirst() Returns the first DataStream.
protected <OUT> TwoInputStreamOperator<IN1,IN2,OUT>	getReduceOperator(CoReduceFunction<IN1,IN2,OUT> coReducer)
DataStream<IN2>	getSecond() Returns the second DataStream.
org.apache.flink.api.common.typeinfo.TypeInformation<IN1>	getType1() Gets the type of the first input
org.apache.flink.api.common.typeinfo.TypeInformation<IN2>	getType2() Gets the type of the second input
ConnectedDataStream<IN1,IN2>	groupBy(int[] keyPositions1, int[] keyPositions2) GroupBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	groupBy(int keyPosition1, int keyPosition2) GroupBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	groupBy(org.apache.flink.api.java.functions.KeySelector<IN1,?> keySelector1, org.apache.flink.api.java.functions.KeySelector<IN2,?> keySelector2) GroupBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	groupBy(String[] fields1, String[] fields2) GroupBy operation for connected data stream using key expressions.
ConnectedDataStream<IN1,IN2>	groupBy(String field1, String field2) GroupBy operation for connected data stream using key expressions.
<OUT> SingleOutputStreamOperator<OUT,?>	map(CoMapFunction<IN1,IN2,OUT> coMapper) Applies a CoMap transformation on a ConnectedDataStream and maps the output to a common type.
ConnectedDataStream<IN1,IN2>	partitionByHash(int[] keyPositions1, int[] keyPositions2) PartitionBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	partitionByHash(int keyPosition1, int keyPosition2) PartitionBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	partitionByHash(org.apache.flink.api.java.functions.KeySelector<IN1,?> keySelector1, org.apache.flink.api.java.functions.KeySelector<IN2,?> keySelector2) PartitionBy operation for connected data stream.
ConnectedDataStream<IN1,IN2>	partitionByHash(String[] fields1, String[] fields2) PartitionBy operation for connected data stream using key expressions.
ConnectedDataStream<IN1,IN2>	partitionByHash(String field1, String field2) PartitionBy operation for connected data stream using key expressions.
<OUT> SingleOutputStreamOperator<OUT,?>	reduce(CoReduceFunction<IN1,IN2,OUT> coReducer) Applies a reduce transformation on a ConnectedDataStream and maps the outputs to a common type.
<OUT> SingleOutputStreamOperator<OUT,?>	transform(String functionName, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo, TwoInputStreamOperator<IN1,IN2,OUT> operator)
<OUT> SingleOutputStreamOperator<OUT,?>	windowReduce(CoWindowFunction<IN1,IN2,OUT> coWindowFunction, long windowSize, long slideInterval) Applies a CoWindow transformation on the connected DataStreams.
<OUT> SingleOutputStreamOperator<OUT,?>	windowReduce(CoWindowFunction<IN1,IN2,OUT> coWindowFunction, long windowSize, long slideInterval, TimestampWrapper<IN1> timestamp1, TimestampWrapper<IN2> timestamp2) Applies a CoWindow transformation on the connected DataStreams.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

environment

protected StreamExecutionEnvironment environment

jobGraphBuilder

protected StreamGraph jobGraphBuilder

dataStream1

protected DataStream<IN1> dataStream1

dataStream2

protected DataStream<IN2> dataStream2

isGrouped

protected boolean isGrouped

keySelector1

protected org.apache.flink.api.java.functions.KeySelector<IN1,?> keySelector1

keySelector2

protected org.apache.flink.api.java.functions.KeySelector<IN2,?> keySelector2

Constructor Detail

ConnectedDataStream

protected ConnectedDataStream(DataStream<IN1> input1,
                              DataStream<IN2> input2)

ConnectedDataStream

protected ConnectedDataStream(ConnectedDataStream<IN1,IN2> coDataStream)

Method Detail

clean

public <F> F clean(F f)

getExecutionEnvironment

public StreamExecutionEnvironment getExecutionEnvironment()

getFirst

public DataStream<IN1> getFirst()

Returns the first DataStream.

Returns:

The first DataStream.

getSecond

public DataStream<IN2> getSecond()

Returns the second DataStream.

Returns:

The second DataStream.

getType1

public org.apache.flink.api.common.typeinfo.TypeInformation<IN1> getType1()

Gets the type of the first input

Returns:

The type of the first input

getType2

public org.apache.flink.api.common.typeinfo.TypeInformation<IN2> getType2()

Gets the type of the second input

Returns:

The type of the second input

groupBy

public ConnectedDataStream<IN1,IN2> groupBy(int keyPosition1,
                                            int keyPosition2)

GroupBy operation for connected data stream. Groups the elements of input1 and input2 according to keyPosition1 and keyPosition2. Used for applying function on grouped data streams for example reduce(org.apache.flink.streaming.api.functions.co.CoReduceFunction<IN1, IN2, OUT>)

Parameters:

keyPosition1 - The field used to compute the hashcode of the elements in the first input stream.

keyPosition2 - The field used to compute the hashcode of the elements in the second input stream.

Returns:

The grouped ConnectedDataStream

groupBy

public ConnectedDataStream<IN1,IN2> groupBy(int[] keyPositions1,
                                            int[] keyPositions2)

GroupBy operation for connected data stream. Groups the elements of input1 and input2 according to keyPositions1 and keyPositions2. Used for applying function on grouped data streams for example reduce(org.apache.flink.streaming.api.functions.co.CoReduceFunction<IN1, IN2, OUT>)

Parameters:

keyPositions1 - The fields used to group the first input stream.

keyPositions2 - The fields used to group the second input stream.

Returns:

The grouped ConnectedDataStream

groupBy

public ConnectedDataStream<IN1,IN2> groupBy(String field1,
                                            String field2)

GroupBy operation for connected data stream using key expressions. Groups the elements of input1 and input2 according to field1 and field2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field1 - The grouping expression for the first input

field2 - The grouping expression for the second input

Returns:

The grouped ConnectedDataStream

groupBy

public ConnectedDataStream<IN1,IN2> groupBy(String[] fields1,
                                            String[] fields2)

GroupBy operation for connected data stream using key expressions. Groups the elements of input1 and input2 according to fields1 and fields2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

fields1 - The grouping expressions for the first input

fields2 - The grouping expressions for the second input

Returns:

The grouped ConnectedDataStream

groupBy

public ConnectedDataStream<IN1,IN2> groupBy(org.apache.flink.api.java.functions.KeySelector<IN1,?> keySelector1,
                                            org.apache.flink.api.java.functions.KeySelector<IN2,?> keySelector2)

GroupBy operation for connected data stream. Groups the elements of input1 and input2 using keySelector1 and keySelector2. Used for applying function on grouped data streams for example reduce(org.apache.flink.streaming.api.functions.co.CoReduceFunction<IN1, IN2, OUT>)

Parameters:

keySelector1 - The KeySelector used for grouping the first input

keySelector2 - The KeySelector used for grouping the second input

Returns:

The partitioned ConnectedDataStream

partitionByHash

public ConnectedDataStream<IN1,IN2> partitionByHash(int keyPosition1,
                                                    int keyPosition2)

PartitionBy operation for connected data stream. Partitions the elements of input1 and input2 according to keyPosition1 and keyPosition2.

Parameters:

keyPosition1 - The field used to compute the hashcode of the elements in the first input stream.

keyPosition2 - The field used to compute the hashcode of the elements in the second input stream.

Returns:

The partitioned ConnectedDataStream

partitionByHash

public ConnectedDataStream<IN1,IN2> partitionByHash(int[] keyPositions1,
                                                    int[] keyPositions2)

PartitionBy operation for connected data stream. Partitions the elements of input1 and input2 according to keyPositions1 and keyPositions2.

Parameters:

keyPositions1 - The fields used to group the first input stream.

keyPositions2 - The fields used to group the second input stream.

Returns:

The partitioned ConnectedDataStream

partitionByHash

public ConnectedDataStream<IN1,IN2> partitionByHash(String field1,
                                                    String field2)

PartitionBy operation for connected data stream using key expressions. Partitions the elements of input1 and input2 according to field1 and field2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreams underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()"

Parameters:

field1 - The partitioning expressions for the first input

field2 - The partitioning expressions for the second input

Returns:

The partitioned ConnectedDataStream

partitionByHash

public ConnectedDataStream<IN1,IN2> partitionByHash(String[] fields1,
                                                    String[] fields2)

PartitionBy operation for connected data stream using key expressions. Partitions the elements of input1 and input2 according to fields1 and fields2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreams underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()"

Parameters:

fields1 - The partitioning expressions for the first input

fields2 - The partitioning expressions for the second input

Returns:

The partitioned ConnectedDataStream

partitionByHash

public ConnectedDataStream<IN1,IN2> partitionByHash(org.apache.flink.api.java.functions.KeySelector<IN1,?> keySelector1,
                                                    org.apache.flink.api.java.functions.KeySelector<IN2,?> keySelector2)

PartitionBy operation for connected data stream. Partitions the elements of input1 and input2 using keySelector1 and keySelector2.

Parameters:

keySelector1 - The KeySelector used for partitioning the first input

keySelector2 - The KeySelector used for partitioning the second input

Returns:

@return The partitioned ConnectedDataStream

map

public <OUT> SingleOutputStreamOperator<OUT,?> map(CoMapFunction<IN1,IN2,OUT> coMapper)

Applies a CoMap transformation on a ConnectedDataStream and maps the output to a common type. The transformation calls a CoMapFunction.map1(IN1) for each element of the first input and CoMapFunction.map2(IN2) for each element of the second input. Each CoMapFunction call returns exactly one element.

Parameters:

coMapper - The CoMapFunction used to jointly transform the two input DataStreams

Returns:

The transformed DataStream

flatMap

public <OUT> SingleOutputStreamOperator<OUT,?> flatMap(CoFlatMapFunction<IN1,IN2,OUT> coFlatMapper)

Applies a CoFlatMap transformation on a ConnectedDataStream and maps the output to a common type. The transformation calls a CoFlatMapFunction.flatMap1(IN1, org.apache.flink.util.Collector<OUT>) for each element of the first input and CoFlatMapFunction.flatMap2(IN2, org.apache.flink.util.Collector<OUT>) for each element of the second input. Each CoFlatMapFunction call returns any number of elements including none.

Parameters:

coFlatMapper - The CoFlatMapFunction used to jointly transform the two input DataStreams

Returns:

The transformed DataStream

reduce

public <OUT> SingleOutputStreamOperator<OUT,?> reduce(CoReduceFunction<IN1,IN2,OUT> coReducer)

Applies a reduce transformation on a ConnectedDataStream and maps the outputs to a common type. If the ConnectedDataStream is batched or windowed then the reduce transformation is applied on every sliding batch/window of the data stream. If the connected data stream is grouped then the reducer is applied on every group of elements sharing the same key. This type of reduce is much faster than reduceGroup since the reduce function can be applied incrementally.

Parameters:

coReducer - The CoReduceFunction that will be called for every element of the inputs.

Returns:

The transformed DataStream.

windowReduce

public <OUT> SingleOutputStreamOperator<OUT,?> windowReduce(CoWindowFunction<IN1,IN2,OUT> coWindowFunction,
                                                            long windowSize,
                                                            long slideInterval)

Applies a CoWindow transformation on the connected DataStreams. The transformation calls the CoWindowFunction.coWindow(java.util.List<IN1>, java.util.List<IN2>, org.apache.flink.util.Collector<O>) method for for time aligned windows of the two data streams. System time is used as default to compute windows.

Parameters:

coWindowFunction - The CoWindowFunction that will be applied for the time windows.

windowSize - Size of the windows that will be aligned for both streams in milliseconds.

slideInterval - After every function call the windows will be slid by this interval.

Returns:

The transformed DataStream.

windowReduce

public <OUT> SingleOutputStreamOperator<OUT,?> windowReduce(CoWindowFunction<IN1,IN2,OUT> coWindowFunction,
                                                            long windowSize,
                                                            long slideInterval,
                                                            TimestampWrapper<IN1> timestamp1,
                                                            TimestampWrapper<IN2> timestamp2)

Applies a CoWindow transformation on the connected DataStreams. The transformation calls the CoWindowFunction.coWindow(java.util.List<IN1>, java.util.List<IN2>, org.apache.flink.util.Collector<O>) method for time aligned windows of the two data streams. The user can implement their own time stamps or use the system time by default.

Parameters:

coWindowFunction - The CoWindowFunction that will be applied for the time windows.

windowSize - Size of the windows that will be aligned for both streams. If system time is used it is milliseconds. User defined time stamps are assumed to be monotonically increasing.

slideInterval - After every function call the windows will be slid by this interval.

timestamp1 - User defined time stamps for the first input.

timestamp2 - User defined time stamps for the second input.

Returns:

The transformed DataStream.

getReduceOperator

protected <OUT> TwoInputStreamOperator<IN1,IN2,OUT> getReduceOperator(CoReduceFunction<IN1,IN2,OUT> coReducer)

addGeneralWindowCombine

public <OUT> SingleOutputStreamOperator<OUT,?> addGeneralWindowCombine(CoWindowFunction<IN1,IN2,OUT> coWindowFunction,
                                                                       org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                                                       long windowSize,
                                                                       long slideInterval,
                                                                       TimestampWrapper<IN1> timestamp1,
                                                                       TimestampWrapper<IN2> timestamp2)

transform

public <OUT> SingleOutputStreamOperator<OUT,?> transform(String functionName,
                                                         org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                                         TwoInputStreamOperator<IN1,IN2,OUT> operator)

copy

protected ConnectedDataStream<IN1,IN2> copy()