org.apache.flink.streaming.api.datastream

Class WindowedDataStream<OUT>

java.lang.Object

org.apache.flink.streaming.api.datastream.WindowedDataStream<OUT>

Type Parameters:

OUT - The output type of the WindowedDataStream

Direct Known Subclasses:

DiscretizedStream

public class WindowedDataStream<OUT>
extends Object

A WindowedDataStream represents a data stream that has been discretised into windows. User defined function such as reduceWindow(ReduceFunction), mapWindow(WindowMapFunction) or aggregations can be applied to the windows. The results of these transformations are also WindowedDataStreams of the same discretisation unit.

Field Summary

Fields

Modifier and Type	Field and Description
protected DataStream<OUT>	dataStream
protected org.apache.flink.api.java.functions.KeySelector<OUT,?>	discretizerKey
protected WindowingHelper<OUT>	evictionHelper
protected org.apache.flink.api.java.functions.KeySelector<OUT,?>	groupByKey
protected boolean	isLocal
protected WindowingHelper<OUT>	triggerHelper
protected EvictionPolicy<OUT>	userEvicter
protected TriggerPolicy<OUT>	userTrigger

Constructor Summary

Constructors

Modifier	Constructor and Description
	WindowedDataStream()
protected	WindowedDataStream(DataStream<OUT> dataStream, TriggerPolicy<OUT> trigger, EvictionPolicy<OUT> evicter)
protected	WindowedDataStream(DataStream<OUT> dataStream, WindowingHelper<OUT> policyHelper)
protected	WindowedDataStream(WindowedDataStream<OUT> windowedDataStream)

Method Summary

Modifier and Type	Method and Description
<F> F	clean(F f)
protected WindowedDataStream<OUT>	copy()
WindowedDataStream<OUT>	every(WindowingHelper policyHelper) Defines the slide size (trigger frequency) for the windowed data stream.
DataStream<OUT>	flatten() Flattens the results of the window computations and streams out the window elements.
<R> DiscretizedStream<R>	foldWindow(R initialValue, org.apache.flink.api.common.functions.FoldFunction<OUT,R> foldFunction) Applies a fold transformation on the windowed data stream by folding the current window at every trigger.The user can also extend the RichFoldFunction to gain access to other features provided by the RichFunction interface.
<R> DiscretizedStream<R>	foldWindow(R initialValue, org.apache.flink.api.common.functions.FoldFunction<OUT,R> foldFunction, org.apache.flink.api.common.typeinfo.TypeInformation<R> outType) Applies a fold transformation on the windowed data stream by folding the current window at every trigger.The user can also extend the RichFoldFunction to gain access to other features provided by the RichFunction interface.
protected Class<?>	getClassAtPos(int pos)
DataStream<StreamWindow<OUT>>	getDiscretizedStream() Returns the DataStream of StreamWindows which represent the discretised stream.
protected EvictionPolicy<OUT>	getEviction()
org.apache.flink.api.common.ExecutionConfig	getExecutionConfig()
protected TriggerPolicy<OUT>	getTrigger()
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getType() Gets the output type.
WindowedDataStream<OUT>	groupBy(int... fields) Groups the elements of the WindowedDataStream by the given key positions.
WindowedDataStream<OUT>	groupBy(org.apache.flink.api.java.functions.KeySelector<OUT,?> keySelector) Groups the elements of the WindowedDataStream using the given KeySelector.
WindowedDataStream<OUT>	groupBy(String... fields) Groups the elements of the WindowedDataStream by the given field expressions.
protected boolean	isGrouped()
WindowedDataStream<OUT>	local() Sets the window discretisation local, meaning that windows will be created in parallel at environment parallelism.
<R> DiscretizedStream<R>	mapWindow(WindowMapFunction<OUT,R> windowMapFunction) Applies a mapWindow transformation on the windowed data stream by calling the mapWindow function on the window at every trigger.
<R> DiscretizedStream<R>	mapWindow(WindowMapFunction<OUT,R> windowMapFunction, org.apache.flink.api.common.typeinfo.TypeInformation<R> outType) Applies a mapWindow transformation on the windowed data stream by calling the mapWindow function on the window at every trigger.
WindowedDataStream<OUT>	max(int positionToMax) Applies an aggregation that gives the maximum value of every window of the data stream at the given position.
WindowedDataStream<OUT>	max(String field) Applies an aggregation that that gives the maximum value of the pojo data stream at the given field expression for every window.
WindowedDataStream<OUT>	maxBy(int positionToMaxBy) Applies an aggregation that gives the maximum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	maxBy(int positionToMaxBy, boolean first) Applies an aggregation that gives the maximum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	maxBy(String positionToMaxBy) Applies an aggregation that gives the maximum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	maxBy(String field, boolean first) Applies an aggregation that that gives the maximum element of the pojo data stream by the given field expression for every window.
WindowedDataStream<OUT>	min(int positionToMin) Applies an aggregation that that gives the minimum value of every window of the data stream at the given position.
WindowedDataStream<OUT>	min(String field) Applies an aggregation that that gives the minimum value of the pojo data stream at the given field expression for every window.
WindowedDataStream<OUT>	minBy(int positionToMinBy) Applies an aggregation that gives the minimum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	minBy(int positionToMinBy, boolean first) Applies an aggregation that gives the minimum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	minBy(String positionToMinBy) Applies an aggregation that gives the minimum element of every window of the data stream by the given position.
WindowedDataStream<OUT>	minBy(String field, boolean first) Applies an aggregation that that gives the minimum element of the pojo data stream by the given field expression for every window.
DiscretizedStream<OUT>	reduceWindow(org.apache.flink.api.common.functions.ReduceFunction<OUT> reduceFunction) Applies a reduce transformation on the windowed data stream by reducing the current window at every trigger.The user can also extend the RichReduceFunction to gain access to other features provided by the RichFunction interface.
WindowedDataStream<OUT>	sum(int positionToSum) Applies an aggregation that sums every window of the data stream at the given position.
WindowedDataStream<OUT>	sum(String field) Applies an aggregation that sums every window of the pojo data stream at the given field for every window.
protected DiscretizedStream<OUT>	timeReduce(org.apache.flink.api.common.functions.ReduceFunction<OUT> reduceFunction) Dedicated method for applying parallel time reduce transformations on windows

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dataStream

protected DataStream<OUT> dataStream

isLocal

protected boolean isLocal

discretizerKey

protected org.apache.flink.api.java.functions.KeySelector<OUT,?> discretizerKey

groupByKey

protected org.apache.flink.api.java.functions.KeySelector<OUT,?> groupByKey

triggerHelper

protected WindowingHelper<OUT> triggerHelper

evictionHelper

protected WindowingHelper<OUT> evictionHelper

userTrigger

protected TriggerPolicy<OUT> userTrigger

userEvicter

protected EvictionPolicy<OUT> userEvicter

Constructor Detail

WindowedDataStream

protected WindowedDataStream(DataStream<OUT> dataStream,
                             WindowingHelper<OUT> policyHelper)

WindowedDataStream

protected WindowedDataStream(DataStream<OUT> dataStream,
                             TriggerPolicy<OUT> trigger,
                             EvictionPolicy<OUT> evicter)

WindowedDataStream

protected WindowedDataStream(WindowedDataStream<OUT> windowedDataStream)

WindowedDataStream

public WindowedDataStream()

Method Detail

every

public WindowedDataStream<OUT> every(WindowingHelper policyHelper)

Defines the slide size (trigger frequency) for the windowed data stream. This controls how often the user defined function will be triggered on the window.

For example to get a window of 5 elements with a slide of 2 seconds use:

ds.window(Count.of(5)).every(Time.of(2,TimeUnit.SECONDS))

The user function in this case will be called on the 5 most recent elements every 2 seconds

Parameters:

policyHelper - The policy that define the triggering frequency

Returns:

The windowed data stream with triggering set

groupBy

public WindowedDataStream<OUT> groupBy(int... fields)

Groups the elements of the WindowedDataStream by the given key positions. The window sizes (evictions) and slide sizes (triggers) will be calculated on the whole stream (in a global fashion), but the user defined functions will be applied on a per group basis.

To get windows and triggers on a per group basis apply the DataStream.window(org.apache.flink.streaming.api.windowing.helper.WindowingHelper) operator on an already grouped data stream.

Parameters:

fields - The position of the fields to group by.

Returns:

The grouped WindowedDataStream

groupBy

public WindowedDataStream<OUT> groupBy(String... fields)

Groups the elements of the WindowedDataStream by the given field expressions. The window sizes (evictions) and slide sizes (triggers) will be calculated on the whole stream (in a global fashion), but the user defined functions will be applied on a per group basis.

To get windows and triggers on a per group basis apply the DataStream.window(org.apache.flink.streaming.api.windowing.helper.WindowingHelper) operator on an already grouped data stream.

A field expression is either the name of a public field or a getter method with parentheses of the stream's underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

fields - The fields to group by

Returns:

The grouped WindowedDataStream

groupBy

public WindowedDataStream<OUT> groupBy(org.apache.flink.api.java.functions.KeySelector<OUT,?> keySelector)

Groups the elements of the WindowedDataStream using the given KeySelector. The window sizes (evictions) and slide sizes (triggers) will be calculated on the whole stream (in a global fashion), but the user defined functions will be applied on a per group basis.

To get windows and triggers on a per group basis apply the DataStream.window(org.apache.flink.streaming.api.windowing.helper.WindowingHelper) operator on an already grouped data stream.

Parameters:

keySelector - The keySelector used to extract the key for grouping.

Returns:

The grouped WindowedDataStream

local

public WindowedDataStream<OUT> local()

Sets the window discretisation local, meaning that windows will be created in parallel at environment parallelism.

Returns:

The WindowedDataStream with local discretisation

getDiscretizedStream

public DataStream<StreamWindow<OUT>> getDiscretizedStream()

Returns the DataStream of StreamWindows which represent the discretised stream. There is no ordering guarantee for the received windows.

Returns:

The discretised stream

flatten

public DataStream<OUT> flatten()

Flattens the results of the window computations and streams out the window elements.

Returns:

The data stream consisting of the individual records.

reduceWindow

public DiscretizedStream<OUT> reduceWindow(org.apache.flink.api.common.functions.ReduceFunction<OUT> reduceFunction)

Applies a reduce transformation on the windowed data stream by reducing the current window at every trigger.The user can also extend the RichReduceFunction to gain access to other features provided by the RichFunction interface.

Parameters:

reduceFunction - The reduce function that will be applied to the windows.

Returns:

The transformed DataStream

foldWindow

public <R> DiscretizedStream<R> foldWindow(R initialValue,
                                           org.apache.flink.api.common.functions.FoldFunction<OUT,R> foldFunction,
                                           org.apache.flink.api.common.typeinfo.TypeInformation<R> outType)

Applies a fold transformation on the windowed data stream by folding the current window at every trigger.The user can also extend the RichFoldFunction to gain access to other features provided by the RichFunction interface. This version of foldWindow uses user supplied typeinformation for serializaton. Use this only when the system is unable to detect type information.

Parameters:

foldFunction - The fold function that will be applied to the windows.

initialValue - Initial value given to foldFunction

outType - The output type of the operator

Returns:

The transformed DataStream

foldWindow

public <R> DiscretizedStream<R> foldWindow(R initialValue,
                                           org.apache.flink.api.common.functions.FoldFunction<OUT,R> foldFunction)

Applies a fold transformation on the windowed data stream by folding the current window at every trigger.The user can also extend the RichFoldFunction to gain access to other features provided by the RichFunction interface.

Parameters:

foldFunction - The fold function that will be applied to the windows.

initialValue - Initial value given to foldFunction

Returns:

The transformed DataStream

mapWindow

public <R> DiscretizedStream<R> mapWindow(WindowMapFunction<OUT,R> windowMapFunction)

Applies a mapWindow transformation on the windowed data stream by calling the mapWindow function on the window at every trigger. In contrast with the standard binary reducer, with mapWindow allows the user to access all elements of the window at the same time through the iterable interface. The user can also extend the RichWindowMapFunction to gain access to other features provided by the RichFunction interface.

Parameters:

windowMapFunction - The function that will be applied to the windows.

Returns:

The transformed DataStream

mapWindow

public <R> DiscretizedStream<R> mapWindow(WindowMapFunction<OUT,R> windowMapFunction,
                                          org.apache.flink.api.common.typeinfo.TypeInformation<R> outType)

Applies a mapWindow transformation on the windowed data stream by calling the mapWindow function on the window at every trigger. In contrast with the standard binary reducer, with mapWindow allows the user to access all elements of the window at the same time through the iterable interface. The user can also extend the RichWindowMapFunction to gain access to other features provided by the RichFunction interface.

This version of mapWindow uses user supplied typeinformation for serializaton. Use this only when the system is unable to detect type information.

Parameters:

windowMapFunction - The function that will be applied to the windows.

outType - The output type of the operator.

Returns:

The transformed DataStream

timeReduce

protected DiscretizedStream<OUT> timeReduce(org.apache.flink.api.common.functions.ReduceFunction<OUT> reduceFunction)

Dedicated method for applying parallel time reduce transformations on windows

Parameters:

reduceFunction - Reduce function to apply

Returns:

The transformed stream

sum

public WindowedDataStream<OUT> sum(int positionToSum)

Applies an aggregation that sums every window of the data stream at the given position.

Parameters:

positionToSum - The position in the tuple/array to sum

Returns:

The transformed DataStream.

sum

public WindowedDataStream<OUT> sum(String field)

Applies an aggregation that sums every window of the pojo data stream at the given field for every window.

A field expression is either the name of a public field or a getter method with parentheses of the stream's underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field - The field to sum

Returns:

The transformed DataStream.

min

public WindowedDataStream<OUT> min(int positionToMin)

Applies an aggregation that that gives the minimum value of every window of the data stream at the given position.

Parameters:

positionToMin - The position to minimize

Returns:

The transformed DataStream.

min

public WindowedDataStream<OUT> min(String field)

Applies an aggregation that that gives the minimum value of the pojo data stream at the given field expression for every window.

A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field - The field expression based on which the aggregation will be applied.

Returns:

The transformed DataStream.

minBy

public WindowedDataStream<OUT> minBy(int positionToMinBy)

Applies an aggregation that gives the minimum element of every window of the data stream by the given position. If more elements have the same minimum value the operator returns the first element by default.

Parameters:

positionToMinBy - The position to minimize by

Returns:

The transformed DataStream.

minBy

public WindowedDataStream<OUT> minBy(String positionToMinBy)

Applies an aggregation that gives the minimum element of every window of the data stream by the given position. If more elements have the same minimum value the operator returns the first element by default.

Parameters:

positionToMinBy - The position to minimize by

Returns:

The transformed DataStream.

minBy

public WindowedDataStream<OUT> minBy(int positionToMinBy,
                                     boolean first)

Applies an aggregation that gives the minimum element of every window of the data stream by the given position. If more elements have the same minimum value the operator returns either the first or last one depending on the parameter setting.

Parameters:

positionToMinBy - The position to minimize

first - If true, then the operator return the first element with the minimum value, otherwise returns the last

Returns:

The transformed DataStream.

minBy

public WindowedDataStream<OUT> minBy(String field,
                                     boolean first)

Applies an aggregation that that gives the minimum element of the pojo data stream by the given field expression for every window. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field - The field expression based on which the aggregation will be applied.

first - If True then in case of field equality the first object will be returned

Returns:

The transformed DataStream.

max

public WindowedDataStream<OUT> max(int positionToMax)

Applies an aggregation that gives the maximum value of every window of the data stream at the given position.

Parameters:

positionToMax - The position to maximize

Returns:

The transformed DataStream.

max

public WindowedDataStream<OUT> max(String field)

Applies an aggregation that that gives the maximum value of the pojo data stream at the given field expression for every window. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field - The field expression based on which the aggregation will be applied.

Returns:

The transformed DataStream.

maxBy

public WindowedDataStream<OUT> maxBy(int positionToMaxBy)

Applies an aggregation that gives the maximum element of every window of the data stream by the given position. If more elements have the same maximum value the operator returns the first by default.

Parameters:

positionToMaxBy - The position to maximize by

Returns:

The transformed DataStream.

maxBy

public WindowedDataStream<OUT> maxBy(String positionToMaxBy)

Applies an aggregation that gives the maximum element of every window of the data stream by the given position. If more elements have the same maximum value the operator returns the first by default.

Parameters:

positionToMaxBy - The position to maximize by

Returns:

The transformed DataStream.

maxBy

public WindowedDataStream<OUT> maxBy(int positionToMaxBy,
                                     boolean first)

Applies an aggregation that gives the maximum element of every window of the data stream by the given position. If more elements have the same maximum value the operator returns either the first or last one depending on the parameter setting.

Parameters:

positionToMaxBy - The position to maximize by

first - If true, then the operator return the first element with the maximum value, otherwise returns the last

Returns:

The transformed DataStream.

maxBy

public WindowedDataStream<OUT> maxBy(String field,
                                     boolean first)

Applies an aggregation that that gives the maximum element of the pojo data stream by the given field expression for every window. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field - The field expression based on which the aggregation will be applied.

first - If True then in case of field equality the first object will be returned

Returns:

The transformed DataStream.

getTrigger

protected TriggerPolicy<OUT> getTrigger()

getEviction

protected EvictionPolicy<OUT> getEviction()

clean

public <F> F clean(F f)

isGrouped

protected boolean isGrouped()

getType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getType()

Gets the output type.

Returns:

The output type.

getExecutionConfig

public org.apache.flink.api.common.ExecutionConfig getExecutionConfig()

getClassAtPos

protected Class<?> getClassAtPos(int pos)

copy

protected WindowedDataStream<OUT> copy()