org.apache.flink.table.runtime.operators.window.tvf.common

Class WindowAggOperator<K,W>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.table.runtime.operators.TableStreamOperator<org.apache.flink.table.data.RowData>

org.apache.flink.table.runtime.operators.window.tvf.common.WindowAggOperator<K,W>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<K,W>

@Internal
public final class WindowAggOperator<K,W>
extends TableStreamOperator<org.apache.flink.table.data.RowData>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.Triggerable<K,W>, org.apache.flink.streaming.api.operators.KeyContext

We divide windows into 2 categories: Aligned Windows and Unaligned Windows.

Concept of Aligned Window and Unaligned Window

Aligned Windows are windows have predetermined window boundaries and windows can be divided into finite number of non-overlapping chunks. The boundary of an aligned window is determined independently from the time characteristic of the data stream, or messages it receives. For example, hopping (sliding) window is an aligned window as the window boundaries are predetermined based on the window size and slide. Aligned windows include tumbling, hopping, cumulative windows.

Unaligned Windows are windows determined dynamically based on elements. For example, session window is an unaligned window as the window boundaries are determined based on the messages timestamps and their correlations. Currently, unaligned windows include session window only.

Because aligned windows can be divided into finite number of non-overlapping chunks (a.k.a. slices), which can apply efficient processing to share intermediate results.

 Window
 |
 +-- Aligned Window (Slicing Window)
 |    |
 |    +-- Tumble (Slice Unshared Window)
 |    |
 |    +-- Hop (Slice Shared Window)
 |    |
 |    +-- Cumulate (Slice Shared Window)
 |
 +-- Unaligned Window (Unslice Window)
      |
      +-- Session

 

Note: currently, WindowAggOperator doesn't support early-fire and late-arrival. Thus late elements (elements belong to emitted windows) will be simply dropped.

See more in SlicingWindowProcessor and UnslicingWindowProcessor.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.table.runtime.operators.TableStreamOperator

TableStreamOperator.ContextImpl

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.streaming.api.operators.TimestampedCollector<org.apache.flink.table.data.RowData>	collector This is used for emitting elements with a given timestamp.
protected org.apache.flink.streaming.api.operators.InternalTimerService<W>	internalTimerService The service to register timers.
protected long	lastTriggeredProcessingTime The tracked processing time triggered last time.
protected org.apache.flink.metrics.Meter	lateRecordsDroppedRate
protected org.apache.flink.metrics.Counter	numLateRecordsDropped
protected org.apache.flink.metrics.Gauge<Long>	watermarkLatency
protected org.apache.flink.api.common.state.ListState<Long>	watermarkState The operator state to store watermark.
protected WindowProcessor<W>	windowProcessor The concrete window operator implementation.

Fields inherited from class org.apache.flink.table.runtime.operators.TableStreamOperator

ctx, currentWatermark

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
WindowAggOperator(WindowProcessor<W> windowProcessor)

Method Summary

Modifier and Type	Method and Description
void	close()
org.apache.flink.metrics.Counter	getNumLateRecordsDropped()
org.apache.flink.metrics.Gauge<Long>	getWatermarkLatency()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<K,W> timer)
void	onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<K,W> timer)
void	open()
void	prepareSnapshotPreBarrier(long checkpointId)
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.flink.table.runtime.operators.TableStreamOperator

computeMemorySize

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

finish, getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Field Detail

windowProcessor

protected final WindowProcessor<W> windowProcessor

The concrete window operator implementation.

collector

protected transient org.apache.flink.streaming.api.operators.TimestampedCollector<org.apache.flink.table.data.RowData> collector

This is used for emitting elements with a given timestamp.

internalTimerService

protected transient org.apache.flink.streaming.api.operators.InternalTimerService<W> internalTimerService

The service to register timers.

lastTriggeredProcessingTime

protected transient long lastTriggeredProcessingTime

The tracked processing time triggered last time.

watermarkState

protected transient org.apache.flink.api.common.state.ListState<Long> watermarkState

The operator state to store watermark.

numLateRecordsDropped

protected transient org.apache.flink.metrics.Counter numLateRecordsDropped

lateRecordsDroppedRate

protected transient org.apache.flink.metrics.Meter lateRecordsDroppedRate

watermarkLatency

protected transient org.apache.flink.metrics.Gauge<Long> watermarkLatency

Constructor Detail

WindowAggOperator

public WindowAggOperator(WindowProcessor<W> windowProcessor)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Overrides:

open in class TableStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Overrides:

close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>

Throws:

Exception

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>

Overrides:

processWatermark in class TableStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

onEventTime

public void onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<K,W> timer)
                 throws Exception

Specified by:

onEventTime in interface org.apache.flink.streaming.api.operators.Triggerable<K,W>

Throws:

Exception

onProcessingTime

public void onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<K,W> timer)
                      throws Exception

Specified by:

onProcessingTime in interface org.apache.flink.streaming.api.operators.Triggerable<K,W>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Specified by:

prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Overrides:

prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

getNumLateRecordsDropped

@VisibleForTesting
public org.apache.flink.metrics.Counter getNumLateRecordsDropped()

getWatermarkLatency

@VisibleForTesting
public org.apache.flink.metrics.Gauge<Long> getWatermarkLatency()