org.apache.flink.table.runtime.operators.bundle

Class AbstractMapBundleOperator<K,V,IN,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator<K,V,IN,OUT>

Type Parameters:

K - The type of the key in the bundle map

V - The type of the value in the bundle map

IN - Input type for the operator.

OUT - Output type for the operator.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, BundleTriggerCallback

Direct Known Subclasses:

KeyedMapBundleOperator, MapBundleOperator

public abstract class AbstractMapBundleOperator<K,V,IN,OUT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, BundleTriggerCallback

The AbstractMapBundleOperator simply used a java Map to store the input elements in key-value form. The map key is typically the same with the state key, so we can do some optimizations before accessing states, like pre aggregate values for each key. And we will only need to access state every key we have, but not every element we processed.

NOTES: if all elements we processed have different keys, such operator will only increase memory footprint, and will not have any performance improvement.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService

Method Summary

Modifier and Type	Method and Description
void	close()
void	finish()
void	finishBundle() This method is invoked to finish current bundle and start a new one when the trigger was fired.
protected abstract K	getKey(IN input) Get the key for current processing element, which will be used as the map bundle's key.
void	open()
void	prepareSnapshotPreBarrier(long checkpointId)
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<IN>

Throws:

Exception

getKey

protected abstract K getKey(IN input)
                     throws Exception

Get the key for current processing element, which will be used as the map bundle's key.

Throws:

Exception

finishBundle

public void finishBundle()
                  throws Exception

Description copied from interface: BundleTriggerCallback

This method is invoked to finish current bundle and start a new one when the trigger was fired.

Specified by:

finishBundle in interface BundleTriggerCallback

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.Input<IN>

Overrides:

processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Specified by:

prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

finish

public void finish()
            throws Exception

Specified by:

finish in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

finish in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception