org.apache.flink.streaming.api.operators.python

Class PythonTimestampsAndWatermarksOperator<IN>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator<IN,OUT>

org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator<IN,IN,org.apache.flink.types.Row,Long>

org.apache.flink.streaming.api.operators.python.PythonTimestampsAndWatermarksOperator<IN>

Type Parameters:

IN - The type of the input elements

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,IN>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<IN>, org.apache.flink.streaming.api.operators.StreamOperator<IN>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback, org.apache.flink.util.Disposable

@Internal
public class PythonTimestampsAndWatermarksOperator<IN>
extends OneInputPythonFunctionOperator<IN,IN,org.apache.flink.types.Row,Long>
implements org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

A stream operator that may do one or both of the following: extract timestamps from events and generate watermarks by user specify TimestampAssigner and WatermarkStrategy.

These two responsibilities run in the same operator rather than in two different ones, because the implementation of the timestamp assigner and the watermark generator is frequently in the same class (and should be run in the same instance), even though the separate interfaces support the use of different classes.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	STREAM_TIMESTAMP_AND_WATERMARK_OPERATOR_NAME

Fields inherited from class org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator

bais, baisWrapper, baos, baosWrapper, collector, DATA_STREAM_STATELESS_FUNCTION_URN, jobOptions

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

elementCount, maxBundleSize, pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
PythonTimestampsAndWatermarksOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.api.common.typeinfo.TypeInformation<IN> inputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo, org.apache.flink.api.common.eventtime.WatermarkStrategy<IN> watermarkStrategy)

Method Summary

Modifier and Type	Method and Description
void	close()
void	configureEmitProgressiveWatermarks(boolean emitProgressiveWatermarks)
void	emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple) Sends the execution result to the downstream operator.
String	getCoderUrn()
void	onProcessingTime(long timestamp)
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)

Methods inherited from class org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator

createPythonFunctionRunner, getFunctionUrn, getInternalParameters, getPythonEnv

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, createPythonEnvironmentManager, dispose, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, setCurrentKey, setPythonConfig

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker

Field Detail

STREAM_TIMESTAMP_AND_WATERMARK_OPERATOR_NAME

public static final String STREAM_TIMESTAMP_AND_WATERMARK_OPERATOR_NAME

See Also:

Constant Field Values

Constructor Detail

PythonTimestampsAndWatermarksOperator

public PythonTimestampsAndWatermarksOperator(org.apache.flink.configuration.Configuration config,
                                             org.apache.flink.api.common.typeinfo.TypeInformation<IN> inputTypeInfo,
                                             DataStreamPythonFunctionInfo pythonFunctionInfo,
                                             org.apache.flink.api.common.eventtime.WatermarkStrategy<IN> watermarkStrategy)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>

Overrides:

open in class OneInputPythonFunctionOperator<IN,IN,org.apache.flink.types.Row,Long>

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<IN>

Overrides:

processElement in class OneInputPythonFunctionOperator<IN,IN,org.apache.flink.types.Row,Long>

Throws:

Exception

emitResult

public void emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple)
                throws Exception

Description copied from class: AbstractPythonFunctionOperator

Sends the execution result to the downstream operator.

Specified by:

emitResult in class AbstractPythonFunctionOperator<IN>

Throws:

Exception

getCoderUrn

public String getCoderUrn()

Specified by:

getCoderUrn in class OneInputPythonFunctionOperator<IN,IN,org.apache.flink.types.Row,Long>

configureEmitProgressiveWatermarks

public void configureEmitProgressiveWatermarks(boolean emitProgressiveWatermarks)

onProcessingTime

public void onProcessingTime(long timestamp)

Specified by:

onProcessingTime in interface org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.Input<IN>

Overrides:

processWatermark in class AbstractPythonFunctionOperator<IN>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>

Overrides:

close in class AbstractPythonFunctionOperator<IN>

Throws:

Exception