org.apache.flink.streaming.api.operators.python

Class AbstractPythonFunctionOperator<OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.util.Disposable

Direct Known Subclasses:

AbstractOneInputPythonFunctionOperator, AbstractTwoInputPythonFunctionOperator

@Internal
public abstract class AbstractPythonFunctionOperator<OUT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Base class for all stream operators to execute Python functions.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	elementCount Number of processed elements in the current bundle.
protected int	maxBundleSize Max number of elements to include in a bundle.
protected PythonFunctionRunner	pythonFunctionRunner The PythonFunctionRunner which is responsible for Python user-defined function execution.

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
AbstractPythonFunctionOperator(org.apache.flink.configuration.Configuration config)

Method Summary

Modifier and Type	Method and Description
protected void	checkInvokeFinishBundleByCount() Checks whether to invoke finishBundle by elements count.
void	close()
protected PythonEnvironmentManager	createPythonEnvironmentManager()
abstract PythonFunctionRunner	createPythonFunctionRunner() Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
void	dispose()
abstract void	emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple) Sends the execution result to the downstream operator.
protected void	emitResults()
PythonConfig	getConfig() Returns the PythonConfig.
protected FlinkMetricContainer	getFlinkMetricContainer()
PythonConfig	getPythonConfig()
abstract org.apache.flink.table.functions.python.PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
protected void	invokeFinishBundle()
boolean	isBundleFinished() Returns whether the bundle is finished.
void	open()
void	prepareSnapshotPreBarrier(long checkpointId)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
void	setCurrentKey(Object key)
void	setPythonConfig(PythonConfig pythonConfig) Reset the PythonConfig if needed.

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pythonFunctionRunner

protected transient PythonFunctionRunner pythonFunctionRunner

The PythonFunctionRunner which is responsible for Python user-defined function execution.

maxBundleSize

protected transient int maxBundleSize

Max number of elements to include in a bundle.

elementCount

protected transient int elementCount

Number of processed elements in the current bundle.

Constructor Detail

AbstractPythonFunctionOperator

public AbstractPythonFunctionOperator(org.apache.flink.configuration.Configuration config)

Method Detail

getPythonConfig

public PythonConfig getPythonConfig()

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

dispose

public void dispose()
             throws Exception

Specified by:

dispose in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Specified by:

dispose in interface org.apache.flink.util.Disposable

Overrides:

dispose in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Specified by:

prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception

Overrides:

processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

setCurrentKey

public void setCurrentKey(Object key)

Specified by:

setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

setCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

isBundleFinished

public boolean isBundleFinished()

Returns whether the bundle is finished.

setPythonConfig

public void setPythonConfig(PythonConfig pythonConfig)

Reset the PythonConfig if needed.

getConfig

public PythonConfig getConfig()

Returns the PythonConfig.

createPythonFunctionRunner

public abstract PythonFunctionRunner createPythonFunctionRunner()
                                                         throws Exception

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Throws:

Exception

getPythonEnv

public abstract org.apache.flink.table.functions.python.PythonEnv getPythonEnv()

Returns the PythonEnv used to create PythonEnvironmentManager..

emitResult

public abstract void emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple)
                         throws Exception

Sends the execution result to the downstream operator.

Throws:

Exception

emitResults

protected void emitResults()
                    throws Exception

Throws:

Exception

checkInvokeFinishBundleByCount

protected void checkInvokeFinishBundleByCount()
                                       throws Exception

Checks whether to invoke finishBundle by elements count. Called in processElement.

Throws:

Exception

invokeFinishBundle

protected void invokeFinishBundle()
                           throws Exception

Throws:

Exception

createPythonEnvironmentManager

protected PythonEnvironmentManager createPythonEnvironmentManager()
                                                           throws IOException

Throws:

IOException

getFlinkMetricContainer

protected FlinkMetricContainer getFlinkMetricContainer()