org.apache.flink.streaming.api.operators.python

Class OneInputPythonFunctionOperator<IN,OUT,UDFIN,UDFOUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator<IN,OUT>

org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator<IN,OUT,UDFIN,UDFOUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.util.Disposable

Direct Known Subclasses:

PythonFlatMapOperator, PythonMapOperator, PythonPartitionCustomOperator, PythonProcessOperator, PythonTimestampsAndWatermarksOperator

@Internal
public abstract class OneInputPythonFunctionOperator<IN,OUT,UDFIN,UDFOUT>
extends AbstractOneInputPythonFunctionOperator<IN,OUT>

OneInputPythonFunctionOperator is responsible for launching beam runner which will start a python harness to execute user defined python function.

The operator will buffer the timestamp of input elements in a queue, and set into the produced output element.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.core.memory.ByteArrayInputStreamWithPos	bais Reusable InputStream used to holding the execution results to be deserialized.
protected org.apache.flink.core.memory.DataInputViewStreamWrapper	baisWrapper InputStream Wrapper.
protected org.apache.flink.core.memory.ByteArrayOutputStreamWithPos	baos Reusable OutputStream used to holding the serialized input elements.
protected org.apache.flink.core.memory.DataOutputViewStreamWrapper	baosWrapper OutputStream Wrapper.
protected org.apache.flink.streaming.api.operators.TimestampedCollector<OUT>	collector
protected static String	DATA_STREAM_STATELESS_FUNCTION_URN
protected Map<String,String>	jobOptions The options used to configure the Python worker process.

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

elementCount, maxBundleSize, pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
OneInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.api.common.typeinfo.TypeInformation<UDFIN> runnerInputTypeInfo, org.apache.flink.api.common.typeinfo.TypeInformation<UDFOUT> runnerOutputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo)

Method Summary

Modifier and Type	Method and Description
PythonFunctionRunner	createPythonFunctionRunner() Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
abstract String	getCoderUrn()
String	getFunctionUrn()
Map<String,String>	getInternalParameters()
org.apache.flink.table.functions.python.PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResult, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark

Field Detail

DATA_STREAM_STATELESS_FUNCTION_URN

protected static final String DATA_STREAM_STATELESS_FUNCTION_URN

See Also:

Constant Field Values

jobOptions

protected final Map<String,String> jobOptions

The options used to configure the Python worker process.

bais

protected transient org.apache.flink.core.memory.ByteArrayInputStreamWithPos bais

Reusable InputStream used to holding the execution results to be deserialized.

baisWrapper

protected transient org.apache.flink.core.memory.DataInputViewStreamWrapper baisWrapper

InputStream Wrapper.

baos

protected transient org.apache.flink.core.memory.ByteArrayOutputStreamWithPos baos

Reusable OutputStream used to holding the serialized input elements.

baosWrapper

protected transient org.apache.flink.core.memory.DataOutputViewStreamWrapper baosWrapper

OutputStream Wrapper.

collector

protected transient org.apache.flink.streaming.api.operators.TimestampedCollector<OUT> collector

Constructor Detail

OneInputPythonFunctionOperator

public OneInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<UDFIN> runnerInputTypeInfo,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<UDFOUT> runnerOutputTypeInfo,
                                      DataStreamPythonFunctionInfo pythonFunctionInfo)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

open in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

createPythonFunctionRunner

public PythonFunctionRunner createPythonFunctionRunner()
                                                throws Exception

Description copied from class: AbstractPythonFunctionOperator

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Specified by:

createPythonFunctionRunner in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

getPythonEnv

public org.apache.flink.table.functions.python.PythonEnv getPythonEnv()

Description copied from class: AbstractPythonFunctionOperator

Returns the PythonEnv used to create PythonEnvironmentManager..

Specified by:

getPythonEnv in class AbstractPythonFunctionOperator<OUT>

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception

Throws:

Exception

getFunctionUrn

public String getFunctionUrn()

getInternalParameters

public Map<String,String> getInternalParameters()

getCoderUrn

public abstract String getCoderUrn()