org.apache.flink.streaming.api.operators.python

Class TwoInputPythonFunctionOperator<IN1,IN2,RUNNER_OUT,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractTwoInputPythonFunctionOperator<IN1,IN2,OUT>

org.apache.flink.streaming.api.operators.python.TwoInputPythonFunctionOperator<IN1,IN2,RUNNER_OUT,OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedMultiInput, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<IN1,IN2,OUT>, org.apache.flink.util.Disposable

Direct Known Subclasses:

PythonCoFlatMapOperator, PythonCoMapOperator, PythonKeyedCoProcessOperator

@Internal
public abstract class TwoInputPythonFunctionOperator<IN1,IN2,RUNNER_OUT,OUT>
extends AbstractTwoInputPythonFunctionOperator<IN1,IN2,OUT>

TwoInputPythonFunctionOperator is responsible for launching beam runner which will start a python harness to execute two-input user defined python function.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.core.memory.ByteArrayInputStreamWithPos	bais
protected org.apache.flink.core.memory.DataInputViewStreamWrapper	baisWrapper
protected org.apache.flink.core.memory.ByteArrayOutputStreamWithPos	baos
protected org.apache.flink.core.memory.DataOutputViewStreamWrapper	baosWrapper
protected org.apache.flink.streaming.api.operators.TimestampedCollector	collector
protected org.apache.flink.types.Row	reuseRow

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

elementCount, maxBundleSize, pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
TwoInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, String coderUrn, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> runnerInputTypeInfo, org.apache.flink.api.common.typeinfo.TypeInformation<RUNNER_OUT> runnerOutputTypeInfo)
TwoInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.api.common.typeinfo.TypeInformation<IN1> inputTypeInfo1, org.apache.flink.api.common.typeinfo.TypeInformation<IN2> inputTypeInfo2, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo, String coderUrn)

Method Summary

Modifier and Type	Method and Description
PythonFunctionRunner	createPythonFunctionRunner() Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
protected String	getCoderUrn()
protected Map<String,String>	getJobOptions()
org.apache.flink.table.functions.python.PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
protected DataStreamPythonFunctionInfo	getPythonFunctionInfo()
protected org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row>	getRunnerInputTypeInfo()
protected org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row>	getRunnerInputTypeSerializer()
protected org.apache.flink.api.common.typeinfo.TypeInformation<RUNNER_OUT>	getRunnerOutputTypeInfo()
protected org.apache.flink.api.common.typeutils.TypeSerializer<RUNNER_OUT>	getRunnerOutputTypeSerializer()
void	open()
void	processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN1> element)
void	processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN2> element)

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractTwoInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResult, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Field Detail

bais

protected transient org.apache.flink.core.memory.ByteArrayInputStreamWithPos bais

baisWrapper

protected transient org.apache.flink.core.memory.DataInputViewStreamWrapper baisWrapper

baos

protected transient org.apache.flink.core.memory.ByteArrayOutputStreamWithPos baos

baosWrapper

protected transient org.apache.flink.core.memory.DataOutputViewStreamWrapper baosWrapper

collector

protected transient org.apache.flink.streaming.api.operators.TimestampedCollector collector

reuseRow

protected transient org.apache.flink.types.Row reuseRow

Constructor Detail

TwoInputPythonFunctionOperator

public TwoInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config,
                                      DataStreamPythonFunctionInfo pythonFunctionInfo,
                                      String coderUrn,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> runnerInputTypeInfo,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<RUNNER_OUT> runnerOutputTypeInfo)

TwoInputPythonFunctionOperator

public TwoInputPythonFunctionOperator(org.apache.flink.configuration.Configuration config,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<IN1> inputTypeInfo1,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<IN2> inputTypeInfo2,
                                      org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outputTypeInfo,
                                      DataStreamPythonFunctionInfo pythonFunctionInfo,
                                      String coderUrn)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

open in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

createPythonFunctionRunner

public PythonFunctionRunner createPythonFunctionRunner()
                                                throws Exception

Description copied from class: AbstractPythonFunctionOperator

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Specified by:

createPythonFunctionRunner in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

getPythonEnv

public org.apache.flink.table.functions.python.PythonEnv getPythonEnv()

Description copied from class: AbstractPythonFunctionOperator

Returns the PythonEnv used to create PythonEnvironmentManager..

Specified by:

getPythonEnv in class AbstractPythonFunctionOperator<OUT>

processElement1

public void processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN1> element)
                     throws Exception

Throws:

Exception

processElement2

public void processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN2> element)
                     throws Exception

Throws:

Exception

getJobOptions

protected Map<String,String> getJobOptions()

getCoderUrn

protected String getCoderUrn()

getRunnerInputTypeInfo

protected org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> getRunnerInputTypeInfo()

getRunnerOutputTypeInfo

protected org.apache.flink.api.common.typeinfo.TypeInformation<RUNNER_OUT> getRunnerOutputTypeInfo()

getPythonFunctionInfo

protected DataStreamPythonFunctionInfo getPythonFunctionInfo()

getRunnerInputTypeSerializer

protected org.apache.flink.api.common.typeutils.TypeSerializer<org.apache.flink.types.Row> getRunnerInputTypeSerializer()

getRunnerOutputTypeSerializer

protected org.apache.flink.api.common.typeutils.TypeSerializer<RUNNER_OUT> getRunnerOutputTypeSerializer()