org.apache.flink.streaming.api.runners.python.beam

类 BeamPythonFunctionRunner

java.lang.Object

org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner

所有已实现的接口:

AutoCloseable, PythonFunctionRunner

直接已知子类:

BeamDataStreamPythonFunctionRunner, BeamTablePythonFunctionRunner

@Internal
public abstract class BeamPythonFunctionRunner
extends Object
implements PythonFunctionRunner

A BeamPythonFunctionRunner used to execute Python functions.

字段概要

字段

限定符和类型	字段和说明
protected FlinkFnApi.CoderInfoDescriptor	inputCoderDescriptor
protected static org.slf4j.Logger	LOG
protected org.apache.beam.sdk.fn.data.FnDataReceiver<org.apache.beam.sdk.util.WindowedValue<byte[]>>	mainInputReceiver The receiver which forwards the input elements to a remote environment for processing.
protected FlinkFnApi.CoderInfoDescriptor	outputCoderDescriptor
protected LinkedBlockingQueue<org.apache.flink.api.java.tuple.Tuple2<String,byte[]>>	resultBuffer Buffers the Python function execution result which has still not been processed.
protected Map<String,FlinkFnApi.CoderInfoDescriptor>	sideOutputCoderDescriptors

构造器概要

构造器

构造器和说明

BeamPythonFunctionRunner(String taskName, ProcessPythonEnvironmentManager environmentManager, FlinkMetricContainer flinkMetricContainer, org.apache.flink.runtime.state.KeyedStateBackend<?> keyedStateBackend, org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend, org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer, org.apache.flink.api.common.typeutils.TypeSerializer<?> namespaceSerializer, TimerRegistration timerRegistration, org.apache.flink.runtime.memory.MemoryManager memoryManager, double managedMemoryFraction, FlinkFnApi.CoderInfoDescriptor inputCoderDescriptor, FlinkFnApi.CoderInfoDescriptor outputCoderDescriptor, Map<String,FlinkFnApi.CoderInfoDescriptor> sideOutputCoderDescriptors)

方法概要

限定符和类型	方法和说明
protected abstract void	buildTransforms(org.apache.beam.model.pipeline.v1.RunnerApi.Components.Builder componentsBuilder)
void	close() Tear-down the Python function runner.
org.apache.beam.runners.fnexecution.control.JobBundleFactory	createJobBundleFactory(org.apache.beam.vendor.grpc.v1p48p1.com.google.protobuf.Struct pipelineOptions)
void	flush() Forces to finish the processing of the current bundle of elements.
protected abstract Optional<org.apache.beam.model.pipeline.v1.RunnerApi.Coder>	getOptionalTimerCoderProto()
protected abstract List<org.apache.beam.runners.core.construction.graph.TimerReference>	getTimers(org.apache.beam.model.pipeline.v1.RunnerApi.Components components)
void	notifyNoMoreResults() Interrupts the progress of takeResult.
void	open(org.apache.flink.configuration.ReadableConfig config) Prepares the Python function runner, such as preparing the Python execution environment, etc.
org.apache.flink.api.java.tuple.Tuple3<String,byte[],Integer>	pollResult() Retrieves the Python function result.
void	process(byte[] data) Executes the Python function with the input byte array.
void	processTimer(byte[] timerData) Send the triggered timer to the Python function.
protected void	startBundle()
org.apache.flink.api.java.tuple.Tuple3<String,byte[],Integer>	takeResult() Retrieves the Python function result, waiting if necessary until an element becomes available.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

LOG

protected static final org.slf4j.Logger LOG

inputCoderDescriptor

protected final FlinkFnApi.CoderInfoDescriptor inputCoderDescriptor

outputCoderDescriptor

protected final FlinkFnApi.CoderInfoDescriptor outputCoderDescriptor

sideOutputCoderDescriptors

protected final Map<String,FlinkFnApi.CoderInfoDescriptor> sideOutputCoderDescriptors

resultBuffer

@VisibleForTesting
protected transient LinkedBlockingQueue<org.apache.flink.api.java.tuple.Tuple2<String,byte[]>> resultBuffer

Buffers the Python function execution result which has still not been processed.

mainInputReceiver

@VisibleForTesting
protected transient org.apache.beam.sdk.fn.data.FnDataReceiver<org.apache.beam.sdk.util.WindowedValue<byte[]>> mainInputReceiver

The receiver which forwards the input elements to a remote environment for processing.

构造器详细资料

BeamPythonFunctionRunner

public BeamPythonFunctionRunner(String taskName,
                                ProcessPythonEnvironmentManager environmentManager,
                                @Nullable
                                FlinkMetricContainer flinkMetricContainer,
                                @Nullable
                                org.apache.flink.runtime.state.KeyedStateBackend<?> keyedStateBackend,
                                @Nullable
                                org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend,
                                @Nullable
                                org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer,
                                @Nullable
                                org.apache.flink.api.common.typeutils.TypeSerializer<?> namespaceSerializer,
                                @Nullable
                                TimerRegistration timerRegistration,
                                org.apache.flink.runtime.memory.MemoryManager memoryManager,
                                double managedMemoryFraction,
                                FlinkFnApi.CoderInfoDescriptor inputCoderDescriptor,
                                FlinkFnApi.CoderInfoDescriptor outputCoderDescriptor,
                                Map<String,FlinkFnApi.CoderInfoDescriptor> sideOutputCoderDescriptors)

方法详细资料

open

public void open(org.apache.flink.configuration.ReadableConfig config)
          throws Exception

从接口复制的说明: PythonFunctionRunner

Prepares the Python function runner, such as preparing the Python execution environment, etc.

指定者:

open 在接口中 PythonFunctionRunner

抛出:

Exception

close

public void close()
           throws Exception

从接口复制的说明: PythonFunctionRunner

Tear-down the Python function runner.

指定者:

close 在接口中 AutoCloseable

指定者:

close 在接口中 PythonFunctionRunner

抛出:

Exception

process

public void process(byte[] data)
             throws Exception

从接口复制的说明: PythonFunctionRunner

Executes the Python function with the input byte array.

指定者:

process 在接口中 PythonFunctionRunner

参数:

data - the byte array data.

抛出:

Exception

processTimer

public void processTimer(byte[] timerData)
                  throws Exception

从接口复制的说明: PythonFunctionRunner

Send the triggered timer to the Python function.

指定者:

processTimer 在接口中 PythonFunctionRunner

抛出:

Exception

startBundle

@VisibleForTesting
protected void startBundle()

pollResult

public org.apache.flink.api.java.tuple.Tuple3<String,byte[],Integer> pollResult()
                                                                         throws Exception

从接口复制的说明: PythonFunctionRunner

Retrieves the Python function result.

指定者:

pollResult 在接口中 PythonFunctionRunner

返回:

the head of he Python function result buffer, or null if the result buffer is empty. f0 means the byte array buffer which stores the Python function result. f1 means the length of the Python function result byte array.

抛出:

Exception

takeResult

public org.apache.flink.api.java.tuple.Tuple3<String,byte[],Integer> takeResult()
                                                                         throws Exception

从接口复制的说明: PythonFunctionRunner

Retrieves the Python function result, waiting if necessary until an element becomes available.

指定者:

takeResult 在接口中 PythonFunctionRunner

返回:

the head of he Python function result buffer. f0 means the byte array buffer which stores the Python function result. f1 means the length of the Python function result byte array.

抛出:

Exception

flush

public void flush()
           throws Exception

从接口复制的说明: PythonFunctionRunner

Forces to finish the processing of the current bundle of elements. It will flush the data cached in the data buffer for processing and retrieves the state mutations (if exists) made by the Python function. The call blocks until all of the outputs produced by this bundle have been received.

指定者:

flush 在接口中 PythonFunctionRunner

抛出:

Exception

notifyNoMoreResults

public void notifyNoMoreResults()

Interrupts the progress of takeResult.

buildTransforms

protected abstract void buildTransforms(org.apache.beam.model.pipeline.v1.RunnerApi.Components.Builder componentsBuilder)

getTimers

protected abstract List<org.apache.beam.runners.core.construction.graph.TimerReference> getTimers(org.apache.beam.model.pipeline.v1.RunnerApi.Components components)

getOptionalTimerCoderProto

protected abstract Optional<org.apache.beam.model.pipeline.v1.RunnerApi.Coder> getOptionalTimerCoderProto()

createJobBundleFactory

@VisibleForTesting
public org.apache.beam.runners.fnexecution.control.JobBundleFactory createJobBundleFactory(org.apache.beam.vendor.grpc.v1p48p1.com.google.protobuf.Struct pipelineOptions)
                                                                                                       throws Exception

抛出:

Exception