org.apache.flink.streaming.api.operators.python

Class PythonPartitionCustomOperator<IN,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator<IN,OUT>

org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator<IN,OUT,IN,OUT>

org.apache.flink.streaming.api.operators.python.PythonPartitionCustomOperator<IN,OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.util.Disposable

@Internal
public class PythonPartitionCustomOperator<IN,OUT>
extends OneInputPythonFunctionOperator<IN,OUT,IN,OUT>

The PythonPartitionCustomOperator enables us to set the number of partitions for current operator dynamically when generating the StreamGraph before executing the job. The number of partitions will be set in environment variables for python Worker, so that we can obtain the number of partitions when executing user defined partitioner function.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator

bais, baisWrapper, baos, baosWrapper, collector, DATA_STREAM_STATELESS_FUNCTION_URN, jobOptions

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

elementCount, maxBundleSize, pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
PythonPartitionCustomOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.api.common.typeinfo.TypeInformation<IN> inputTypeInfo, org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo)

Method Summary

Modifier and Type	Method and Description
void	emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple) Sends the execution result to the downstream operator.
String	getCoderUrn()
Map<String,String>	getInternalParameters()
void	setNumPartitions(int numPartitions)

Methods inherited from class org.apache.flink.streaming.api.operators.python.OneInputPythonFunctionOperator

createPythonFunctionRunner, getFunctionUrn, getPythonEnv, open, processElement

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark

Constructor Detail

PythonPartitionCustomOperator

public PythonPartitionCustomOperator(org.apache.flink.configuration.Configuration config,
                                     org.apache.flink.api.common.typeinfo.TypeInformation<IN> inputTypeInfo,
                                     org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outputTypeInfo,
                                     DataStreamPythonFunctionInfo pythonFunctionInfo)

Method Detail

emitResult

public void emitResult(org.apache.flink.api.java.tuple.Tuple2<byte[],Integer> resultTuple)
                throws Exception

Description copied from class: AbstractPythonFunctionOperator

Sends the execution result to the downstream operator.

Specified by:

emitResult in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

getInternalParameters

public Map<String,String> getInternalParameters()

Overrides:

getInternalParameters in class OneInputPythonFunctionOperator<IN,OUT,IN,OUT>

getCoderUrn

public String getCoderUrn()

Specified by:

getCoderUrn in class OneInputPythonFunctionOperator<IN,OUT,IN,OUT>

setNumPartitions

public void setNumPartitions(int numPartitions)