org.apache.flink.table.runtime.operators.python.aggregate.arrow

Class AbstractArrowPythonAggregateFunctionOperator

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator<IN,OUT>

org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

org.apache.flink.table.runtime.operators.python.aggregate.arrow.AbstractArrowPythonAggregateFunctionOperator

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.util.Disposable

Direct Known Subclasses:

AbstractStreamArrowPythonOverWindowAggregateFunctionOperator, BatchArrowPythonGroupAggregateFunctionOperator, BatchArrowPythonGroupWindowAggregateFunctionOperator, BatchArrowPythonOverWindowAggregateFunctionOperator, StreamArrowPythonGroupWindowAggregateFunctionOperator

@Internal
public abstract class AbstractArrowPythonAggregateFunctionOperator
extends AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

The Abstract class of Arrow Aggregate Operator for Pandas AggregateFunction.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected ArrowSerializer<org.apache.flink.table.data.RowData>	arrowSerializer
protected int	currentBatchCount The current number of elements to be included in an arrow batch.
protected int[]	groupingSet
protected org.apache.flink.table.functions.python.PythonFunctionInfo[]	pandasAggFunctions The Pandas AggregateFunctions to be executed.
protected org.apache.flink.table.data.utils.JoinedRowData	reuseJoinedRow The JoinedRowData reused holding the execution result.
protected StreamRecordRowDataWrappingCollector	rowDataWrapper The collector used to collect records.

Fields inherited from class org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator

bais, baisWrapper, baos, baosWrapper, forwardedInputQueue, inputType, outputType, userDefinedFunctionInputOffsets, userDefinedFunctionInputType, userDefinedFunctionOutputType

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

elementCount, maxBundleSize, pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
AbstractArrowPythonAggregateFunctionOperator(org.apache.flink.configuration.Configuration config, org.apache.flink.table.functions.python.PythonFunctionInfo[] pandasAggFunctions, org.apache.flink.table.types.logical.RowType inputType, org.apache.flink.table.types.logical.RowType outputType, int[] groupingSet, int[] udafInputOffsets)

Method Summary

Modifier and Type	Method and Description
void	dispose()
org.apache.flink.table.data.RowData	getFunctionInput(org.apache.flink.table.data.RowData element)
String	getFunctionUrn()
String	getInputOutputCoderUrn()
org.apache.flink.table.functions.python.PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
FlinkFnApi.UserDefinedFunctions	getUserDefinedFunctionsProto() Gets the proto representation of the Python user-defined functions to be executed.
boolean	isBundleFinished() Returns whether the bundle is finished.
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)

Methods inherited from class org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator

bufferInput, createPythonFunctionRunner, processElementInternal

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractOneInputPythonFunctionOperator

endInput

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, emitResult, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark

Field Detail

pandasAggFunctions

protected final org.apache.flink.table.functions.python.PythonFunctionInfo[] pandasAggFunctions

The Pandas AggregateFunctions to be executed.

groupingSet

protected final int[] groupingSet

arrowSerializer

protected transient ArrowSerializer<org.apache.flink.table.data.RowData> arrowSerializer

rowDataWrapper

protected transient StreamRecordRowDataWrappingCollector rowDataWrapper

The collector used to collect records.

reuseJoinedRow

protected transient org.apache.flink.table.data.utils.JoinedRowData reuseJoinedRow

The JoinedRowData reused holding the execution result.

currentBatchCount

protected transient int currentBatchCount

The current number of elements to be included in an arrow batch.

Constructor Detail

AbstractArrowPythonAggregateFunctionOperator

public AbstractArrowPythonAggregateFunctionOperator(org.apache.flink.configuration.Configuration config,
                                                    org.apache.flink.table.functions.python.PythonFunctionInfo[] pandasAggFunctions,
                                                    org.apache.flink.table.types.logical.RowType inputType,
                                                    org.apache.flink.table.types.logical.RowType outputType,
                                                    int[] groupingSet,
                                                    int[] udafInputOffsets)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Overrides:

open in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

Throws:

Exception

dispose

public void dispose()
             throws Exception

Specified by:

dispose in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.table.data.RowData>

Specified by:

dispose in interface org.apache.flink.util.Disposable

Overrides:

dispose in class AbstractPythonFunctionOperator<org.apache.flink.table.data.RowData>

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.table.data.RowData> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<org.apache.flink.table.data.RowData>

Overrides:

processElement in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

Throws:

Exception

isBundleFinished

public boolean isBundleFinished()

Description copied from class: AbstractPythonFunctionOperator

Returns whether the bundle is finished.

Overrides:

isBundleFinished in class AbstractPythonFunctionOperator<org.apache.flink.table.data.RowData>

getPythonEnv

public org.apache.flink.table.functions.python.PythonEnv getPythonEnv()

Description copied from class: AbstractPythonFunctionOperator

Returns the PythonEnv used to create PythonEnvironmentManager..

Specified by:

getPythonEnv in class AbstractPythonFunctionOperator<org.apache.flink.table.data.RowData>

getFunctionUrn

public String getFunctionUrn()

Specified by:

getFunctionUrn in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

getInputOutputCoderUrn

public String getInputOutputCoderUrn()

Specified by:

getInputOutputCoderUrn in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

getFunctionInput

public org.apache.flink.table.data.RowData getFunctionInput(org.apache.flink.table.data.RowData element)

Specified by:

getFunctionInput in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>

getUserDefinedFunctionsProto

public FlinkFnApi.UserDefinedFunctions getUserDefinedFunctionsProto()

Description copied from class: AbstractStatelessFunctionOperator

Gets the proto representation of the Python user-defined functions to be executed.

Specified by:

getUserDefinedFunctionsProto in class AbstractStatelessFunctionOperator<org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData,org.apache.flink.table.data.RowData>