org.apache.flink.streaming.api.operators

Class SourceOperator<OUT,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.SourceOperator<OUT,SplitT>

Type Parameters:

OUT - The output type of the operator.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.io.AvailabilityProvider, org.apache.flink.runtime.operators.coordination.OperatorEventHandler, KeyContext, SetupableStreamOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, PushingAsyncDataInput<OUT>, org.apache.flink.util.Disposable

@Internal
public class SourceOperator<OUT,SplitT extends org.apache.flink.api.connector.source.SourceSplit>
extends AbstractStreamOperator<OUT>
implements org.apache.flink.runtime.operators.coordination.OperatorEventHandler, PushingAsyncDataInput<OUT>

Base source operator only used for integrating the source reader which is proposed by FLIP-27. It implements the interface of PushingAsyncDataInput for naturally compatible with one input processing in runtime stack.

Important Note on Serialization: The SourceOperator inherits the Serializable interface from the StreamOperator, but is in fact NOT serializable. The operator must only be instantiates in the StreamTask from its factory.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

PushingAsyncDataInput.DataOutput<T>

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

org.apache.flink.runtime.io.AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description

SourceOperator(org.apache.flink.util.function.FunctionWithException<org.apache.flink.api.connector.source.SourceReaderContext,org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>,Exception> readerFactory, org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway, org.apache.flink.core.io.SimpleVersionedSerializer<SplitT> splitSerializer, org.apache.flink.api.common.eventtime.WatermarkStrategy<OUT> watermarkStrategy, ProcessingTimeService timeService, org.apache.flink.configuration.Configuration configuration, String localHostname, boolean emitProgressiveWatermarks)

Method Summary

Modifier and Type	Method and Description
void	close() This method is called after all records have been added to the operators via the methods Input.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).
void	dispose() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
org.apache.flink.core.io.InputStatus	emitNext(PushingAsyncDataInput.DataOutput<OUT> output) Pushes the next element to the output from current data input, and returns the input status to indicate whether there are more available data in current input.
CompletableFuture<?>	getAvailableFuture()
org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>	getSourceReader()
void	handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context) Stream operators with state which can be restored need to override this hook method.
void	initReader() Initializes the reader.
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g.
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) Stream operators with state, which want to participate in a snapshot need to override this hook method.

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Constructor Detail

SourceOperator

public SourceOperator(org.apache.flink.util.function.FunctionWithException<org.apache.flink.api.connector.source.SourceReaderContext,org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>,Exception> readerFactory,
                      org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway,
                      org.apache.flink.core.io.SimpleVersionedSerializer<SplitT> splitSerializer,
                      org.apache.flink.api.common.eventtime.WatermarkStrategy<OUT> watermarkStrategy,
                      ProcessingTimeService timeService,
                      org.apache.flink.configuration.Configuration configuration,
                      String localHostname,
                      boolean emitProgressiveWatermarks)

Method Detail

initReader

public void initReader()
                throws Exception

Initializes the reader. The code from this method should ideally happen in the constructor or in the operator factory even. It has to happen here at a slightly later stage, because of the lazy metric initialization.

Calling this method explicitly is an optional way to have the reader initialization a bit earlier than in open(), as needed by the SourceOperatorStreamTask

This code should move to the constructor once the metric groups are available at task setup time.

Throws:

Exception

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from class: AbstractStreamOperator

This method is called after all records have been added to the operators via the methods Input.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized asa failed, because the last data items are not processed properly.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

dispose

public void dispose()
             throws Exception

Description copied from class: AbstractStreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

Specified by:

dispose in interface StreamOperator<OUT>

Specified by:

dispose in interface org.apache.flink.util.Disposable

Overrides:

dispose in class AbstractStreamOperator<OUT>

Throws:

Exception

emitNext

public org.apache.flink.core.io.InputStatus emitNext(PushingAsyncDataInput.DataOutput<OUT> output)
                                              throws Exception

Description copied from interface: PushingAsyncDataInput

Pushes the next element to the output from current data input, and returns the input status to indicate whether there are more available data in current input.

This method should be non blocking.

Specified by:

emitNext in interface PushingAsyncDataInput<OUT>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Description copied from class: AbstractStreamOperator

Stream operators with state, which want to participate in a snapshot need to override this hook method.

Specified by:

snapshotState in interface StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class AbstractStreamOperator<OUT>

Parameters:

context - context that provides information and means required for taking a snapshot

Throws:

Exception

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface org.apache.flink.runtime.io.AvailabilityProvider

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Description copied from class: AbstractStreamOperator

Stream operators with state which can be restored need to override this hook method.

Specified by:

initializeState in interface StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class AbstractStreamOperator<OUT>

Parameters:

context - context that allows to register different states.

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Overrides:

notifyCheckpointComplete in class AbstractStreamOperator<OUT>

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)
                             throws Exception

Specified by:

notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

Overrides:

notifyCheckpointAborted in class AbstractStreamOperator<OUT>

Throws:

Exception

handleOperatorEvent

public void handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)

Specified by:

handleOperatorEvent in interface org.apache.flink.runtime.operators.coordination.OperatorEventHandler

getSourceReader

@VisibleForTesting
public org.apache.flink.api.connector.source.SourceReader<OUT,SplitT> getSourceReader()