org.apache.hudi.source

Class StreamReadMonitoringFunction

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<MergeOnReadInputSplit>

org.apache.hudi.source.StreamReadMonitoringFunction

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<MergeOnReadInputSplit>

public class StreamReadMonitoringFunction
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<MergeOnReadInputSplit>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

This is the single (non-parallel) monitoring task which takes a MergeOnReadInputSplit , it is responsible for:

1. Monitoring a user-provided hoodie table path.
2. Deciding which files(or split) should be further read and processed.
3. Creating the splits corresponding to those files.
4. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream StreamReadOperator which can have parallelism greater than one.

IMPORTANT NOTE: Splits are forwarded downstream for reading in ascending instant commits time order, in each downstream task, the splits are also read in receiving sequence. We do not ensure split consuming sequence among the downstream tasks.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
StreamReadMonitoringFunction(org.apache.flink.configuration.Configuration conf, org.apache.flink.core.fs.Path path, org.apache.flink.table.types.logical.RowType rowType, long maxCompactionMemoryInBytes, Set<String> requiredPartitionPaths)

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	close()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	monitorDirAndForwardSplits(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<MergeOnReadInputSplit> context)
void	open(org.apache.flink.configuration.Configuration parameters)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<MergeOnReadInputSplit> context)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamReadMonitoringFunction

public StreamReadMonitoringFunction(org.apache.flink.configuration.Configuration conf,
                                    org.apache.flink.core.fs.Path path,
                                    org.apache.flink.table.types.logical.RowType rowType,
                                    long maxCompactionMemoryInBytes,
                                    @Nullable
                                    Set<String> requiredPartitionPaths)

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<MergeOnReadInputSplit> context)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<MergeOnReadInputSplit>

Throws:

Exception

monitorDirAndForwardSplits

@VisibleForTesting
public void monitorDirAndForwardSplits(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<MergeOnReadInputSplit> context)

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<MergeOnReadInputSplit>

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception