org.apache.hudi.sink.bootstrap.batch

Class BatchBootstrapOperator<I,O extends HoodieRecord<?>>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<O>

org.apache.hudi.sink.bootstrap.BootstrapOperator<I,O>

org.apache.hudi.sink.bootstrap.batch.BatchBootstrapOperator<I,O>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<I>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<I,O>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<O>, org.apache.flink.streaming.api.operators.StreamOperator<O>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

public class BatchBootstrapOperator<I,O extends HoodieRecord<?>>
extends BootstrapOperator<I,O>

The operator to load index from existing hoodieTable.

This function should only be used for bounded source.

When a record comes in, the function firstly checks whether the partition path of the record is already loaded, if the partition is not loaded yet, loads the entire partition and sends the index records to downstream operators before it sends the input record; if the partition is loaded already, sends the input record directly.

The input records should shuffle by the partition path to avoid repeated loading.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.sink.bootstrap.BootstrapOperator

conf, hadoopConf, hoodieTable, writeConfig

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
BatchBootstrapOperator(org.apache.flink.configuration.Configuration conf)

Method Summary

Modifier and Type	Method and Description
void	open()
protected void	preLoadIndexRecords() Load the index records before BootstrapOperator.processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I>).
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I> element)
protected boolean	shouldLoadFile(String fileId, int maxParallelism, int parallelism, int taskID)

Methods inherited from class org.apache.hudi.sink.bootstrap.BootstrapOperator

generateHoodieRecord, initializeState, isAlreadyBootstrap, loadRecords, snapshotState

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark, processWatermarkStatus

Constructor Detail

BatchBootstrapOperator

public BatchBootstrapOperator(org.apache.flink.configuration.Configuration conf)

Method Detail

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<O extends HoodieRecord<?>>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<O extends HoodieRecord<?>>

Throws:

Exception

preLoadIndexRecords

protected void preLoadIndexRecords()

Description copied from class: BootstrapOperator

Load the index records before BootstrapOperator.processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I>).

Overrides:

preLoadIndexRecords in class BootstrapOperator<I,O extends HoodieRecord<?>>

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<I>

Overrides:

processElement in class BootstrapOperator<I,O extends HoodieRecord<?>>

Throws:

Exception

shouldLoadFile

protected boolean shouldLoadFile(String fileId,
                                 int maxParallelism,
                                 int parallelism,
                                 int taskID)

Overrides:

shouldLoadFile in class BootstrapOperator<I,O extends HoodieRecord<?>>