org.apache.hudi.sink.bootstrap

Class BootstrapOperator<I,O extends HoodieRecord<?>>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<O>

org.apache.hudi.sink.bootstrap.BootstrapOperator<I,O>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<I>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<I,O>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<O>, org.apache.flink.streaming.api.operators.StreamOperator<O>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Direct Known Subclasses:

BatchBootstrapOperator

public class BootstrapOperator<I,O extends HoodieRecord<?>>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<O>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<I,O>

The operator to load index from existing hoodieTable.

Each subtask of the function triggers the index bootstrap when the first element came in, the record cannot be sent until all the index records have been sent.

The output records should then shuffle by the recordKey and thus do scalable write.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.configuration.Configuration	conf
protected org.apache.hadoop.conf.Configuration	hadoopConf
protected HoodieTable<?,?,?,?>	hoodieTable
protected HoodieWriteConfig	writeConfig

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
BootstrapOperator(org.apache.flink.configuration.Configuration conf)

Method Summary

Modifier and Type	Method and Description
static HoodieRecord	generateHoodieRecord(HoodieKey hoodieKey, FileSlice fileSlice)
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
boolean	isAlreadyBootstrap()
protected void	loadRecords(String partitionPath) Loads all the indices of give partition path into the backup state.
protected void	preLoadIndexRecords() Load the index records before processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I>).
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I> element)
protected boolean	shouldLoadFile(String fileId, int maxParallelism, int parallelism, int taskID)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Field Detail

hoodieTable

protected HoodieTable<?,?,?,?> hoodieTable

conf

protected final org.apache.flink.configuration.Configuration conf

hadoopConf

protected transient org.apache.hadoop.conf.Configuration hadoopConf

writeConfig

protected transient HoodieWriteConfig writeConfig

Constructor Detail

BootstrapOperator

public BootstrapOperator(org.apache.flink.configuration.Configuration conf)

Method Detail

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<O extends HoodieRecord<?>>

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<O extends HoodieRecord<?>>

Throws:

Exception

preLoadIndexRecords

protected void preLoadIndexRecords()
                            throws Exception

Load the index records before processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I>).

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<I> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<I>

Throws:

Exception

loadRecords

protected void loadRecords(String partitionPath)
                    throws Exception

Loads all the indices of give partition path into the backup state.

Parameters:

partitionPath - The partition path

Throws:

Exception

generateHoodieRecord

public static HoodieRecord generateHoodieRecord(HoodieKey hoodieKey,
                                                FileSlice fileSlice)

shouldLoadFile

protected boolean shouldLoadFile(String fileId,
                                 int maxParallelism,
                                 int parallelism,
                                 int taskID)

isAlreadyBootstrap

@VisibleForTesting
public boolean isAlreadyBootstrap()
                                              throws Exception

Throws:

Exception