org.apache.hudi.common.table.log

Class BaseHoodieLogRecordReader<T>

java.lang.Object

org.apache.hudi.common.table.log.BaseHoodieLogRecordReader<T>

Type Parameters:

T - type of engine-specific record representation.

Direct Known Subclasses:

HoodieMergedLogRecordReader

public abstract class BaseHoodieLogRecordReader<T>
extends Object

A new abstract log record reader, returning records in the type of engine-specific record representation.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BaseHoodieLogRecordReader.Builder<T> Builder used to build AbstractHoodieLogRecordScanner.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	forceFullScan
protected HoodieTableMetaClient	hoodieTableMetaClient
protected List<String>	logFilePaths
protected String	preCombineField
protected HoodieReaderContext<T>	readerContext
protected org.apache.avro.Schema	readerSchema
protected HoodieFileGroupRecordBuffer<T>	recordBuffer
protected RecordMergeMode	recordMergeMode
protected HoodieRecordMerger	recordMerger
protected HoodieRecord.HoodieRecordType	recordType

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

BaseHoodieLogRecordReader(HoodieReaderContext readerContext, HoodieStorage storage, List<String> logFilePaths, boolean reverseReader, int bufferSize, Option<InstantRange> instantRange, boolean withOperationField, boolean forceFullScan, Option<String> partitionNameOverride, Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger, RecordMergeMode recordMergeMode, HoodieFileGroupRecordBuffer<T> recordBuffer)

Method Summary

Modifier and Type	Method and Description
Deque<HoodieLogBlock>	getCurrentInstantLogBlocks()
Option<String>	getPartitionNameOverride()
protected String	getPayloadClassFQN()
protected TypedProperties	getPayloadProps()
float	getProgress() Return progress of scanning as a float between 0.0 to 1.0.
long	getTotalCorruptBlocks()
long	getTotalLogBlocks()
long	getTotalLogFiles()
long	getTotalLogRecords()
long	getTotalRollbacks()
List<String>	getValidBlockInstants()
boolean	isWithOperationField()
protected void	scanInternal(Option<KeySpec> keySpecOpt, boolean skipProcessingBlocks)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected final org.apache.avro.Schema readerSchema

readerContext

protected final HoodieReaderContext<T> readerContext

hoodieTableMetaClient

protected final HoodieTableMetaClient hoodieTableMetaClient

preCombineField

protected final String preCombineField

recordMerger

protected final HoodieRecordMerger recordMerger

recordMergeMode

protected final RecordMergeMode recordMergeMode

logFilePaths

protected final List<String> logFilePaths

forceFullScan

protected final boolean forceFullScan

recordType

protected final HoodieRecord.HoodieRecordType recordType

recordBuffer

protected HoodieFileGroupRecordBuffer<T> recordBuffer

Constructor Detail

BaseHoodieLogRecordReader

protected BaseHoodieLogRecordReader(HoodieReaderContext readerContext,
                                    HoodieStorage storage,
                                    List<String> logFilePaths,
                                    boolean reverseReader,
                                    int bufferSize,
                                    Option<InstantRange> instantRange,
                                    boolean withOperationField,
                                    boolean forceFullScan,
                                    Option<String> partitionNameOverride,
                                    Option<String> keyFieldOverride,
                                    boolean enableOptimizedLogBlocksScan,
                                    HoodieRecordMerger recordMerger,
                                    RecordMergeMode recordMergeMode,
                                    HoodieFileGroupRecordBuffer<T> recordBuffer)

Method Detail

scanInternal

protected final void scanInternal(Option<KeySpec> keySpecOpt,
                                  boolean skipProcessingBlocks)

Parameters:

keySpecOpt - specifies target set of keys to be scanned

skipProcessingBlocks - controls, whether (delta) blocks have to actually be processed

getProgress

public float getProgress()

Return progress of scanning as a float between 0.0 to 1.0.

getTotalLogFiles

public long getTotalLogFiles()

getTotalLogRecords

public long getTotalLogRecords()

getTotalLogBlocks

public long getTotalLogBlocks()

getPayloadClassFQN

protected String getPayloadClassFQN()

getPartitionNameOverride

public Option<String> getPartitionNameOverride()

getTotalRollbacks

public long getTotalRollbacks()

getTotalCorruptBlocks

public long getTotalCorruptBlocks()

isWithOperationField

public boolean isWithOperationField()

getPayloadProps

protected TypedProperties getPayloadProps()

getCurrentInstantLogBlocks

public Deque<HoodieLogBlock> getCurrentInstantLogBlocks()

getValidBlockInstants

public List<String> getValidBlockInstants()