org.apache.hudi.common.table.log

Class BaseHoodieMergedLogRecordScanner<K extends Serializable>

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

org.apache.hudi.common.table.log.BaseHoodieMergedLogRecordScanner<K>

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<HoodieRecord>

Direct Known Subclasses:

HoodieMergedLogRecordScanner, HoodieMetadataMergedLogRecordScanner

public abstract class BaseHoodieMergedLogRecordScanner<K extends Serializable>
extends AbstractHoodieLogRecordReader
implements Iterable<HoodieRecord>, Closeable

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

AbstractHoodieLogRecordReader.Builder, AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields

Modifier and Type	Field and Description
protected ExternalSpillableMap<K,HoodieRecord>	records
HoodieTimer	timer

Fields inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

forceFullScan, hoodieTableMetaClient, logFilePaths, preCombineField, readerSchema, recordMerger, recordType

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

BaseHoodieMergedLogRecordScanner(HoodieStorage storage, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, Long maxMemorySizeInBytes, boolean reverseReader, int bufferSize, String spillableMapBasePath, Option<InstantRange> instantRange, ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled, boolean withOperationField, boolean forceFullScan, Option<String> partitionName, InternalSchema internalSchema, Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger, Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Summary

Modifier and Type	Method and Description
void	close()
protected static <T> HoodieRecord	getLatestHoodieRecord(HoodieRecord<T> newRecord, HoodieRecord<T> combinedRecord, String key)
long	getNumMergedRecordsInLog()
abstract Map<K,HoodieRecord>	getRecords()
HoodieRecord.HoodieRecordType	getRecordType()
long	getTotalTimeTakenToReadAndMergeBlocks()
Iterator<HoodieRecord>	iterator()
protected void	processNextDeletedRecord(DeleteRecord deleteRecord) Process next deleted record.
void	scan() Scans delta-log files processing blocks
void	scan(boolean skipProcessingBlocks)
void	scanByFullKeys(List<String> keys) Provides incremental scanning capability where only provided keys will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache
void	scanByKeyPrefixes(List<String> keyPrefixes) Provides incremental scanning capability where only keys matching provided key-prefixes will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Methods inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

getCurrentInstantLogBlocks, getPartitionNameOverride, getPayloadClassFQN, getPayloadProps, getProgress, getTotalCorruptBlocks, getTotalLogBlocks, getTotalLogFiles, getTotalLogRecords, getTotalRollbacks, getValidBlockInstants, isWithOperationField, processNextRecord, scanInternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

timer

public final HoodieTimer timer

records

protected final ExternalSpillableMap<K extends Serializable,HoodieRecord> records

Constructor Detail

BaseHoodieMergedLogRecordScanner

protected BaseHoodieMergedLogRecordScanner(HoodieStorage storage,
                                           String basePath,
                                           List<String> logFilePaths,
                                           org.apache.avro.Schema readerSchema,
                                           String latestInstantTime,
                                           Long maxMemorySizeInBytes,
                                           boolean reverseReader,
                                           int bufferSize,
                                           String spillableMapBasePath,
                                           Option<InstantRange> instantRange,
                                           ExternalSpillableMap.DiskMapType diskMapType,
                                           boolean isBitCaskDiskMapCompressionEnabled,
                                           boolean withOperationField,
                                           boolean forceFullScan,
                                           Option<String> partitionName,
                                           InternalSchema internalSchema,
                                           Option<String> keyFieldOverride,
                                           boolean enableOptimizedLogBlocksScan,
                                           HoodieRecordMerger recordMerger,
                                           Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Detail

scan

public final void scan()

Scans delta-log files processing blocks

scan

public final void scan(boolean skipProcessingBlocks)

scanByFullKeys

public void scanByFullKeys(List<String> keys)

Provides incremental scanning capability where only provided keys will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Parameters:

keys - to be looked up

scanByKeyPrefixes

public void scanByKeyPrefixes(List<String> keyPrefixes)

Provides incremental scanning capability where only keys matching provided key-prefixes will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Parameters:

keyPrefixes - to be looked up

iterator

public Iterator<HoodieRecord> iterator()

Specified by:

iterator in interface Iterable<HoodieRecord>

processNextDeletedRecord

protected void processNextDeletedRecord(DeleteRecord deleteRecord)

Description copied from class: AbstractHoodieLogRecordReader

Process next deleted record.

Specified by:

processNextDeletedRecord in class AbstractHoodieLogRecordReader

Parameters:

deleteRecord - Deleted record(hoodie key and ordering value)

getRecords

public abstract Map<K,HoodieRecord> getRecords()

getRecordType

public HoodieRecord.HoodieRecordType getRecordType()

getNumMergedRecordsInLog

public long getNumMergedRecordsInLog()

getLatestHoodieRecord

protected static <T> HoodieRecord getLatestHoodieRecord(HoodieRecord<T> newRecord,
                                                        HoodieRecord<T> combinedRecord,
                                                        String key)

getTotalTimeTakenToReadAndMergeBlocks

public long getTotalTimeTakenToReadAndMergeBlocks()

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable