org.apache.hudi.common.table.log

Class HoodieMergedLogRecordScanner

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

org.apache.hudi.common.table.log.BaseHoodieMergedLogRecordScanner<String>

org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<HoodieRecord>

@NotThreadSafe
public class HoodieMergedLogRecordScanner
extends BaseHoodieMergedLogRecordScanner<String>

Scans through all the blocks in a list of HoodieLogFile and builds up a compacted/merged list of records which will be used as a lookup table when merging the base columnar file with the redo log file.

NOTE: If readBlockLazily is turned on, does not merge, instead keeps reading log blocks and merges everything at once This is an optimization to avoid seek() back and forth to read new block (forward seek()) and lazily read content of seen block (reverse and forward seek()) during merge | | Read Block 1 Metadata | | Read Block 1 Data | | | Read Block 2 Metadata | | Read Block 2 Data | | I/O Pass 1 | ..................... | I/O Pass 2 | ................. | | | Read Block N Metadata | | Read Block N Data |

This results in two I/O passes over the log file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieMergedLogRecordScanner.Builder Builder used to build HoodieMergedLogRecordScanner.

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.BaseHoodieMergedLogRecordScanner

records, timer

Fields inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

forceFullScan, hoodieTableMetaClient, logFilePaths, preCombineField, readerSchema, recordMerger, recordType

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

HoodieMergedLogRecordScanner(HoodieStorage storage, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, Long maxMemorySizeInBytes, boolean reverseReader, int bufferSize, String spillableMapBasePath, Option<InstantRange> instantRange, ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled, boolean withOperationField, boolean forceFullScan, Option<String> partitionName, InternalSchema internalSchema, Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger, Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Summary

Modifier and Type	Method and Description
Map<String,HoodieRecord>	getRecords()
static HoodieMergedLogRecordScanner.Builder	newBuilder() Returns the builder for HoodieMergedLogRecordScanner.
protected <T> void	processNextRecord(HoodieRecord<T> newRecord) Process next record.

Methods inherited from class org.apache.hudi.common.table.log.BaseHoodieMergedLogRecordScanner

close, getLatestHoodieRecord, getNumMergedRecordsInLog, getRecordType, getTotalTimeTakenToReadAndMergeBlocks, iterator, processNextDeletedRecord, scan, scan, scanByFullKeys, scanByKeyPrefixes

Methods inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

getCurrentInstantLogBlocks, getPartitionNameOverride, getPayloadClassFQN, getPayloadProps, getProgress, getTotalCorruptBlocks, getTotalLogBlocks, getTotalLogFiles, getTotalLogRecords, getTotalRollbacks, getValidBlockInstants, isWithOperationField, scanInternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

HoodieMergedLogRecordScanner

protected HoodieMergedLogRecordScanner(HoodieStorage storage,
                                       String basePath,
                                       List<String> logFilePaths,
                                       org.apache.avro.Schema readerSchema,
                                       String latestInstantTime,
                                       Long maxMemorySizeInBytes,
                                       boolean reverseReader,
                                       int bufferSize,
                                       String spillableMapBasePath,
                                       Option<InstantRange> instantRange,
                                       ExternalSpillableMap.DiskMapType diskMapType,
                                       boolean isBitCaskDiskMapCompressionEnabled,
                                       boolean withOperationField,
                                       boolean forceFullScan,
                                       Option<String> partitionName,
                                       InternalSchema internalSchema,
                                       Option<String> keyFieldOverride,
                                       boolean enableOptimizedLogBlocksScan,
                                       HoodieRecordMerger recordMerger,
                                       Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Detail

getRecords

public Map<String,HoodieRecord> getRecords()

Specified by:

getRecords in class BaseHoodieMergedLogRecordScanner<String>

processNextRecord

protected <T> void processNextRecord(HoodieRecord<T> newRecord)
                              throws IOException

Description copied from class: AbstractHoodieLogRecordReader

Process next record.

Specified by:

processNextRecord in class AbstractHoodieLogRecordReader

Parameters:

newRecord - Hoodie Record to process

Throws:

IOException

newBuilder

public static HoodieMergedLogRecordScanner.Builder newBuilder()

Returns the builder for HoodieMergedLogRecordScanner.