org.apache.hudi.common.table.read

Class HoodieKeyBasedFileGroupRecordBuffer<T>

java.lang.Object

org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer<T>

org.apache.hudi.common.table.read.HoodieKeyBasedFileGroupRecordBuffer<T>

All Implemented Interfaces:

HoodieFileGroupRecordBuffer<T>

Direct Known Subclasses:

HoodiePositionBasedFileGroupRecordBuffer

public class HoodieKeyBasedFileGroupRecordBuffer<T>
extends HoodieBaseFileGroupRecordBuffer<T>

A buffer that is used to store log records by HoodieMergedLogRecordReader by calling the processDataBlock(org.apache.hudi.common.table.log.block.HoodieDataBlock, org.apache.hudi.common.util.Option<org.apache.hudi.common.table.log.KeySpec>) and processDeleteBlock(org.apache.hudi.common.table.log.block.HoodieDeleteBlock) methods into a record key based map. The records from the base file is accessed from an iterator object. These records are merged when the HoodieBaseFileGroupRecordBuffer.hasNext() method is called.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.read.HoodieFileGroupRecordBuffer

HoodieFileGroupRecordBuffer.BufferType

Field Summary

Fields inherited from class org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer

baseFileIterator, enablePartialMerging, hoodieTableMetaClient, internalSchema, logRecordIterator, nextRecord, partitionNameOverrideOpt, partitionPathFieldOpt, props, readerContext, readerSchema, recordMergeMode, recordMerger, records

Constructor Summary

Constructors

Constructor and Description
HoodieKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext, HoodieTableMetaClient hoodieTableMetaClient, Option<String> partitionNameOverrideOpt, Option<String[]> partitionPathFieldOpt, HoodieRecordMerger recordMerger, TypedProperties props)

Method Summary

Modifier and Type	Method and Description
boolean	containsLogRecord(String recordKey) Check if a record exists in the buffered records.
protected boolean	doHasNext() This allows hasNext() to be called multiple times without incrementing the iterator by more than 1 record.
HoodieFileGroupRecordBuffer.BufferType	getBufferType()
protected boolean	hasNextBaseRecord(T baseRecord)
void	processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) Process a log data block, and store the resulting records into the buffer.
void	processDeleteBlock(HoodieDeleteBlock deleteBlock) Process a log delete block, and store the resulting records into the buffer.
void	processNextDataRecord(T record, Map<String,Object> metadata, Serializable recordKey) Process a next record in a log data block.
void	processNextDeletedRecord(DeleteRecord deleteRecord, Serializable recordKey) Process next delete record.

Methods inherited from class org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer

close, composeEvolvedSchemaTransformer, doProcessNextDataRecord, doProcessNextDeletedRecord, getLogRecordIterator, getLogRecords, getRecordsIterator, getSchemaTransformerWithEvolvedSchema, hasNext, hasNextBaseRecord, hasNextLogRecord, merge, next, setBaseFileIterator, size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieKeyBasedFileGroupRecordBuffer

public HoodieKeyBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
                                           HoodieTableMetaClient hoodieTableMetaClient,
                                           Option<String> partitionNameOverrideOpt,
                                           Option<String[]> partitionPathFieldOpt,
                                           HoodieRecordMerger recordMerger,
                                           TypedProperties props)

Method Detail

getBufferType

public HoodieFileGroupRecordBuffer.BufferType getBufferType()

Returns:

The merge strategy implemented.

processDataBlock

public void processDataBlock(HoodieDataBlock dataBlock,
                             Option<KeySpec> keySpecOpt)
                      throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Process a log data block, and store the resulting records into the buffer.

Throws:

IOException

processNextDataRecord

public void processNextDataRecord(T record,
                                  Map<String,Object> metadata,
                                  Serializable recordKey)
                           throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Process a next record in a log data block.

Throws:

IOException

processDeleteBlock

public void processDeleteBlock(HoodieDeleteBlock deleteBlock)
                        throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Process a log delete block, and store the resulting records into the buffer.

Throws:

IOException

processNextDeletedRecord

public void processNextDeletedRecord(DeleteRecord deleteRecord,
                                     Serializable recordKey)

Description copied from interface: HoodieFileGroupRecordBuffer

Process next delete record.

containsLogRecord

public boolean containsLogRecord(String recordKey)

Description copied from interface: HoodieFileGroupRecordBuffer

Check if a record exists in the buffered records.

hasNextBaseRecord

protected boolean hasNextBaseRecord(T baseRecord)
                             throws IOException

Throws:

IOException

doHasNext

protected boolean doHasNext()
                     throws IOException

Description copied from class: HoodieBaseFileGroupRecordBuffer

This allows hasNext() to be called multiple times without incrementing the iterator by more than 1 record. It does come with the caveat that hasNext() must be called every time before next(). But that is pretty expected behavior and every user should be doing that anyway.

Specified by:

doHasNext in class HoodieBaseFileGroupRecordBuffer<T>

Throws:

IOException