org.apache.hudi.common.table.read

Class HoodiePositionBasedFileGroupRecordBuffer<T>

java.lang.Object

org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer<T>

org.apache.hudi.common.table.read.HoodieKeyBasedFileGroupRecordBuffer<T>

org.apache.hudi.common.table.read.HoodiePositionBasedFileGroupRecordBuffer<T>

All Implemented Interfaces:

HoodieFileGroupRecordBuffer<T>

public class HoodiePositionBasedFileGroupRecordBuffer<T>
extends HoodieKeyBasedFileGroupRecordBuffer<T>

A buffer that is used to store log records by HoodieMergedLogRecordReader by calling the processDataBlock(org.apache.hudi.common.table.log.block.HoodieDataBlock, org.apache.hudi.common.util.Option<org.apache.hudi.common.table.log.KeySpec>) and processDeleteBlock(org.apache.hudi.common.table.log.block.HoodieDeleteBlock) methods into record position based map. Here the position means that record position in the base file. The records from the base file is accessed from an iterator object. These records are merged when the HoodieBaseFileGroupRecordBuffer.hasNext() method is called.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.read.HoodieFileGroupRecordBuffer

HoodieFileGroupRecordBuffer.BufferType

Field Summary

Fields

Modifier and Type	Field and Description
static String	ROW_INDEX_TEMPORARY_COLUMN_NAME

Fields inherited from class org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer

baseFileIterator, enablePartialMerging, hoodieTableMetaClient, internalSchema, logRecordIterator, nextRecord, partitionNameOverrideOpt, partitionPathFieldOpt, props, readerContext, readerSchema, recordMergeMode, recordMerger, records

Constructor Summary

Constructors

Constructor and Description
HoodiePositionBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext, HoodieTableMetaClient hoodieTableMetaClient, Option<String> partitionNameOverrideOpt, Option<String[]> partitionPathFieldOpt, HoodieRecordMerger recordMerger, TypedProperties props)

Method Summary

Modifier and Type	Method and Description
boolean	containsLogRecord(String recordKey) Check if a record exists in the buffered records.
protected static List<Long>	extractRecordPositions(HoodieLogBlock logBlock) Extract the record positions from a log block header.
HoodieFileGroupRecordBuffer.BufferType	getBufferType()
protected boolean	hasNextBaseRecord(T baseRecord)
void	processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) Process a log data block, and store the resulting records into the buffer.
void	processDeleteBlock(HoodieDeleteBlock deleteBlock) Process a log delete block, and store the resulting records into the buffer.
void	processNextDeletedRecord(DeleteRecord deleteRecord, Serializable recordPosition) Process next delete record.
protected boolean	shouldSkip(T record, String keyFieldName, boolean isFullKey, Set<String> keys, org.apache.avro.Schema writerSchema) Filter a record for downstream processing when: 1.

Methods inherited from class org.apache.hudi.common.table.read.HoodieKeyBasedFileGroupRecordBuffer

doHasNext, processNextDataRecord

Methods inherited from class org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer

close, composeEvolvedSchemaTransformer, doProcessNextDataRecord, doProcessNextDeletedRecord, getLogRecordIterator, getLogRecords, getRecordsIterator, getSchemaTransformerWithEvolvedSchema, hasNext, hasNextBaseRecord, hasNextLogRecord, merge, next, setBaseFileIterator, size

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ROW_INDEX_TEMPORARY_COLUMN_NAME

public static final String ROW_INDEX_TEMPORARY_COLUMN_NAME

See Also:

Constant Field Values

Constructor Detail

HoodiePositionBasedFileGroupRecordBuffer

public HoodiePositionBasedFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
                                                HoodieTableMetaClient hoodieTableMetaClient,
                                                Option<String> partitionNameOverrideOpt,
                                                Option<String[]> partitionPathFieldOpt,
                                                HoodieRecordMerger recordMerger,
                                                TypedProperties props)

Method Detail

getBufferType

public HoodieFileGroupRecordBuffer.BufferType getBufferType()

Specified by:

getBufferType in interface HoodieFileGroupRecordBuffer<T>

Overrides:

getBufferType in class HoodieKeyBasedFileGroupRecordBuffer<T>

Returns:

The merge strategy implemented.

processDataBlock

public void processDataBlock(HoodieDataBlock dataBlock,
                             Option<KeySpec> keySpecOpt)
                      throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Process a log data block, and store the resulting records into the buffer.

Specified by:

processDataBlock in interface HoodieFileGroupRecordBuffer<T>

Overrides:

processDataBlock in class HoodieKeyBasedFileGroupRecordBuffer<T>

Throws:

IOException

processDeleteBlock

public void processDeleteBlock(HoodieDeleteBlock deleteBlock)
                        throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Process a log delete block, and store the resulting records into the buffer.

Specified by:

processDeleteBlock in interface HoodieFileGroupRecordBuffer<T>

Overrides:

processDeleteBlock in class HoodieKeyBasedFileGroupRecordBuffer<T>

Throws:

IOException

processNextDeletedRecord

public void processNextDeletedRecord(DeleteRecord deleteRecord,
                                     Serializable recordPosition)

Description copied from interface: HoodieFileGroupRecordBuffer

Process next delete record.

Specified by:

processNextDeletedRecord in interface HoodieFileGroupRecordBuffer<T>

Overrides:

processNextDeletedRecord in class HoodieKeyBasedFileGroupRecordBuffer<T>

containsLogRecord

public boolean containsLogRecord(String recordKey)

Description copied from interface: HoodieFileGroupRecordBuffer

Check if a record exists in the buffered records.

Specified by:

containsLogRecord in interface HoodieFileGroupRecordBuffer<T>

Overrides:

containsLogRecord in class HoodieKeyBasedFileGroupRecordBuffer<T>

hasNextBaseRecord

protected boolean hasNextBaseRecord(T baseRecord)
                             throws IOException

Overrides:

hasNextBaseRecord in class HoodieKeyBasedFileGroupRecordBuffer<T>

Throws:

IOException

shouldSkip

protected boolean shouldSkip(T record,
                             String keyFieldName,
                             boolean isFullKey,
                             Set<String> keys,
                             org.apache.avro.Schema writerSchema)

Filter a record for downstream processing when: 1. A set of pre-specified keys exists. 2. The key of the record is not contained in the set.

extractRecordPositions

protected static List<Long> extractRecordPositions(HoodieLogBlock logBlock)
                                            throws IOException

Extract the record positions from a log block header.

Parameters:

logBlock -

Returns:

Throws:

IOException