org.apache.hudi.common.table.read

Class HoodieBaseFileGroupRecordBuffer<T>

java.lang.Object

org.apache.hudi.common.table.read.HoodieBaseFileGroupRecordBuffer<T>

All Implemented Interfaces:

HoodieFileGroupRecordBuffer<T>

Direct Known Subclasses:

HoodieKeyBasedFileGroupRecordBuffer, HoodieUnmergedFileGroupRecordBuffer

public abstract class HoodieBaseFileGroupRecordBuffer<T>
extends Object
implements HoodieFileGroupRecordBuffer<T>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.read.HoodieFileGroupRecordBuffer

HoodieFileGroupRecordBuffer.BufferType

Field Summary

Fields

Modifier and Type	Field and Description
protected ClosableIterator<T>	baseFileIterator
protected boolean	enablePartialMerging
protected HoodieTableMetaClient	hoodieTableMetaClient
protected InternalSchema	internalSchema
protected Iterator<Pair<Option<T>,Map<String,Object>>>	logRecordIterator
protected T	nextRecord
protected Option<String>	partitionNameOverrideOpt
protected Option<String[]>	partitionPathFieldOpt
protected TypedProperties	props
protected HoodieReaderContext<T>	readerContext
protected org.apache.avro.Schema	readerSchema
protected RecordMergeMode	recordMergeMode
protected HoodieRecordMerger	recordMerger
protected ExternalSpillableMap<Serializable,Pair<Option<T>,Map<String,Object>>>	records

Constructor Summary

Constructors

Constructor and Description
HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext, HoodieTableMetaClient hoodieTableMetaClient, Option<String> partitionNameOverrideOpt, Option<String[]> partitionPathFieldOpt, HoodieRecordMerger recordMerger, TypedProperties props)

Method Summary

Modifier and Type	Method and Description
void	close()
protected Option<Pair<Function<T,T>,org.apache.avro.Schema>>	composeEvolvedSchemaTransformer(HoodieDataBlock dataBlock) Get final Read Schema for support evolution.
protected abstract boolean	doHasNext() This allows hasNext() to be called multiple times without incrementing the iterator by more than 1 record.
protected Option<Pair<T,Map<String,Object>>>	doProcessNextDataRecord(T record, Map<String,Object> metadata, Pair<Option<T>,Map<String,Object>> existingRecordMetadataPair) Merge two log data records if needed.
protected Option<DeleteRecord>	doProcessNextDeletedRecord(DeleteRecord deleteRecord, Pair<Option<T>,Map<String,Object>> existingRecordMetadataPair) Merge a delete record with another record (data, or delete).
Iterator<Pair<Option<T>,Map<String,Object>>>	getLogRecordIterator()
Map<Serializable,Pair<Option<T>,Map<String,Object>>>	getLogRecords()
protected Pair<ClosableIterator<T>,org.apache.avro.Schema>	getRecordsIterator(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) Create a record iterator for a data block.
protected Pair<Function<T,T>,org.apache.avro.Schema>	getSchemaTransformerWithEvolvedSchema(HoodieDataBlock dataBlock)
boolean	hasNext() Check if next merged record exists.
protected boolean	hasNextBaseRecord(T baseRecord, Pair<Option<T>,Map<String,Object>> logRecordInfo)
protected boolean	hasNextLogRecord()
protected Option<T>	merge(Option<T> older, Map<String,Object> olderInfoMap, Option<T> newer, Map<String,Object> newerInfoMap) Merge two records using the configured record merger.
T	next()
void	setBaseFileIterator(ClosableIterator<T> baseFileIterator) Link the base file iterator for consequential merge.
int	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.table.read.HoodieFileGroupRecordBuffer

containsLogRecord, getBufferType, processDataBlock, processDeleteBlock, processNextDataRecord, processNextDeletedRecord

Field Detail

readerContext

protected final HoodieReaderContext<T> readerContext

readerSchema

protected final org.apache.avro.Schema readerSchema

partitionNameOverrideOpt

protected final Option<String> partitionNameOverrideOpt

partitionPathFieldOpt

protected final Option<String[]> partitionPathFieldOpt

recordMergeMode

protected final RecordMergeMode recordMergeMode

recordMerger

protected final HoodieRecordMerger recordMerger

props

protected final TypedProperties props

records

protected final ExternalSpillableMap<Serializable,Pair<Option<T>,Map<String,Object>>> records

baseFileIterator

protected ClosableIterator<T> baseFileIterator

logRecordIterator

protected Iterator<Pair<Option<T>,Map<String,Object>>> logRecordIterator

nextRecord

protected T nextRecord

enablePartialMerging

protected boolean enablePartialMerging

internalSchema

protected InternalSchema internalSchema

hoodieTableMetaClient

protected HoodieTableMetaClient hoodieTableMetaClient

Constructor Detail

HoodieBaseFileGroupRecordBuffer

public HoodieBaseFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
                                       HoodieTableMetaClient hoodieTableMetaClient,
                                       Option<String> partitionNameOverrideOpt,
                                       Option<String[]> partitionPathFieldOpt,
                                       HoodieRecordMerger recordMerger,
                                       TypedProperties props)

Method Detail

setBaseFileIterator

public void setBaseFileIterator(ClosableIterator<T> baseFileIterator)

Description copied from interface: HoodieFileGroupRecordBuffer

Link the base file iterator for consequential merge.

Specified by:

setBaseFileIterator in interface HoodieFileGroupRecordBuffer<T>

doHasNext

protected abstract boolean doHasNext()
                              throws IOException

This allows hasNext() to be called multiple times without incrementing the iterator by more than 1 record. It does come with the caveat that hasNext() must be called every time before next(). But that is pretty expected behavior and every user should be doing that anyway.

Throws:

IOException

hasNext

public final boolean hasNext()
                      throws IOException

Description copied from interface: HoodieFileGroupRecordBuffer

Check if next merged record exists.

Specified by:

hasNext in interface HoodieFileGroupRecordBuffer<T>

Returns:

true if it has, otherwise false.

Throws:

IOException

next

public final T next()

Specified by:

next in interface HoodieFileGroupRecordBuffer<T>

Returns:

output the next merged record.

getLogRecords

public Map<Serializable,Pair<Option<T>,Map<String,Object>>> getLogRecords()

Specified by:

getLogRecords in interface HoodieFileGroupRecordBuffer<T>

Returns:

The underlying data stored in the buffer.

size

public int size()

Specified by:

size in interface HoodieFileGroupRecordBuffer<T>

Returns:

the number of log records in the buffer.

getLogRecordIterator

public Iterator<Pair<Option<T>,Map<String,Object>>> getLogRecordIterator()

Specified by:

getLogRecordIterator in interface HoodieFileGroupRecordBuffer<T>

Returns:

An iterator on the log records.

close

public void close()

Specified by:

close in interface HoodieFileGroupRecordBuffer<T>

doProcessNextDataRecord

protected Option<Pair<T,Map<String,Object>>> doProcessNextDataRecord(T record,
                                                                     Map<String,Object> metadata,
                                                                     Pair<Option<T>,Map<String,Object>> existingRecordMetadataPair)
                                                              throws IOException

Merge two log data records if needed.

Parameters:

record -

metadata -

existingRecordMetadataPair -

Returns:

Throws:

IOException

doProcessNextDeletedRecord

protected Option<DeleteRecord> doProcessNextDeletedRecord(DeleteRecord deleteRecord,
                                                          Pair<Option<T>,Map<String,Object>> existingRecordMetadataPair)

Merge a delete record with another record (data, or delete).

Parameters:

deleteRecord -

existingRecordMetadataPair -

Returns:

getRecordsIterator

protected Pair<ClosableIterator<T>,org.apache.avro.Schema> getRecordsIterator(HoodieDataBlock dataBlock,
                                                                              Option<KeySpec> keySpecOpt)

Create a record iterator for a data block. The records are filtered by a key set specified by keySpecOpt.

Parameters:

dataBlock -

keySpecOpt -

Returns:

composeEvolvedSchemaTransformer

protected Option<Pair<Function<T,T>,org.apache.avro.Schema>> composeEvolvedSchemaTransformer(HoodieDataBlock dataBlock)

Get final Read Schema for support evolution. step1: find the fileSchema for current dataBlock. step2: determine whether fileSchema is compatible with the final read internalSchema. step3: merge fileSchema and read internalSchema to produce final read schema.

Parameters:

dataBlock - current processed block

Returns:

final read schema.

merge

protected Option<T> merge(Option<T> older,
                          Map<String,Object> olderInfoMap,
                          Option<T> newer,
                          Map<String,Object> newerInfoMap)
                   throws IOException

Merge two records using the configured record merger.

Parameters:

older -

olderInfoMap -

newer -

newerInfoMap -

Returns:

Throws:

IOException

hasNextBaseRecord

protected boolean hasNextBaseRecord(T baseRecord,
                                    Pair<Option<T>,Map<String,Object>> logRecordInfo)
                             throws IOException

Throws:

IOException

hasNextLogRecord

protected boolean hasNextLogRecord()
                            throws IOException

Throws:

IOException

getSchemaTransformerWithEvolvedSchema

protected Pair<Function<T,T>,org.apache.avro.Schema> getSchemaTransformerWithEvolvedSchema(HoodieDataBlock dataBlock)