org.apache.hudi.common.table.log.block

Class HoodieDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

Direct Known Subclasses:

HoodieAvroDataBlock, HoodieHFileDataBlock, HoodieParquetDataBlock

public abstract class HoodieDataBlock
extends HoodieLogBlock

DataBlock contains a list of records serialized using formats compatible with the base file format. For each base file format there is a corresponding DataBlock format.

The Datablock contains: 1. Data Block version 2. Total number of records in the block 3. Actual serialized content of the records

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.avro.Schema	readerSchema
protected boolean	shouldWriteRecordPositions

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieDataBlock(List<HoodieRecord> records, boolean shouldWriteRecordPositions, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
protected	HoodieDataBlock(Option<byte[]> content, Supplier<SeekableDataInputStream> inputStreamSupplier, boolean readBlockLazily, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> headers, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName, boolean enablePointLookups) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

Method Summary

Modifier and Type	Method and Description
boolean	containsPartialUpdates()
protected abstract <T> ClosableIterator<HoodieRecord<T>>	deserializeRecords(byte[] content, HoodieRecord.HoodieRecordType type)
protected abstract <T> ClosableIterator<T>	deserializeRecords(HoodieReaderContext<T> readerContext, byte[] content) Deserializes the content bytes of the data block to the records in engine-specific representation.
abstract HoodieLogBlock.HoodieLogBlockType	getBlockType()
byte[]	getContentBytes(HoodieStorage storage)
<T> ClosableIterator<T>	getEngineRecordIterator(HoodieReaderContext<T> readerContext) Returns all the records in the type of engine-specific record representation contained within this block in an iterator.
<T> ClosableIterator<T>	getEngineRecordIterator(HoodieReaderContext<T> readerContext, List<String> keys, boolean fullKey) Batch get of keys of interest.
protected Option<org.apache.avro.Schema.Field>	getKeyField(org.apache.avro.Schema schema)
String	getKeyFieldName()
<T> ClosableIterator<HoodieRecord<T>>	getRecordIterator(HoodieRecord.HoodieRecordType type) Returns all the records iterator contained w/in this block.
<T> ClosableIterator<HoodieRecord<T>>	getRecordIterator(List<String> keys, boolean fullKey, HoodieRecord.HoodieRecordType type) Batch get of keys of interest.
protected Option<String>	getRecordKey(HoodieRecord record)
org.apache.avro.Schema	getSchema()
protected org.apache.avro.Schema	getSchemaFromHeader()
protected static org.apache.avro.Schema	getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)
protected <T> ClosableIterator<HoodieRecord<T>>	lookupRecords(List<String> keys, boolean fullKey)
protected <T> ClosableIterator<T>	readRecordsFromBlockPayload(HoodieReaderContext<T> readerContext)
protected <T> ClosableIterator<HoodieRecord<T>>	readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
protected abstract byte[]	serializeRecords(List<HoodieRecord> records, HoodieStorage storage)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

addRecordPositionsToHeader, deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, getRecordPositions, inflate, isCompactedLogBlock, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected org.apache.avro.Schema readerSchema

shouldWriteRecordPositions

protected final boolean shouldWriteRecordPositions

Constructor Detail

HoodieDataBlock

public HoodieDataBlock(List<HoodieRecord> records,
                       boolean shouldWriteRecordPositions,
                       Map<HoodieLogBlock.HeaderMetadataType,String> header,
                       Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                       String keyFieldName)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

HoodieDataBlock

protected HoodieDataBlock(Option<byte[]> content,
                          Supplier<SeekableDataInputStream> inputStreamSupplier,
                          boolean readBlockLazily,
                          Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                          Option<org.apache.avro.Schema> readerSchema,
                          Map<HoodieLogBlock.HeaderMetadataType,String> headers,
                          Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                          String keyFieldName,
                          boolean enablePointLookups)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

Method Detail

getContentBytes

public byte[] getContentBytes(HoodieStorage storage)
                       throws IOException

Overrides:

getContentBytes in class HoodieLogBlock

Throws:

IOException

getKeyFieldName

public String getKeyFieldName()

containsPartialUpdates

public boolean containsPartialUpdates()

getWriterSchema

protected static org.apache.avro.Schema getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)

getRecordIterator

public final <T> ClosableIterator<HoodieRecord<T>> getRecordIterator(HoodieRecord.HoodieRecordType type)

Returns all the records iterator contained w/in this block.

getSchema

public org.apache.avro.Schema getSchema()

getRecordIterator

public final <T> ClosableIterator<HoodieRecord<T>> getRecordIterator(List<String> keys,
                                                                     boolean fullKey,
                                                                     HoodieRecord.HoodieRecordType type)
                                                              throws IOException

Batch get of keys of interest. Implementation can choose to either do full scan and return matched entries or do a seek based parsing and return matched entries.

Parameters:

keys - keys of interest.

Returns:

List of IndexedRecords for the keys of interest.

Throws:

IOException - in case of failures encountered when reading/parsing records

getEngineRecordIterator

public final <T> ClosableIterator<T> getEngineRecordIterator(HoodieReaderContext<T> readerContext)

Returns all the records in the type of engine-specific record representation contained within this block in an iterator.

Type Parameters:

T - The type of engine-specific record representation to return.

Parameters:

readerContext - HoodieReaderContext instance with type T.

Returns:

An iterator containing all records in specified type.

getEngineRecordIterator

public final <T> ClosableIterator<T> getEngineRecordIterator(HoodieReaderContext<T> readerContext,
                                                             List<String> keys,
                                                             boolean fullKey)

Batch get of keys of interest. Implementation can choose to either do full scan and return matched entries or do a seek based parsing and return matched entries.

Type Parameters:

T - The type of engine-specific record representation to return.

Parameters:

readerContext - HoodieReaderContext instance with type T.

keys - Keys of interest.

fullKey - Whether the key is full or not.

Returns:

An iterator containing the records of interest in specified type.

readRecordsFromBlockPayload

protected <T> ClosableIterator<HoodieRecord<T>> readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

Throws:

IOException

readRecordsFromBlockPayload

protected <T> ClosableIterator<T> readRecordsFromBlockPayload(HoodieReaderContext<T> readerContext)
                                                       throws IOException

Throws:

IOException

lookupRecords

protected <T> ClosableIterator<HoodieRecord<T>> lookupRecords(List<String> keys,
                                                              boolean fullKey)
                                                       throws IOException

Throws:

IOException

serializeRecords

protected abstract byte[] serializeRecords(List<HoodieRecord> records,
                                           HoodieStorage storage)
                                    throws IOException

Throws:

IOException

deserializeRecords

protected abstract <T> ClosableIterator<HoodieRecord<T>> deserializeRecords(byte[] content,
                                                                            HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

Throws:

IOException

deserializeRecords

protected abstract <T> ClosableIterator<T> deserializeRecords(HoodieReaderContext<T> readerContext,
                                                              byte[] content)
                                                       throws IOException

Deserializes the content bytes of the data block to the records in engine-specific representation.

Type Parameters:

T - Record type of engine-specific representation.

Parameters:

readerContext - Hudi reader context with engine-specific implementation.

content - Content in byte array.

Returns:

ClosableIterator of records in engine-specific representation.

Throws:

IOException - upon deserialization error.

getBlockType

public abstract HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieLogBlock

getKeyField

protected Option<org.apache.avro.Schema.Field> getKeyField(org.apache.avro.Schema schema)

getRecordKey

protected Option<String> getRecordKey(HoodieRecord record)

getSchemaFromHeader

protected org.apache.avro.Schema getSchemaFromHeader()