org.apache.hudi.common.table.log.block

Class HoodieParquetDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

org.apache.hudi.common.table.log.block.HoodieParquetDataBlock

public class HoodieParquetDataBlock
extends HoodieDataBlock

HoodieParquetDataBlock contains a list of records serialized using Parquet.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

readerSchema, shouldWriteRecordPositions

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Constructor and Description
HoodieParquetDataBlock(List<HoodieRecord> records, boolean shouldWriteRecordPositions, Map<HoodieLogBlock.HeaderMetadataType,String> header, String keyField, String compressionCodecName, double expectedCompressionRatio, boolean useDictionaryEncoding)
HoodieParquetDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier, Option<byte[]> content, boolean readBlockLazily, HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyField)

Method Summary

Modifier and Type	Method and Description
protected <T> ClosableIterator<HoodieRecord<T>>	deserializeRecords(byte[] content, HoodieRecord.HoodieRecordType type)
protected <T> ClosableIterator<T>	deserializeRecords(HoodieReaderContext<T> readerContext, byte[] content) Deserializes the content bytes of the data block to the records in engine-specific representation.
HoodieLogBlock.HoodieLogBlockType	getBlockType()
protected <T> ClosableIterator<T>	readRecordsFromBlockPayload(HoodieReaderContext<T> readerContext)
protected <T> ClosableIterator<HoodieRecord<T>>	readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type) NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)
protected byte[]	serializeRecords(List<HoodieRecord> records, HoodieStorage storage)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

containsPartialUpdates, getContentBytes, getEngineRecordIterator, getEngineRecordIterator, getKeyField, getKeyFieldName, getRecordIterator, getRecordIterator, getRecordKey, getSchema, getSchemaFromHeader, getWriterSchema, lookupRecords

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

addRecordPositionsToHeader, deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, getRecordPositions, inflate, isCompactedLogBlock, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieParquetDataBlock

public HoodieParquetDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier,
                              Option<byte[]> content,
                              boolean readBlockLazily,
                              HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation,
                              Option<org.apache.avro.Schema> readerSchema,
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                              String keyField)

HoodieParquetDataBlock

public HoodieParquetDataBlock(List<HoodieRecord> records,
                              boolean shouldWriteRecordPositions,
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              String keyField,
                              String compressionCodecName,
                              double expectedCompressionRatio,
                              boolean useDictionaryEncoding)

Method Detail

getBlockType

public HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieDataBlock

serializeRecords

protected byte[] serializeRecords(List<HoodieRecord> records,
                                  HoodieStorage storage)
                           throws IOException

Specified by:

serializeRecords in class HoodieDataBlock

Throws:

IOException

readRecordsFromBlockPayload

protected <T> ClosableIterator<HoodieRecord<T>> readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)

Overrides:

readRecordsFromBlockPayload in class HoodieDataBlock

Throws:

IOException

readRecordsFromBlockPayload

protected <T> ClosableIterator<T> readRecordsFromBlockPayload(HoodieReaderContext<T> readerContext)
                                                       throws IOException

Overrides:

readRecordsFromBlockPayload in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected <T> ClosableIterator<HoodieRecord<T>> deserializeRecords(byte[] content,
                                                                   HoodieRecord.HoodieRecordType type)
                                                            throws IOException

Specified by:

deserializeRecords in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected <T> ClosableIterator<T> deserializeRecords(HoodieReaderContext<T> readerContext,
                                                     byte[] content)
                                              throws IOException

Description copied from class: HoodieDataBlock

Deserializes the content bytes of the data block to the records in engine-specific representation.

Specified by:

deserializeRecords in class HoodieDataBlock

Type Parameters:

T - Record type of engine-specific representation.

Parameters:

readerContext - Hudi reader context with engine-specific implementation.

content - Content in byte array.

Returns:

ClosableIterator of records in engine-specific representation.

Throws:

IOException - upon deserialization error.