org.apache.hudi.common.table.log.block

Class HoodieAvroDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

org.apache.hudi.common.table.log.block.HoodieAvroDataBlock

Direct Known Subclasses:

HoodieCDCDataBlock

public class HoodieAvroDataBlock
extends HoodieDataBlock

HoodieAvroDataBlock contains a list of records serialized using Avro. It is used with the Parquet base file format.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

readerSchema, shouldWriteRecordPositions

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Constructor and Description
HoodieAvroDataBlock(List<HoodieRecord> records, boolean shouldWriteRecordPositions, Map<HoodieLogBlock.HeaderMetadataType,String> header, String keyField)
HoodieAvroDataBlock(List<HoodieRecord> records, org.apache.avro.Schema schema) Deprecated.
HoodieAvroDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier, Option<byte[]> content, boolean readBlockLazily, HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyField)

Method Summary

Modifier and Type	Method and Description
protected <T> ClosableIterator<HoodieRecord<T>>	deserializeRecords(byte[] content, HoodieRecord.HoodieRecordType type)
protected <T> ClosableIterator<T>	deserializeRecords(HoodieReaderContext<T> readerContext, byte[] content) Deserializes the content bytes of the data block to the records in engine-specific representation.
static HoodieAvroDataBlock	getBlock(byte[] content, org.apache.avro.Schema readerSchema)
static HoodieAvroDataBlock	getBlock(byte[] content, org.apache.avro.Schema readerSchema, InternalSchema internalSchema) Deprecated.
HoodieLogBlock.HoodieLogBlockType	getBlockType()
byte[]	getBytes(org.apache.avro.Schema schema) Deprecated.
protected byte[]	serializeRecords(List<HoodieRecord> records, HoodieStorage storage)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

containsPartialUpdates, getContentBytes, getEngineRecordIterator, getEngineRecordIterator, getKeyField, getKeyFieldName, getRecordIterator, getRecordIterator, getRecordKey, getSchema, getSchemaFromHeader, getWriterSchema, lookupRecords, readRecordsFromBlockPayload, readRecordsFromBlockPayload

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

addRecordPositionsToHeader, deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, getRecordPositions, inflate, isCompactedLogBlock, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieAvroDataBlock

public HoodieAvroDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier,
                           Option<byte[]> content,
                           boolean readBlockLazily,
                           HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation,
                           Option<org.apache.avro.Schema> readerSchema,
                           Map<HoodieLogBlock.HeaderMetadataType,String> header,
                           Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                           String keyField)

HoodieAvroDataBlock

public HoodieAvroDataBlock(@Nonnull
                           List<HoodieRecord> records,
                           boolean shouldWriteRecordPositions,
                           @Nonnull
                           Map<HoodieLogBlock.HeaderMetadataType,String> header,
                           @Nonnull
                           String keyField)

HoodieAvroDataBlock

@Deprecated
public HoodieAvroDataBlock(List<HoodieRecord> records,
                                       org.apache.avro.Schema schema)

Deprecated.

This constructor is retained to provide backwards compatibility to HoodieArchivedLogs which were written using HoodieLogFormat V1.

Method Detail

getBlockType

public HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieDataBlock

serializeRecords

protected byte[] serializeRecords(List<HoodieRecord> records,
                                  HoodieStorage storage)
                           throws IOException

Specified by:

serializeRecords in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected <T> ClosableIterator<HoodieRecord<T>> deserializeRecords(byte[] content,
                                                                   HoodieRecord.HoodieRecordType type)
                                                            throws IOException

Specified by:

deserializeRecords in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected <T> ClosableIterator<T> deserializeRecords(HoodieReaderContext<T> readerContext,
                                                     byte[] content)
                                              throws IOException

Description copied from class: HoodieDataBlock

Deserializes the content bytes of the data block to the records in engine-specific representation.

Specified by:

deserializeRecords in class HoodieDataBlock

Type Parameters:

T - Record type of engine-specific representation.

Parameters:

readerContext - Hudi reader context with engine-specific implementation.

content - Content in byte array.

Returns:

ClosableIterator of records in engine-specific representation.

Throws:

IOException - upon deserialization error.

getBlock

public static HoodieAvroDataBlock getBlock(byte[] content,
                                           org.apache.avro.Schema readerSchema)
                                    throws IOException

Throws:

IOException

getBlock

@Deprecated
public static HoodieAvroDataBlock getBlock(byte[] content,
                                                       org.apache.avro.Schema readerSchema,
                                                       InternalSchema internalSchema)
                                                throws IOException

Deprecated.

This method is retained to provide backwards compatibility to HoodieArchivedLogs which were written using HoodieLogFormat V1.

Throws:

IOException

getBytes

@Deprecated
public byte[] getBytes(org.apache.avro.Schema schema)
                            throws IOException

Deprecated.

Throws:

IOException