org.apache.hudi.common.table.log.block

Class HoodieLogBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

Direct Known Subclasses:

HoodieCommandBlock, HoodieCorruptBlock, HoodieDataBlock, HoodieDeleteBlock

public abstract class HoodieLogBlock
extends Object

Abstract class defining a block in HoodieLogFile.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieLogBlock.FooterMetadataType Log Metadata footers abstraction for a HoodieLogBlock WARNING : This enum is serialized as the ordinal.
static class	HoodieLogBlock.HeaderMetadataType Log Metadata headers abstraction for a HoodieLogBlock WARNING : This enum is serialized as the ordinal.
static class	HoodieLogBlock.HoodieLogBlockContentLocation This class is used to store the Location of the Content of a Log Block.
static class	HoodieLogBlock.HoodieLogBlockType Type of the log block WARNING: This enum is serialized as the ordinal.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	readBlockLazily
static int	version The current version of the log block.

Constructor Summary

Constructors

Constructor and Description
HoodieLogBlock(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader, Map<HoodieLogBlock.HeaderMetadataType,String> logBlockFooter, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<byte[]> content, Supplier<SeekableDataInputStream> inputStreamSupplier, boolean readBlockLazily)

Method Summary

Modifier and Type	Method and Description
protected void	addRecordPositionsToHeader(Set<Long> positionSet, int numRecords)
protected void	deflate() After the content bytes is converted into the required DataStructure by a logBlock, deflate the content to release byte [] and relieve memory pressure when GC kicks in.
Option<HoodieLogBlock.HoodieLogBlockContentLocation>	getBlockContentLocation()
abstract HoodieLogBlock.HoodieLogBlockType	getBlockType()
Option<byte[]>	getContent()
byte[]	getContentBytes(HoodieStorage storage)
Map<HoodieLogBlock.HeaderMetadataType,String>	getLogBlockFooter()
Map<HoodieLogBlock.HeaderMetadataType,String>	getLogBlockHeader()
long	getLogBlockLength()
static Map<HoodieLogBlock.HeaderMetadataType,String>	getLogMetadata(SeekableDataInputStream dis) Convert bytes to LogMetadata, follow the same order as getLogMetadataBytes(java.util.Map<org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType, java.lang.String>).
static byte[]	getLogMetadataBytes(Map<HoodieLogBlock.HeaderMetadataType,String> metadata) Convert log metadata to bytes 1.
byte[]	getMagic()
org.roaringbitmap.longlong.Roaring64NavigableMap	getRecordPositions()
protected void	inflate() When lazyReading of blocks is turned on, inflate the content of a log block from disk.
boolean	isCompactedLogBlock() Compacted blocks are created using log compaction which basically merges the consecutive blocks together and create huge block with all the changes.
static Option<byte[]>	tryReadContent(SeekableDataInputStream inputStream, Integer contentLength, boolean readLazily) Read or Skip block content of a log block in the log file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

version

public static int version

The current version of the log block. Anytime the logBlock format changes this version needs to be bumped and corresponding changes need to be made to HoodieLogBlockVersion TODO : Change this to a class, something like HoodieLogBlockVersionV1/V2 and implement/override operations there Current log block version is V3.

readBlockLazily

protected boolean readBlockLazily

Constructor Detail

HoodieLogBlock

public HoodieLogBlock(@Nonnull
                      Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader,
                      @Nonnull
                      Map<HoodieLogBlock.HeaderMetadataType,String> logBlockFooter,
                      @Nonnull
                      Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                      @Nonnull
                      Option<byte[]> content,
                      @Nullable
                      Supplier<SeekableDataInputStream> inputStreamSupplier,
                      boolean readBlockLazily)

Method Detail

getContentBytes

public byte[] getContentBytes(HoodieStorage storage)
                       throws IOException

Throws:

IOException

getMagic

public byte[] getMagic()

getBlockType

public abstract HoodieLogBlock.HoodieLogBlockType getBlockType()

getLogBlockLength

public long getLogBlockLength()

getBlockContentLocation

public Option<HoodieLogBlock.HoodieLogBlockContentLocation> getBlockContentLocation()

getLogBlockHeader

public Map<HoodieLogBlock.HeaderMetadataType,String> getLogBlockHeader()

getLogBlockFooter

public Map<HoodieLogBlock.HeaderMetadataType,String> getLogBlockFooter()

getContent

public Option<byte[]> getContent()

isCompactedLogBlock

public boolean isCompactedLogBlock()

Compacted blocks are created using log compaction which basically merges the consecutive blocks together and create huge block with all the changes.

getRecordPositions

public org.roaringbitmap.longlong.Roaring64NavigableMap getRecordPositions()
                                                                    throws IOException

Returns:

A Roaring64NavigableMap bitmap containing the record positions in long type if the HoodieLogBlock.HeaderMetadataType.RECORD_POSITIONS block header exists; otherwise, an empty Roaring64NavigableMap bitmap.

Throws:

IOException - upon I/O error.

addRecordPositionsToHeader

protected void addRecordPositionsToHeader(Set<Long> positionSet,
                                          int numRecords)

getLogMetadataBytes

public static byte[] getLogMetadataBytes(Map<HoodieLogBlock.HeaderMetadataType,String> metadata)
                                  throws IOException

Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes

Throws:

IOException

getLogMetadata

public static Map<HoodieLogBlock.HeaderMetadataType,String> getLogMetadata(SeekableDataInputStream dis)
                                                                    throws IOException

Convert bytes to LogMetadata, follow the same order as getLogMetadataBytes(java.util.Map<org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType, java.lang.String>).

Throws:

IOException

tryReadContent

public static Option<byte[]> tryReadContent(SeekableDataInputStream inputStream,
                                            Integer contentLength,
                                            boolean readLazily)
                                     throws IOException

Read or Skip block content of a log block in the log file. Depends on lazy reading enabled in HoodieMergedLogRecordScanner

Throws:

IOException

inflate

protected void inflate()
                throws HoodieIOException

When lazyReading of blocks is turned on, inflate the content of a log block from disk.

Throws:

HoodieIOException

deflate

protected void deflate()

After the content bytes is converted into the required DataStructure by a logBlock, deflate the content to release byte [] and relieve memory pressure when GC kicks in. NOTE: This still leaves the heap fragmented