org.apache.hudi.common.util

Class SpillableMapUtils

java.lang.Object

org.apache.hudi.common.util.SpillableMapUtils

public class SpillableMapUtils
extends Object

A utility class supports spillable map.

Constructor Summary

Constructors

Constructor and Description
SpillableMapUtils()

Method Summary

Modifier and Type	Method and Description
static <R> long	computePayloadSize(R value, SizeEstimator<R> valueSizeEstimator) Compute a bytes representation of the payload by serializing the contents This is used to estimate the size of the payload (either in memory or when written to disk).
static <R> HoodieRecord<R>	convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord rec, String payloadClazz, String preCombineField, boolean withOperationField) Utility method to convert bytes to HoodieRecord using schema and payload class.
static <R> HoodieRecord<R>	convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record, String payloadClazz, String preCombineField, boolean withOperationField, Option<String> partitionName, Option<org.apache.avro.Schema> schemaWithoutMetaFields)
static <R> HoodieRecord<R>	convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record, String payloadClazz, String preCombineField, Pair<String,String> recordKeyPartitionPathFieldPair, boolean withOperationField, Option<String> partitionName, Option<org.apache.avro.Schema> schemaWithoutMetaFields) Utility method to convert bytes to HoodieRecord using schema and payload class.
static <R> R	generateEmptyPayload(String recKey, String partitionPath, Comparable orderingVal, String payloadClazz) Utility method to convert bytes to HoodieRecord using schema and payload class.
static byte[]	readBytesFromDisk(RandomAccessFile file, long valuePosition, int valueLength) Using the schema and payload class, read and convert the bytes on disk to a HoodieRecord.
static long	spillToDisk(SizeAwareDataOutputStream outputStream, BitCaskDiskMap.FileEntry fileEntry) Write Value and other metadata necessary to disk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SpillableMapUtils

public SpillableMapUtils()

Method Detail

readBytesFromDisk

public static byte[] readBytesFromDisk(RandomAccessFile file,
                                       long valuePosition,
                                       int valueLength)
                                throws IOException

Using the schema and payload class, read and convert the bytes on disk to a HoodieRecord.

Throws:

IOException

spillToDisk

public static long spillToDisk(SizeAwareDataOutputStream outputStream,
                               BitCaskDiskMap.FileEntry fileEntry)
                        throws IOException

Write Value and other metadata necessary to disk. Each entry has the following sequence of data

|crc|timestamp|sizeOfKey|SizeOfValue|key|value|

Throws:

IOException

computePayloadSize

public static <R> long computePayloadSize(R value,
                                          SizeEstimator<R> valueSizeEstimator)
                                   throws IOException

Compute a bytes representation of the payload by serializing the contents This is used to estimate the size of the payload (either in memory or when written to disk).

Throws:

IOException

convertToHoodieRecordPayload

public static <R> HoodieRecord<R> convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord rec,
                                                               String payloadClazz,
                                                               String preCombineField,
                                                               boolean withOperationField)

Utility method to convert bytes to HoodieRecord using schema and payload class.

convertToHoodieRecordPayload

public static <R> HoodieRecord<R> convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record,
                                                               String payloadClazz,
                                                               String preCombineField,
                                                               boolean withOperationField,
                                                               Option<String> partitionName,
                                                               Option<org.apache.avro.Schema> schemaWithoutMetaFields)

convertToHoodieRecordPayload

public static <R> HoodieRecord<R> convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record,
                                                               String payloadClazz,
                                                               String preCombineField,
                                                               Pair<String,String> recordKeyPartitionPathFieldPair,
                                                               boolean withOperationField,
                                                               Option<String> partitionName,
                                                               Option<org.apache.avro.Schema> schemaWithoutMetaFields)

Utility method to convert bytes to HoodieRecord using schema and payload class.

generateEmptyPayload

public static <R> R generateEmptyPayload(String recKey,
                                         String partitionPath,
                                         Comparable orderingVal,
                                         String payloadClazz)

Utility method to convert bytes to HoodieRecord using schema and payload class.