org.apache.hudi.metadata

Class HoodieMetadataPayload

java.lang.Object

org.apache.hudi.metadata.HoodieMetadataPayload

All Implemented Interfaces:

Serializable, HoodieRecordPayload<HoodieMetadataPayload>

public class HoodieMetadataPayload
extends Object
implements HoodieRecordPayload<HoodieMetadataPayload>

MetadataTable records are persisted with the schema defined in HoodieMetadata.avsc. This class represents the payload for the MetadataTable.

This single metadata payload is shared by all the partitions under the metadata table. The partition specific records are determined by the field "type" saved within the record. The following types are supported:

METADATA_TYPE_PARTITION_LIST (1): -- List of all partitions. There is a single such record -- key = @HoodieTableMetadata.RECORDKEY_PARTITION_LIST

METADATA_TYPE_FILE_LIST (2): -- List of all files in a partition. There is one such record for each partition -- key = partition name

METADATA_TYPE_COLUMN_STATS (3): -- This is an index for column stats in the table

METADATA_TYPE_BLOOM_FILTER (4): -- This is an index for base file bloom filters. This is a map of FileID to its BloomFilter byte[].

During compaction on the table, the deletions are merged with additions and hence records are pruned.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	COLUMN_STATS_FIELD_COLUMN_NAME
static String	COLUMN_STATS_FIELD_FILE_NAME
static String	COLUMN_STATS_FIELD_IS_DELETED
static String	COLUMN_STATS_FIELD_MAX_VALUE
static String	COLUMN_STATS_FIELD_MIN_VALUE HoodieMetadata column stats payload field ids
static String	COLUMN_STATS_FIELD_NULL_COUNT
static String	COLUMN_STATS_FIELD_TOTAL_SIZE
static String	COLUMN_STATS_FIELD_TOTAL_UNCOMPRESSED_SIZE
static String	COLUMN_STATS_FIELD_VALUE_COUNT
static String	KEY_FIELD_NAME HoodieMetadata schema field ids
static String	RECORD_INDEX_FIELD_FILE_INDEX
static String	RECORD_INDEX_FIELD_FILEID
static String	RECORD_INDEX_FIELD_FILEID_ENCODING
static int	RECORD_INDEX_FIELD_FILEID_ENCODING_RAW_STRING
static int	RECORD_INDEX_FIELD_FILEID_ENCODING_UUID
static String	RECORD_INDEX_FIELD_FILEID_HIGH_BITS
static String	RECORD_INDEX_FIELD_FILEID_LOW_BITS
static String	RECORD_INDEX_FIELD_INSTANT_TIME
static String	RECORD_INDEX_FIELD_PARTITION HoodieMetadata record index payload field ids
static String	RECORD_INDEX_FIELD_POSITION
static int	RECORD_INDEX_MISSING_FILEINDEX_FALLBACK FileIndex value saved in record index record when the fileId has no index (old format of base filename)
static String	SCHEMA_FIELD_ID_BLOOM_FILTER
static String	SCHEMA_FIELD_ID_COLUMN_STATS
static String	SCHEMA_FIELD_ID_RECORD_INDEX
static String	SCHEMA_FIELD_ID_SECONDARY_INDEX
static String	SCHEMA_FIELD_NAME_METADATA
static String	SCHEMA_FIELD_NAME_TYPE
static String	SECONDARY_INDEX_FIELD_IS_DELETED
static String	SECONDARY_INDEX_FIELD_RECORD_KEY HoodieMetadata secondary index payload field ids

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieMetadataPayload(org.apache.avro.generic.GenericRecord record, Comparable<?> orderingVal)
	HoodieMetadataPayload(Option<org.apache.avro.generic.GenericRecord> recordOpt)
protected	HoodieMetadataPayload(String key, int type, Map<String,HoodieMetadataFileInfo> filesystemMetadata, HoodieMetadataBloomFilter metadataBloomFilter, HoodieMetadataColumnStats columnStats, HoodieRecordIndexInfo recordIndexMetadata, HoodieSecondaryIndexInfo secondaryIndexMetadata)

Method Summary

Modifier and Type	Method and Description
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord oldRecord, org.apache.avro.Schema schema) This methods is deprecated.
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord oldRecord, org.apache.avro.Schema schema, Properties properties) This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object.
static Option<HoodieRecord<HoodieMetadataPayload>>	combineSecondaryIndexRecord(HoodieRecord<HoodieMetadataPayload> oldRecord, HoodieRecord<HoodieMetadataPayload> newRecord)
static HoodieRecord<HoodieMetadataPayload>	createBloomFilterMetadataRecord(String partitionName, String baseFileName, String timestamp, String bloomFilterType, ByteBuffer bloomFilter, boolean isDeleted) Create bloom filter metadata record.
static Stream<HoodieRecord>	createColumnStatsRecords(String partitionName, Collection<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList, boolean isDeleted)
static HoodieRecord<HoodieMetadataPayload>	createPartitionFilesRecord(String partition, Map<String,Long> filesAdded, List<String> filesDeleted) Create and return a HoodieMetadataPayload to save list of files within a partition.
static HoodieRecord<HoodieMetadataPayload>	createPartitionListRecord(List<String> partitions) Create and return a HoodieMetadataPayload to save list of partitions.
static HoodieRecord<HoodieMetadataPayload>	createPartitionListRecord(List<String> partitions, boolean isDeleted) Create and return a HoodieMetadataPayload to save list of partitions.
static Stream<HoodieRecord>	createPartitionStatsRecords(String partitionPath, Collection<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList, boolean isDeleted)
static HoodieRecord	createRecordIndexDelete(String recordKey) Create and return a HoodieMetadataPayload to delete a record in the Metadata Table's record index.
static HoodieRecord<HoodieMetadataPayload>	createRecordIndexUpdate(String recordKey, String partition, String fileId, String instantTime, int fileIdEncoding) Create and return a HoodieMetadataPayload to insert or update an entry for the record index.
static HoodieRecord<HoodieMetadataPayload>	createSecondaryIndex(String recordKey, String secondaryKey, String partitionPath, Boolean isDeleted) Create and return a HoodieMetadataPayload to insert or update an entry for the secondary index.
boolean	equals(Object other)
static String	getBloomFilterIndexKey(PartitionIndexID partitionIndexID, FileIndexID fileIndexID) Get bloom filter index key.
Option<HoodieMetadataBloomFilter>	getBloomFilterMetadata() Get the bloom filter metadata from this payload.
Option<HoodieMetadataColumnStats>	getColumnStatMetadata() Get the bloom filter metadata from this payload.
static String	getColumnStatsIndexKey(PartitionIndexID partitionIndexID, FileIndexID fileIndexID, ColumnIndexID columnIndexID) Get column stats index key.
static String	getColumnStatsIndexKey(String partitionName, HoodieColumnRangeMetadata<Comparable> columnRangeMetadata) Get column stats index key from the column range metadata.
List<String>	getDeletions() Returns the list of filenames deleted as part of this record.
List<StoragePathInfo>	getFileList(HoodieStorage storage, StoragePath partitionPath) Returns the files added as part of this record.
List<String>	getFilenames() Returns the list of filenames added as part of this record.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema) This method is deprecated.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schemaIgnored, Properties propertiesIgnored) Generates an avro record out of the given HoodieRecordPayload, to be written out to storage.
static String	getPartitionStatsIndexKey(String partitionPath, String columnName)
HoodieRecordGlobalLocation	getRecordGlobalLocation() If this is a record-level index entry, returns the file to which this is mapped.
String	getRecordKeyFromSecondaryIndex()
int	hashCode()
boolean	isDeleted()
boolean	isSecondaryIndexDeleted()
HoodieMetadataPayload	preCombine(HoodieMetadataPayload previousRecord) This method is deprecated.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.model.HoodieRecordPayload

getMetadata, getOrderingValue, preCombine, preCombine

Field Detail

KEY_FIELD_NAME

public static final String KEY_FIELD_NAME

HoodieMetadata schema field ids

See Also:

Constant Field Values

SCHEMA_FIELD_NAME_TYPE

public static final String SCHEMA_FIELD_NAME_TYPE

See Also:

Constant Field Values

SCHEMA_FIELD_NAME_METADATA

public static final String SCHEMA_FIELD_NAME_METADATA

See Also:

Constant Field Values

SCHEMA_FIELD_ID_COLUMN_STATS

public static final String SCHEMA_FIELD_ID_COLUMN_STATS

See Also:

Constant Field Values

SCHEMA_FIELD_ID_BLOOM_FILTER

public static final String SCHEMA_FIELD_ID_BLOOM_FILTER

See Also:

Constant Field Values

SCHEMA_FIELD_ID_RECORD_INDEX

public static final String SCHEMA_FIELD_ID_RECORD_INDEX

See Also:

Constant Field Values

SCHEMA_FIELD_ID_SECONDARY_INDEX

public static final String SCHEMA_FIELD_ID_SECONDARY_INDEX

See Also:

Constant Field Values

COLUMN_STATS_FIELD_MIN_VALUE

public static final String COLUMN_STATS_FIELD_MIN_VALUE

HoodieMetadata column stats payload field ids

See Also:

Constant Field Values

COLUMN_STATS_FIELD_MAX_VALUE

public static final String COLUMN_STATS_FIELD_MAX_VALUE

See Also:

Constant Field Values

COLUMN_STATS_FIELD_NULL_COUNT

public static final String COLUMN_STATS_FIELD_NULL_COUNT

See Also:

Constant Field Values

COLUMN_STATS_FIELD_VALUE_COUNT

public static final String COLUMN_STATS_FIELD_VALUE_COUNT

See Also:

Constant Field Values

COLUMN_STATS_FIELD_TOTAL_SIZE

public static final String COLUMN_STATS_FIELD_TOTAL_SIZE

See Also:

Constant Field Values

COLUMN_STATS_FIELD_FILE_NAME

public static final String COLUMN_STATS_FIELD_FILE_NAME

See Also:

Constant Field Values

COLUMN_STATS_FIELD_COLUMN_NAME

public static final String COLUMN_STATS_FIELD_COLUMN_NAME

See Also:

Constant Field Values

COLUMN_STATS_FIELD_TOTAL_UNCOMPRESSED_SIZE

public static final String COLUMN_STATS_FIELD_TOTAL_UNCOMPRESSED_SIZE

See Also:

Constant Field Values

COLUMN_STATS_FIELD_IS_DELETED

public static final String COLUMN_STATS_FIELD_IS_DELETED

See Also:

Constant Field Values

RECORD_INDEX_FIELD_PARTITION

public static final String RECORD_INDEX_FIELD_PARTITION

HoodieMetadata record index payload field ids

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID_HIGH_BITS

public static final String RECORD_INDEX_FIELD_FILEID_HIGH_BITS

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID_LOW_BITS

public static final String RECORD_INDEX_FIELD_FILEID_LOW_BITS

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILE_INDEX

public static final String RECORD_INDEX_FIELD_FILE_INDEX

See Also:

Constant Field Values

RECORD_INDEX_FIELD_INSTANT_TIME

public static final String RECORD_INDEX_FIELD_INSTANT_TIME

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID

public static final String RECORD_INDEX_FIELD_FILEID

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID_ENCODING

public static final String RECORD_INDEX_FIELD_FILEID_ENCODING

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID_ENCODING_UUID

public static final int RECORD_INDEX_FIELD_FILEID_ENCODING_UUID

See Also:

Constant Field Values

RECORD_INDEX_FIELD_FILEID_ENCODING_RAW_STRING

public static final int RECORD_INDEX_FIELD_FILEID_ENCODING_RAW_STRING

See Also:

Constant Field Values

RECORD_INDEX_FIELD_POSITION

public static final String RECORD_INDEX_FIELD_POSITION

See Also:

Constant Field Values

RECORD_INDEX_MISSING_FILEINDEX_FALLBACK

public static final int RECORD_INDEX_MISSING_FILEINDEX_FALLBACK

FileIndex value saved in record index record when the fileId has no index (old format of base filename)

See Also:

Constant Field Values

SECONDARY_INDEX_FIELD_RECORD_KEY

public static final String SECONDARY_INDEX_FIELD_RECORD_KEY

HoodieMetadata secondary index payload field ids

See Also:

Constant Field Values

SECONDARY_INDEX_FIELD_IS_DELETED

public static final String SECONDARY_INDEX_FIELD_IS_DELETED

See Also:

Constant Field Values

Constructor Detail

HoodieMetadataPayload

public HoodieMetadataPayload(@Nullable
                             org.apache.avro.generic.GenericRecord record,
                             Comparable<?> orderingVal)

HoodieMetadataPayload

public HoodieMetadataPayload(Option<org.apache.avro.generic.GenericRecord> recordOpt)

HoodieMetadataPayload

protected HoodieMetadataPayload(String key,
                                int type,
                                Map<String,HoodieMetadataFileInfo> filesystemMetadata,
                                HoodieMetadataBloomFilter metadataBloomFilter,
                                HoodieMetadataColumnStats columnStats,
                                HoodieRecordIndexInfo recordIndexMetadata,
                                HoodieSecondaryIndexInfo secondaryIndexMetadata)

Method Detail

createPartitionListRecord

public static HoodieRecord<HoodieMetadataPayload> createPartitionListRecord(List<String> partitions)

Create and return a HoodieMetadataPayload to save list of partitions.

Parameters:

partitions - The list of partitions

createPartitionListRecord

public static HoodieRecord<HoodieMetadataPayload> createPartitionListRecord(List<String> partitions,
                                                                            boolean isDeleted)

Create and return a HoodieMetadataPayload to save list of partitions.

Parameters:

partitions - The list of partitions

createPartitionFilesRecord

public static HoodieRecord<HoodieMetadataPayload> createPartitionFilesRecord(String partition,
                                                                             Map<String,Long> filesAdded,
                                                                             List<String> filesDeleted)

Create and return a HoodieMetadataPayload to save list of files within a partition.

Parameters:

partition - The name of the partition

filesAdded - Mapping of files to their sizes for files which have been added to this partition

filesDeleted - List of files which have been deleted from this partition

createBloomFilterMetadataRecord

public static HoodieRecord<HoodieMetadataPayload> createBloomFilterMetadataRecord(String partitionName,
                                                                                  String baseFileName,
                                                                                  String timestamp,
                                                                                  String bloomFilterType,
                                                                                  ByteBuffer bloomFilter,
                                                                                  boolean isDeleted)

Create bloom filter metadata record.

Parameters:

partitionName - - Partition name

baseFileName - - Base file name for which the bloom filter needs to persisted

timestamp - - Instant timestamp responsible for this record

bloomFilter - - Bloom filter for the File

isDeleted - - Is the bloom filter no more valid

Returns:

Metadata payload containing the fileID and its bloom filter record

preCombine

public HoodieMetadataPayload preCombine(HoodieMetadataPayload previousRecord)

Description copied from interface: HoodieRecordPayload

This method is deprecated. Please use this HoodieRecordPayload.preCombine(HoodieRecordPayload, Properties) method.

Specified by:

preCombine in interface HoodieRecordPayload<HoodieMetadataPayload>

combineSecondaryIndexRecord

public static Option<HoodieRecord<HoodieMetadataPayload>> combineSecondaryIndexRecord(HoodieRecord<HoodieMetadataPayload> oldRecord,
                                                                                      HoodieRecord<HoodieMetadataPayload> newRecord)

combineAndGetUpdateValue

public Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord oldRecord,
                                                                              org.apache.avro.Schema schema,
                                                                              Properties properties)
                                                                       throws IOException

Description copied from interface: HoodieRecordPayload

This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object. Implementations can leverage properties if required.

eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs, and merge them with current image for a database row on storage

Specified by:

combineAndGetUpdateValue in interface HoodieRecordPayload<HoodieMetadataPayload>

Parameters:

oldRecord - Current value in storage, to merge/combine this payload with

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

new combined/merged value to be written back to storage. EMPTY to skip writing this record.

Throws:

IOException

combineAndGetUpdateValue

public Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord oldRecord,
                                                                              org.apache.avro.Schema schema)
                                                                       throws IOException

Description copied from interface: HoodieRecordPayload

This methods is deprecated. Please refer to HoodieRecordPayload.combineAndGetUpdateValue(IndexedRecord, Schema, Properties) for java docs.

Specified by:

combineAndGetUpdateValue in interface HoodieRecordPayload<HoodieMetadataPayload>

Throws:

IOException

getInsertValue

public Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schemaIgnored,
                                                                    Properties propertiesIgnored)
                                                             throws IOException

Description copied from interface: HoodieRecordPayload

Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record. Implementations can leverage properties if required.

Specified by:

getInsertValue in interface HoodieRecordPayload<HoodieMetadataPayload>

Parameters:

schemaIgnored - Schema used for record

propertiesIgnored - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

getInsertValue

public Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema)
                                                             throws IOException

Description copied from interface: HoodieRecordPayload

This method is deprecated. Refer to HoodieRecordPayload.getInsertValue(Schema, Properties) for java docs.

Specified by:

getInsertValue in interface HoodieRecordPayload<HoodieMetadataPayload>

Parameters:

schema - Schema used for record

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

getFilenames

public List<String> getFilenames()

Returns the list of filenames added as part of this record.

getDeletions

public List<String> getDeletions()

Returns the list of filenames deleted as part of this record.

getBloomFilterMetadata

public Option<HoodieMetadataBloomFilter> getBloomFilterMetadata()

Get the bloom filter metadata from this payload.

getColumnStatMetadata

public Option<HoodieMetadataColumnStats> getColumnStatMetadata()

Get the bloom filter metadata from this payload.

getFileList

public List<StoragePathInfo> getFileList(HoodieStorage storage,
                                         StoragePath partitionPath)

Returns the files added as part of this record.

getBloomFilterIndexKey

public static String getBloomFilterIndexKey(PartitionIndexID partitionIndexID,
                                            FileIndexID fileIndexID)

Get bloom filter index key.

Parameters:

partitionIndexID - - Partition index id

fileIndexID - - File index id

Returns:

Bloom filter index key

getColumnStatsIndexKey

public static String getColumnStatsIndexKey(PartitionIndexID partitionIndexID,
                                            FileIndexID fileIndexID,
                                            ColumnIndexID columnIndexID)

Get column stats index key.

Parameters:

partitionIndexID - - Partition index id

fileIndexID - - File index id

columnIndexID - - Column index id

Returns:

Column stats index key

getColumnStatsIndexKey

public static String getColumnStatsIndexKey(String partitionName,
                                            HoodieColumnRangeMetadata<Comparable> columnRangeMetadata)

Get column stats index key from the column range metadata.

Parameters:

partitionName - - Partition name

columnRangeMetadata - - Column range metadata

Returns:

Column stats index key

createColumnStatsRecords

public static Stream<HoodieRecord> createColumnStatsRecords(String partitionName,
                                                            Collection<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
                                                            boolean isDeleted)

createPartitionStatsRecords

public static Stream<HoodieRecord> createPartitionStatsRecords(String partitionPath,
                                                               Collection<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
                                                               boolean isDeleted)

getPartitionStatsIndexKey

public static String getPartitionStatsIndexKey(String partitionPath,
                                               String columnName)

createRecordIndexUpdate

public static HoodieRecord<HoodieMetadataPayload> createRecordIndexUpdate(String recordKey,
                                                                          String partition,
                                                                          String fileId,
                                                                          String instantTime,
                                                                          int fileIdEncoding)

Create and return a HoodieMetadataPayload to insert or update an entry for the record index.

Each entry maps the key of a single record in HUDI to its location.

Parameters:

recordKey - Key of the record

partition - Name of the partition which contains the record

fileId - fileId which contains the record

instantTime - instantTime when the record was added

createSecondaryIndex

public static HoodieRecord<HoodieMetadataPayload> createSecondaryIndex(String recordKey,
                                                                       String secondaryKey,
                                                                       String partitionPath,
                                                                       Boolean isDeleted)

Create and return a HoodieMetadataPayload to insert or update an entry for the secondary index.

Each entry maps the secondary key of a single record in HUDI to its record (or primary) key

Parameters:

recordKey - Primary key of the record

secondaryKey - Secondary key of the record

isDeleted - true if this record is deleted

getRecordKeyFromSecondaryIndex

public String getRecordKeyFromSecondaryIndex()

isSecondaryIndexDeleted

public boolean isSecondaryIndexDeleted()

createRecordIndexDelete

public static HoodieRecord createRecordIndexDelete(String recordKey)

Create and return a HoodieMetadataPayload to delete a record in the Metadata Table's record index.

Parameters:

recordKey - Key of the record to be deleted

getRecordGlobalLocation

public HoodieRecordGlobalLocation getRecordGlobalLocation()

If this is a record-level index entry, returns the file to which this is mapped.

isDeleted

public boolean isDeleted()

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object