org.apache.hudi.common.model

Interface HoodieRecordPayload<T extends HoodieRecordPayload>

All Superinterfaces:

Serializable

All Known Implementing Classes:

AbstractDebeziumAvroPayload, AWSDmsAvroPayload, DefaultHoodieRecordPayload, EmptyHoodieRecordPayload, EventTimeAvroPayload, FirstValueAvroPayload, HoodieAvroPayload, HoodieJsonPayload, HoodieMetadataPayload, MySqlDebeziumAvroPayload, OverwriteNonDefaultsWithLatestAvroPayload, OverwriteWithLatestAvroPayload, PartialUpdateAvroPayload, PostgresDebeziumAvroPayload, RewriteAvroPayload

@PublicAPIClass(maturity=STABLE)
public interface HoodieRecordPayload<T extends HoodieRecordPayload>
extends Serializable

Every Hoodie table has an implementation of the HoodieRecordPayload This abstracts out callbacks which depend on record specific logic.

Method Summary

Modifier and Type	Method and Description
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.Schema schema) Deprecated.
default Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.Schema schema, Properties properties) This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema) Deprecated.
default Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema, Properties properties) Generates an avro record out of the given HoodieRecordPayload, to be written out to storage.
default Option<Map<String,String>>	getMetadata() This method can be used to extract some metadata from HoodieRecordPayload.
default Comparable<?>	getOrderingValue() This method can be used to extract the ordering value of the payload for combining/merging, or 0 if no value is specified which means natural order(arrival time is used).
T	preCombine(T oldValue) Deprecated.
default T	preCombine(T oldValue, Properties properties) When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map.
default T	preCombine(T oldValue, org.apache.avro.Schema schema, Properties properties) When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a schema.

Method Detail

preCombine

@Deprecated
 @PublicAPIMethod(maturity=DEPRECATED)
T preCombine(T oldValue)

Deprecated.

This method is deprecated. Please use this preCombine(HoodieRecordPayload, Properties) method.

preCombine

@PublicAPIMethod(maturity=STABLE)
default T preCombine(T oldValue,
                                                       Properties properties)

When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a property map. Implementation can leverage the property to decide their business logic to do preCombine.

Parameters:

oldValue - instance of the old HoodieRecordPayload to be combined with.

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the combined value

preCombine

@PublicAPIMethod(maturity=EVOLVING)
default T preCombine(T oldValue,
                                                         org.apache.avro.Schema schema,
                                                         Properties properties)

When more than one HoodieRecord have the same HoodieKey in the incoming batch, this function combines them before attempting to insert/upsert by taking in a schema. Implementation can leverage the schema to decide their business logic to do preCombine.

Parameters:

oldValue - instance of the old HoodieRecordPayload to be combined with.

schema - Payload related schema. For example use schema to overwrite old instance for specified fields that doesn't equal to default value.

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the combined value

combineAndGetUpdateValue

@Deprecated
 @PublicAPIMethod(maturity=DEPRECATED)
Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue,
                                                                                                                          org.apache.avro.Schema schema)
                                                                                                                   throws IOException

Deprecated.

This methods is deprecated. Please refer to combineAndGetUpdateValue(IndexedRecord, Schema, Properties) for java docs.

Throws:

IOException

combineAndGetUpdateValue

default Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue,
                                                                               org.apache.avro.Schema schema,
                                                                               Properties properties)
                                                                        throws IOException

This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object. Implementations can leverage properties if required.

eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs, and merge them with current image for a database row on storage

Parameters:

currentValue - Current value in storage, to merge/combine this payload with

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

new combined/merged value to be written back to storage. EMPTY to skip writing this record.

Throws:

IOException

getInsertValue

@Deprecated
 @PublicAPIMethod(maturity=DEPRECATED)
Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema)
                                                                                                         throws IOException

Deprecated.

This method is deprecated. Refer to getInsertValue(Schema, Properties) for java docs.

Parameters:

schema - Schema used for record

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

getInsertValue

@PublicAPIMethod(maturity=STABLE)
default Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema,
                                                                                                       Properties properties)
                                                                                                throws IOException

Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record. Implementations can leverage properties if required.

Parameters:

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

getMetadata

@PublicAPIMethod(maturity=STABLE)
default Option<Map<String,String>> getMetadata()

This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to WriteStatus.markSuccess() and WriteStatus.markFailure() in order to compute some aggregate metrics using the metadata in the context of a write success or failure.

Returns:

the metadata in the form of Map if any.

getOrderingValue

@PublicAPIMethod(maturity=STABLE)
default Comparable<?> getOrderingValue()

This method can be used to extract the ordering value of the payload for combining/merging, or 0 if no value is specified which means natural order(arrival time is used).

Returns:

the ordering value