org.apache.hudi.keygen

Class AutoRecordGenWrapperAvroKeyGenerator

java.lang.Object

org.apache.hudi.keygen.KeyGenerator

org.apache.hudi.keygen.BaseKeyGenerator

org.apache.hudi.keygen.AutoRecordGenWrapperAvroKeyGenerator

All Implemented Interfaces:

Serializable, AutoRecordKeyGeneratorWrapper, KeyGeneratorInterface

public class AutoRecordGenWrapperAvroKeyGenerator
extends BaseKeyGenerator
implements AutoRecordKeyGeneratorWrapper

A wrapper key generator to intercept getRecordKey calls for auto record key generator.

1. Generated keys will be unique not only w/in provided [[org.apache.spark.sql.DataFrame]], but globally unique w/in the target table
2. Generated keys have minimal overhead (to compute, persist and read)

Keys adhere to the following format: [instantTime]_[PartitionId]_[RowId] where instantTime refers to the commit time of the batch being ingested. PartitionId refers to spark's partition Id. RowId refers to the row index within the spark partition.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.keygen.BaseKeyGenerator

consistentLogicalTimestampEnabled, CUSTOM_KEY_GENERATOR_SPLIT_REGEX, EMPTY_PARTITION, encodePartitionPath, FIELD_SEPARATOR, hiveStylePartitioning, partitionPathFields, recordKeyFields

Fields inherited from class org.apache.hudi.keygen.KeyGenerator

config

Constructor Summary

Constructors

Constructor and Description
AutoRecordGenWrapperAvroKeyGenerator(TypedProperties config, BaseKeyGenerator keyGenerator)

Method Summary

Modifier and Type	Method and Description
BaseKeyGenerator	getPartitionKeyGenerator()
String	getPartitionPath(org.apache.avro.generic.GenericRecord record)
List<String>	getPartitionPathFields()
String	getRecordKey(org.apache.avro.generic.GenericRecord record)
List<String>	getRecordKeyFieldNames()
boolean	isConsistentLogicalTimestampEnabled()

Methods inherited from class org.apache.hudi.keygen.BaseKeyGenerator

getKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AutoRecordGenWrapperAvroKeyGenerator

public AutoRecordGenWrapperAvroKeyGenerator(TypedProperties config,
                                            BaseKeyGenerator keyGenerator)

Method Detail

getRecordKey

public String getRecordKey(org.apache.avro.generic.GenericRecord record)

Specified by:

getRecordKey in class BaseKeyGenerator

getPartitionPath

public String getPartitionPath(org.apache.avro.generic.GenericRecord record)

Specified by:

getPartitionPath in class BaseKeyGenerator

getRecordKeyFieldNames

public List<String> getRecordKeyFieldNames()

Specified by:

getRecordKeyFieldNames in interface KeyGeneratorInterface

Overrides:

getRecordKeyFieldNames in class BaseKeyGenerator

getPartitionPathFields

public List<String> getPartitionPathFields()

Overrides:

getPartitionPathFields in class BaseKeyGenerator

isConsistentLogicalTimestampEnabled

public boolean isConsistentLogicalTimestampEnabled()

Overrides:

isConsistentLogicalTimestampEnabled in class BaseKeyGenerator

getPartitionKeyGenerator

public BaseKeyGenerator getPartitionKeyGenerator()

Specified by:

getPartitionKeyGenerator in interface AutoRecordKeyGeneratorWrapper