org.apache.hudi.keygen

Class CustomAvroKeyGenerator

java.lang.Object

org.apache.hudi.keygen.KeyGenerator

org.apache.hudi.keygen.BaseKeyGenerator

org.apache.hudi.keygen.CustomAvroKeyGenerator

All Implemented Interfaces:

Serializable, KeyGeneratorInterface

public class CustomAvroKeyGenerator
extends BaseKeyGenerator

This is a generic implementation of KeyGenerator where users can configure record key as a single field or a combination of fields. Similarly partition path can be configured to have multiple fields or only one field. This class expects value for prop "hoodie.datasource.write.partitionpath.field" in a specific format. For example:

properties.put("hoodie.datasource.write.partitionpath.field", "field1:PartitionKeyType1,field2:PartitionKeyType2").

The complete partition path is created as / and so on.

Few points to consider: 1. If you want to customize some partition path field on a timestamp basis, you can use field1:timestampBased 2. If you simply want to have the value of your configured field in the partition path, use field1:simple 3. If you want your table to be non partitioned, simply leave it as blank.

RecordKey is internally generated using either SimpleKeyGenerator or ComplexKeyGenerator.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CustomAvroKeyGenerator.PartitionKeyType Used as a part of config in CustomKeyGenerator.java.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SPLIT_REGEX

Fields inherited from class org.apache.hudi.keygen.BaseKeyGenerator

consistentLogicalTimestampEnabled, encodePartitionPath, hiveStylePartitioning, partitionPathFields, recordKeyFields

Fields inherited from class org.apache.hudi.keygen.KeyGenerator

config

Constructor Summary

Constructors

Constructor and Description
CustomAvroKeyGenerator(TypedProperties props)

Method Summary

Modifier and Type	Method and Description
String	getDefaultPartitionPathSeparator()
String	getPartitionPath(org.apache.avro.generic.GenericRecord record)
String	getRecordKey(org.apache.avro.generic.GenericRecord record)

Methods inherited from class org.apache.hudi.keygen.BaseKeyGenerator

getKey, getPartitionPathFields, getRecordKeyFieldNames, getRecordKeyFields, isConsistentLogicalTimestampEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPLIT_REGEX

public static final String SPLIT_REGEX

See Also:

Constant Field Values

Constructor Detail

CustomAvroKeyGenerator

public CustomAvroKeyGenerator(TypedProperties props)

Method Detail

getPartitionPath

public String getPartitionPath(org.apache.avro.generic.GenericRecord record)

Specified by:

getPartitionPath in class BaseKeyGenerator

getRecordKey

public String getRecordKey(org.apache.avro.generic.GenericRecord record)

Specified by:

getRecordKey in class BaseKeyGenerator

getDefaultPartitionPathSeparator

public String getDefaultPartitionPathSeparator()