org.apache.hudi.keygen

Class CustomAvroKeyGenerator

java.lang.Object

org.apache.hudi.keygen.KeyGenerator

org.apache.hudi.keygen.BaseKeyGenerator

org.apache.hudi.keygen.CustomAvroKeyGenerator

All Implemented Interfaces:

Serializable, KeyGeneratorInterface

public class CustomAvroKeyGenerator
extends BaseKeyGenerator

This is a generic implementation of KeyGenerator where users can configure record key as a single field or a combination of fields. Similarly partition path can be configured to have multiple fields or only one field. This class expects value for prop "hoodie.datasource.write.partitionpath.field" in a specific format. For example:

properties.put("hoodie.datasource.write.partitionpath.field", "field1:PartitionKeyType1,field2:PartitionKeyType2").

The complete partition path is created as / and so on.

Few points to consider: 1. If you want to customize some partition path field on a timestamp basis, you can use field1:timestampBased 2. If you simply want to have the value of your configured field in the partition path, use field1:simple 3. If you want your table to be non partitioned, simply leave it as blank.

RecordKey is internally generated using either SimpleKeyGenerator or ComplexKeyGenerator.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CustomAvroKeyGenerator.PartitionKeyType Used as a part of config in CustomKeyGenerator.java.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_PARTITION_PATH_SEPARATOR

Fields inherited from class org.apache.hudi.keygen.BaseKeyGenerator

consistentLogicalTimestampEnabled, CUSTOM_KEY_GENERATOR_SPLIT_REGEX, EMPTY_PARTITION, encodePartitionPath, FIELD_SEPARATOR, hiveStylePartitioning, partitionPathFields, recordKeyFields

Fields inherited from class org.apache.hudi.keygen.KeyGenerator

config

Constructor Summary

Constructors

Constructor and Description
CustomAvroKeyGenerator(TypedProperties props)

Method Summary

Modifier and Type	Method and Description
String	getDefaultPartitionPathSeparator()
static Pair<String,Option<CustomAvroKeyGenerator.PartitionKeyType>>	getPartitionFieldAndKeyType(String field)
String	getPartitionPath(org.apache.avro.generic.GenericRecord record)
static List<CustomAvroKeyGenerator.PartitionKeyType>	getPartitionTypes(HoodieTableConfig tableConfig) Returns list of partition types configured in the partition fields for custom key generator.
String	getRecordKey(org.apache.avro.generic.GenericRecord record)
static Option<List<String>>	getTimestampFields(HoodieTableConfig tableConfig) Returns the partition fields with timestamp partition type.

Methods inherited from class org.apache.hudi.keygen.BaseKeyGenerator

getKey, getPartitionPathFields, getRecordKeyFieldNames, isConsistentLogicalTimestampEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PARTITION_PATH_SEPARATOR

public static final String DEFAULT_PARTITION_PATH_SEPARATOR

See Also:

Constant Field Values

Constructor Detail

CustomAvroKeyGenerator

public CustomAvroKeyGenerator(TypedProperties props)

Method Detail

getPartitionTypes

public static List<CustomAvroKeyGenerator.PartitionKeyType> getPartitionTypes(HoodieTableConfig tableConfig)

Returns list of partition types configured in the partition fields for custom key generator.

Parameters:

tableConfig - Table config where partition fields are configured

getTimestampFields

public static Option<List<String>> getTimestampFields(HoodieTableConfig tableConfig)

Returns the partition fields with timestamp partition type.

Parameters:

tableConfig - Table config where partition fields are configured

Returns:

Optional list of partition fields with timestamp partition type

getPartitionFieldAndKeyType

public static Pair<String,Option<CustomAvroKeyGenerator.PartitionKeyType>> getPartitionFieldAndKeyType(String field)

getPartitionPath

public String getPartitionPath(org.apache.avro.generic.GenericRecord record)

Specified by:

getPartitionPath in class BaseKeyGenerator

getRecordKey

public String getRecordKey(org.apache.avro.generic.GenericRecord record)

Specified by:

getRecordKey in class BaseKeyGenerator

getDefaultPartitionPathSeparator

public String getDefaultPartitionPathSeparator()