org.apache.hudi.internal.schema.utils

Class AvroSchemaEvolutionUtils

java.lang.Object

org.apache.hudi.internal.schema.utils.AvroSchemaEvolutionUtils

public class AvroSchemaEvolutionUtils
extends Object

Utility methods to support evolve old avro schema based on a given schema.

Constructor Summary

Constructors

Constructor and Description
AvroSchemaEvolutionUtils()

Method Summary

Modifier and Type	Method and Description
static InternalSchema	reconcileSchema(org.apache.avro.Schema incomingSchema, InternalSchema oldTableSchema, boolean makeMissingFieldsNullable) Support reconcile from a new avroSchema.
static org.apache.avro.Schema	reconcileSchema(org.apache.avro.Schema incomingSchema, org.apache.avro.Schema oldTableSchema, boolean makeMissingFieldsNullable)
static org.apache.avro.Schema	reconcileSchemaRequirements(org.apache.avro.Schema sourceSchema, org.apache.avro.Schema targetSchema, boolean shouldReorderColumns) Reconciles nullability and datatype requirements b/w source and target schemas, by adjusting these of the source schema to be in-line with the ones of the target one.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroSchemaEvolutionUtils

public AvroSchemaEvolutionUtils()

Method Detail

reconcileSchema

public static InternalSchema reconcileSchema(org.apache.avro.Schema incomingSchema,
                                             InternalSchema oldTableSchema,
                                             boolean makeMissingFieldsNullable)

Support reconcile from a new avroSchema. 1) incoming data has missing columns that were already defined in the table –> null values will be injected into missing columns 2) incoming data contains new columns not defined yet in the table -> columns will be added to the table schema (incoming dataframe?) 3) incoming data has missing columns that are already defined in the table and new columns not yet defined in the table -> new columns will be added to the table schema, missing columns will be injected with null values 4) support type change 5) support nested schema change. Notice: the incoming schema should not have delete/rename semantics. for example: incoming schema: int a, int b, int d; oldTableSchema int a, int b, int c, int d we must guarantee the column c is missing semantic, instead of delete semantic.

Parameters:

incomingSchema - implicitly evolution of avro when hoodie write operation

oldTableSchema - old internalSchema

makeMissingFieldsNullable - if true, fields missing from the incoming schema when compared to the oldTableSchema will become nullable in the result. Otherwise, no updates will be made to those fields.

Returns:

reconcile Schema

reconcileSchema

public static org.apache.avro.Schema reconcileSchema(org.apache.avro.Schema incomingSchema,
                                                     org.apache.avro.Schema oldTableSchema,
                                                     boolean makeMissingFieldsNullable)

reconcileSchemaRequirements

public static org.apache.avro.Schema reconcileSchemaRequirements(org.apache.avro.Schema sourceSchema,
                                                                 org.apache.avro.Schema targetSchema,
                                                                 boolean shouldReorderColumns)

Reconciles nullability and datatype requirements b/w source and target schemas, by adjusting these of the source schema to be in-line with the ones of the target one. Source is considered to be new incoming schema, while target could refer to prev table schema. For example, if colA in source is non-nullable, but is nullable in target, output schema will have colA as nullable. if "hoodie.datasource.write.new.columns.nullable" is set to true and if colB is not present in source, but is present in target, output schema will have colB as nullable. if colC has different data type in source schema compared to target schema and if its promotable, (say source is int, and target is long and since int can be promoted to long), colC will be long data type in output schema.

Parameters:

sourceSchema - source schema that needs reconciliation

targetSchema - target schema that source schema will be reconciled against

Returns:

schema (based off source one) that has nullability constraints and datatypes reconciled