org.apache.hudi.common.table

Class TableSchemaResolver

java.lang.Object

org.apache.hudi.common.table.TableSchemaResolver

@ThreadSafe
public class TableSchemaResolver
extends Object

Helper class to read schema from data files and log files and to convert it between different formats.

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieTableMetaClient	metaClient

Constructor Summary

Constructors

Constructor and Description
TableSchemaResolver(HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	appendPartitionColumns(org.apache.avro.Schema dataSchema, Option<String[]> partitionFields)
org.apache.avro.Schema	getTableAvroSchema() Gets full schema (user + metadata) for a hoodie table in Avro format.
org.apache.avro.Schema	getTableAvroSchema(boolean includeMetadataFields) Gets schema for a hoodie table in Avro format, can choice if include metadata fields.
org.apache.avro.Schema	getTableAvroSchema(HoodieInstant instant, boolean includeMetadataFields) Fetches tables schema in Avro format as of the given instant
org.apache.avro.Schema	getTableAvroSchema(String timestamp) Fetches tables schema in Avro format as of the given instant
org.apache.avro.Schema	getTableAvroSchemaFromDataFile()
Option<org.apache.avro.Schema>	getTableAvroSchemaFromLatestCommit(boolean includeMetadataFields) Returns table's latest Avro Schema iff table is non-empty (ie there's at least a single commit) This method differs from getTableAvroSchema(boolean) in that it won't fallback to use table's schema used at creation
Option<org.apache.avro.Schema>	getTableAvroSchemaIfPresent(boolean includeMetadataFields)
org.apache.avro.Schema	getTableAvroSchemaWithoutMetadataFields() Deprecated. use getTableAvroSchema(boolean) instead
Option<String>	getTableHistorySchemaStrFromCommitMetadata() Gets the history schemas as String for a hoodie table from the HoodieCommitMetadata of the instant.
Option<InternalSchema>	getTableInternalSchemaFromCommitMetadata() Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
Option<InternalSchema>	getTableInternalSchemaFromCommitMetadata(String timestamp) Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
boolean	hasOperationField() NOTE: This method could only be used in tests
org.apache.avro.Schema	readSchemaFromLastCompaction(Option<HoodieInstant> lastCompactionCommitOpt) Deprecated. please use getTableAvroSchema(HoodieInstant, boolean) instead
static org.apache.avro.Schema	readSchemaFromLogFile(HoodieStorage storage, StoragePath path) Read the schema from the log file on path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metaClient

protected final HoodieTableMetaClient metaClient

Constructor Detail

TableSchemaResolver

public TableSchemaResolver(HoodieTableMetaClient metaClient)

Method Detail

getTableAvroSchemaFromDataFile

public org.apache.avro.Schema getTableAvroSchemaFromDataFile()
                                                      throws Exception

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema()
                                          throws Exception

Gets full schema (user + metadata) for a hoodie table in Avro format.

Returns:

Avro schema for this table

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(boolean includeMetadataFields)
                                          throws Exception

Gets schema for a hoodie table in Avro format, can choice if include metadata fields.

Parameters:

includeMetadataFields - choice if include metadata fields

Returns:

Avro schema for this table

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(String timestamp)
                                          throws Exception

Fetches tables schema in Avro format as of the given instant

Parameters:

timestamp - as of which table's schema will be fetched

Throws:

Exception

getTableAvroSchema

public org.apache.avro.Schema getTableAvroSchema(HoodieInstant instant,
                                                 boolean includeMetadataFields)
                                          throws Exception

Fetches tables schema in Avro format as of the given instant

Parameters:

instant - as of which table's schema will be fetched

Throws:

Exception

getTableAvroSchemaWithoutMetadataFields

@Deprecated
public org.apache.avro.Schema getTableAvroSchemaWithoutMetadataFields()
                                                                           throws Exception

Deprecated. use getTableAvroSchema(boolean) instead

Gets users data schema for a hoodie table in Avro format.

Returns:

Avro user data schema

Throws:

Exception

getTableAvroSchemaIfPresent

public Option<org.apache.avro.Schema> getTableAvroSchemaIfPresent(boolean includeMetadataFields)

getTableAvroSchemaFromLatestCommit

public Option<org.apache.avro.Schema> getTableAvroSchemaFromLatestCommit(boolean includeMetadataFields)
                                                                  throws Exception

Returns table's latest Avro Schema iff table is non-empty (ie there's at least a single commit) This method differs from getTableAvroSchema(boolean) in that it won't fallback to use table's schema used at creation

Throws:

Exception

readSchemaFromLastCompaction

public org.apache.avro.Schema readSchemaFromLastCompaction(Option<HoodieInstant> lastCompactionCommitOpt)
                                                    throws Exception

Deprecated. please use getTableAvroSchema(HoodieInstant, boolean) instead

Read schema from a data file from the last compaction commit done.

Throws:

Exception

readSchemaFromLogFile

public static org.apache.avro.Schema readSchemaFromLogFile(HoodieStorage storage,
                                                           StoragePath path)
                                                    throws IOException

Read the schema from the log file on path.

Returns:

Throws:

IOException

getTableInternalSchemaFromCommitMetadata

public Option<InternalSchema> getTableInternalSchemaFromCommitMetadata()

Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

InternalSchema for this table

getTableInternalSchemaFromCommitMetadata

public Option<InternalSchema> getTableInternalSchemaFromCommitMetadata(String timestamp)

Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

InternalSchema for this table

getTableHistorySchemaStrFromCommitMetadata

public Option<String> getTableHistorySchemaStrFromCommitMetadata()

Gets the history schemas as String for a hoodie table from the HoodieCommitMetadata of the instant.

Returns:

history schemas string for this table

hasOperationField

public boolean hasOperationField()

NOTE: This method could only be used in tests

appendPartitionColumns

public static org.apache.avro.Schema appendPartitionColumns(org.apache.avro.Schema dataSchema,
                                                            Option<String[]> partitionFields)