org.apache.hudi.gcp.bigquery

Class HoodieBigQuerySyncClient

java.lang.Object

org.apache.hudi.sync.common.HoodieSyncClient

org.apache.hudi.gcp.bigquery.HoodieBigQuerySyncClient

All Implemented Interfaces:

AutoCloseable, HoodieMetaSyncOperations

public class HoodieBigQuerySyncClient
extends HoodieSyncClient

Field Summary

Fields

Modifier and Type	Field and Description
protected BigQuerySyncConfig	config

Fields inherited from class org.apache.hudi.sync.common.HoodieSyncClient

metaClient, partitionValueExtractor

Fields inherited from interface org.apache.hudi.sync.common.HoodieMetaSyncOperations

HOODIE_LAST_COMMIT_COMPLETION_TIME_SYNC, HOODIE_LAST_COMMIT_TIME_SYNC

Constructor Summary

Constructors

Constructor and Description
HoodieBigQuerySyncClient(BigQuerySyncConfig config, HoodieTableMetaClient metaClient)

Method Summary

Modifier and Type	Method and Description
void	close()
void	createManifestTable(String tableName, String sourceUri)
void	createOrUpdateTableUsingBqManifestFile(String tableName, String bqManifestFileUri, String sourceUriPrefix, com.google.cloud.bigquery.Schema schema)
void	createSnapshotView(String viewName, String versionsTableName, String manifestTableName)
void	createVersionsTable(String tableName, String sourceUri, String sourceUriPrefix, List<String> partitionFields)
boolean	datasetExists()
Map<String,String>	getMetastoreSchema(String tableName)
boolean	tableExists(String tableName)
boolean	tableNotExistsOrDoesNotMatchSpecification(String tableName) Checks for the existence of a table that uses the manifest file approach and matches other requirements.
void	updateTableSchema(String tableName, com.google.cloud.bigquery.Schema schema, List<String> partitionFields) Updates the schema for the given table if the schema has changed.

Methods inherited from class org.apache.hudi.sync.common.HoodieSyncClient

generateTempTableName, getActiveTimeline, getAllPartitionPathsOnStorage, getBasePath, getDroppedPartitionsSince, getMetaClient, getPartitionEvents, getPartitionEvents, getStorageSchema, getStorageSchema, getTableType, getWrittenPartitionsSince, isBootstrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.sync.common.HoodieMetaSyncOperations

addPartitionsToTable, createDatabase, createOrReplaceTable, createTable, databaseExists, deleteLastReplicatedTimeStamp, dropPartitions, dropTable, generatePushDownFilter, getAllPartitions, getLastCommitCompletionTimeSynced, getLastCommitTimeSynced, getLastReplicatedTime, getMetastoreFieldSchemas, getPartitionsFromList, getStorageFieldSchemas, getTableLocation, updateLastCommitTimeSynced, updateLastReplicatedTimeStamp, updatePartitionsToTable, updateSerdeProperties, updateTableComments, updateTableProperties, updateTableSchema

Field Detail

config

protected final BigQuerySyncConfig config

Constructor Detail

HoodieBigQuerySyncClient

public HoodieBigQuerySyncClient(BigQuerySyncConfig config,
                                HoodieTableMetaClient metaClient)

Method Detail

createOrUpdateTableUsingBqManifestFile

public void createOrUpdateTableUsingBqManifestFile(String tableName,
                                                   String bqManifestFileUri,
                                                   String sourceUriPrefix,
                                                   com.google.cloud.bigquery.Schema schema)

createManifestTable

public void createManifestTable(String tableName,
                                String sourceUri)

updateTableSchema

public void updateTableSchema(String tableName,
                              com.google.cloud.bigquery.Schema schema,
                              List<String> partitionFields)

Updates the schema for the given table if the schema has changed. The schema passed in will not have the partition columns defined, so we add them back to the schema with the values read from the existing BigQuery table. This allows us to keep the partition field type in sync with how it is registered in BigQuery.

Parameters:

tableName - name of the table in BigQuery

schema - latest schema for the table

createVersionsTable

public void createVersionsTable(String tableName,
                                String sourceUri,
                                String sourceUriPrefix,
                                List<String> partitionFields)

createSnapshotView

public void createSnapshotView(String viewName,
                               String versionsTableName,
                               String manifestTableName)

getMetastoreSchema

public Map<String,String> getMetastoreSchema(String tableName)

datasetExists

public boolean datasetExists()

tableExists

public boolean tableExists(String tableName)

tableNotExistsOrDoesNotMatchSpecification

public boolean tableNotExistsOrDoesNotMatchSpecification(String tableName)

Checks for the existence of a table that uses the manifest file approach and matches other requirements.

Parameters:

tableName - name of the table

Returns:

Returns true if the table does not exist or if the table does exist but does not use the manifest file or table base path is outdated. False otherwise.

close

public void close()