org.apache.hudi.aws.sync

Class AwsGlueCatalogSyncTool

java.lang.Object

org.apache.hudi.sync.common.HoodieSyncTool

org.apache.hudi.hive.HiveSyncTool

org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool

All Implemented Interfaces:

AutoCloseable

public class AwsGlueCatalogSyncTool
extends HiveSyncTool

Currently Experimental. Utility class that implements syncing a Hudi Table with the AWS Glue Data Catalog (https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html) to enable querying via Glue ETLs, Athena etc.

Extends HiveSyncTool since most logic is similar to Hive syncing, expect using a different client AWSGlueCatalogSyncClient that implements the necessary functionality using Glue APIs.

Field Summary

Fields inherited from class org.apache.hudi.hive.HiveSyncTool

config, roTableName, snapshotTableName, SUFFIX_READ_OPTIMIZED_TABLE, SUFFIX_SNAPSHOT_TABLE, syncClient

Fields inherited from class org.apache.hudi.sync.common.HoodieSyncTool

hadoopConf, metrics, props

Constructor Summary

Constructors

Constructor and Description
AwsGlueCatalogSyncTool(Properties props, org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
protected void	initSyncClient(HiveSyncConfig hiveSyncConfig)
static void	main(String[] args)
protected boolean	shouldRecreateAndSyncTable()

Methods inherited from class org.apache.hudi.hive.HiveSyncTool

close, doSync, syncHoodieTable, syncHoodieTable

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AwsGlueCatalogSyncTool

public AwsGlueCatalogSyncTool(Properties props,
                              org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

initSyncClient

protected void initSyncClient(HiveSyncConfig hiveSyncConfig)

Overrides:

initSyncClient in class HiveSyncTool

shouldRecreateAndSyncTable

protected boolean shouldRecreateAndSyncTable()

Overrides:

shouldRecreateAndSyncTable in class HiveSyncTool

main

public static void main(String[] args)