org.apache.hudi.examples.quickstart

Class HoodieSparkQuickstart

java.lang.Object

org.apache.hudi.examples.quickstart.HoodieSparkQuickstart

public final class HoodieSparkQuickstart
extends Object

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	delete(org.apache.spark.sql.SparkSession spark, String tablePath, String tableName) Delete data based in data information.
static void	deleteByPartition(org.apache.spark.sql.SparkSession spark, String tablePath, String tableName) Delete the data of the first partition.
static void	incrementalQuery(org.apache.spark.sql.SparkSession spark, String tablePath, String tableName) Hudi also provides capability to obtain a stream of records that changed since given commit timestamp.
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	insertData(org.apache.spark.sql.SparkSession spark, org.apache.spark.api.java.JavaSparkContext jsc, String tablePath, String tableName, HoodieExampleDataGenerator<HoodieAvroPayload> dataGen) Generate some new trips, load them into a DataFrame and write the DataFrame into the Hudi dataset as below.
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	insertOverwriteData(org.apache.spark.sql.SparkSession spark, org.apache.spark.api.java.JavaSparkContext jsc, String tablePath, String tableName, HoodieExampleDataGenerator<HoodieAvroPayload> dataGen) Generate new records, load them into a Dataset and insert-overwrite it into the Hudi dataset
static void	main(String[] args)
static void	pointInTimeQuery(org.apache.spark.sql.SparkSession spark, String tablePath, String tableName) Lets look at how to query data as of a specific time.
static void	queryData(org.apache.spark.sql.SparkSession spark, org.apache.spark.api.java.JavaSparkContext jsc, String tablePath, String tableName, HoodieExampleDataGenerator<HoodieAvroPayload> dataGen) Load the data files into a DataFrame.
static void	runQuickstart(org.apache.spark.api.java.JavaSparkContext jsc, org.apache.spark.sql.SparkSession spark, String tableName, String tablePath) Visible for testing
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	updateData(org.apache.spark.sql.SparkSession spark, org.apache.spark.api.java.JavaSparkContext jsc, String tablePath, String tableName, HoodieExampleDataGenerator<HoodieAvroPayload> dataGen) This is similar to inserting new data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

main

public static void main(String[] args)

runQuickstart

public static void runQuickstart(org.apache.spark.api.java.JavaSparkContext jsc,
                                 org.apache.spark.sql.SparkSession spark,
                                 String tableName,
                                 String tablePath)

Visible for testing

insertData

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> insertData(org.apache.spark.sql.SparkSession spark,
                                                                                org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                String tablePath,
                                                                                String tableName,
                                                                                HoodieExampleDataGenerator<HoodieAvroPayload> dataGen)

Generate some new trips, load them into a DataFrame and write the DataFrame into the Hudi dataset as below.

insertOverwriteData

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> insertOverwriteData(org.apache.spark.sql.SparkSession spark,
                                                                                         org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                         String tablePath,
                                                                                         String tableName,
                                                                                         HoodieExampleDataGenerator<HoodieAvroPayload> dataGen)

Generate new records, load them into a Dataset and insert-overwrite it into the Hudi dataset

queryData

public static void queryData(org.apache.spark.sql.SparkSession spark,
                             org.apache.spark.api.java.JavaSparkContext jsc,
                             String tablePath,
                             String tableName,
                             HoodieExampleDataGenerator<HoodieAvroPayload> dataGen)

Load the data files into a DataFrame.

updateData

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> updateData(org.apache.spark.sql.SparkSession spark,
                                                                                org.apache.spark.api.java.JavaSparkContext jsc,
                                                                                String tablePath,
                                                                                String tableName,
                                                                                HoodieExampleDataGenerator<HoodieAvroPayload> dataGen)

This is similar to inserting new data. Generate updates to existing trips using the data generator, load into a DataFrame and write DataFrame into the hudi dataset.

delete

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> delete(org.apache.spark.sql.SparkSession spark,
                                                                            String tablePath,
                                                                            String tableName)

Delete data based in data information.

deleteByPartition

public static void deleteByPartition(org.apache.spark.sql.SparkSession spark,
                                     String tablePath,
                                     String tableName)

Delete the data of the first partition.

incrementalQuery

public static void incrementalQuery(org.apache.spark.sql.SparkSession spark,
                                    String tablePath,
                                    String tableName)

Hudi also provides capability to obtain a stream of records that changed since given commit timestamp. This can be achieved using Hudi’s incremental view and providing a begin time from which changes need to be streamed. We do not need to specify endTime, if we want all changes after the given commit (as is the common case).

pointInTimeQuery

public static void pointInTimeQuery(org.apache.spark.sql.SparkSession spark,
                                    String tablePath,
                                    String tableName)

Lets look at how to query data as of a specific time. The specific time can be represented by pointing endTime to a specific commit time and beginTime to “000” (denoting earliest possible commit time).