org.apache.hudi.cli.commands

Class SparkMain

java.lang.Object

org.apache.hudi.cli.commands.SparkMain

public class SparkMain
extends Object

This class deals with initializing spark context based on command entered to hudi-cli.

Constructor Summary

Constructors

Constructor and Description
SparkMain()

Method Summary

Modifier and Type	Method and Description
static void	addAppArgs(org.apache.spark.launcher.SparkLauncher sparkLauncher, org.apache.hudi.cli.commands.SparkMain.SparkCommand cmd, String... args)
protected static void	clean(org.apache.spark.api.java.JavaSparkContext jsc, String basePath, String propsFilePath, List<String> configs)
protected static int	deleteMarker(org.apache.spark.api.java.JavaSparkContext jsc, String instantTime, String basePath)
static void	main(String[] args)
static int	renamePartition(org.apache.spark.api.java.JavaSparkContext jsc, String basePath, String oldPartition, String newPartition)
static int	repairDeprecatedPartition(org.apache.spark.api.java.JavaSparkContext jsc, String basePath)
protected static int	upgradeOrDowngradeTable(org.apache.spark.api.java.JavaSparkContext jsc, String basePath, String toVersion) Upgrade or downgrade table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkMain

public SparkMain()

Method Detail

addAppArgs

public static void addAppArgs(org.apache.spark.launcher.SparkLauncher sparkLauncher,
                              org.apache.hudi.cli.commands.SparkMain.SparkCommand cmd,
                              String... args)

main

public static void main(String[] args)

clean

protected static void clean(org.apache.spark.api.java.JavaSparkContext jsc,
                            String basePath,
                            String propsFilePath,
                            List<String> configs)

deleteMarker

protected static int deleteMarker(org.apache.spark.api.java.JavaSparkContext jsc,
                                  String instantTime,
                                  String basePath)

repairDeprecatedPartition

public static int repairDeprecatedPartition(org.apache.spark.api.java.JavaSparkContext jsc,
                                            String basePath)

renamePartition

public static int renamePartition(org.apache.spark.api.java.JavaSparkContext jsc,
                                  String basePath,
                                  String oldPartition,
                                  String newPartition)

upgradeOrDowngradeTable

protected static int upgradeOrDowngradeTable(org.apache.spark.api.java.JavaSparkContext jsc,
                                             String basePath,
                                             String toVersion)

Upgrade or downgrade table.

Parameters:

jsc - instance of JavaSparkContext to use.

basePath - base path of the dataset.

toVersion - version to which upgrade/downgrade to be done.

Returns:

0 if success, else -1.

Throws:

Exception