org.apache.hudi.cli.utils

Class SparkUtil

java.lang.Object

org.apache.hudi.cli.utils.SparkUtil

public class SparkUtil
extends Object

Utility functions dealing with Spark.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_SPARK_MASTER

Constructor Summary

Constructors

Constructor and Description
SparkUtil()

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.SparkConf	getDefaultConf(String appName, org.apache.hudi.common.util.Option<String> sparkMaster) Get the default spark configuration.
static org.apache.spark.api.java.JavaSparkContext	initJavaSparkContext(org.apache.spark.SparkConf sparkConf)
static org.apache.spark.api.java.JavaSparkContext	initJavaSparkContext(String name)
static org.apache.spark.api.java.JavaSparkContext	initJavaSparkContext(String name, org.apache.hudi.common.util.Option<String> master, org.apache.hudi.common.util.Option<String> executorMemory)
static org.apache.spark.launcher.SparkLauncher	initLauncher(String propertiesFile) TODO: Need to fix a bunch of hardcoded stuff here eg: history server, spark distro.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_SPARK_MASTER

public static final String DEFAULT_SPARK_MASTER

See Also:

Constant Field Values

Constructor Detail

SparkUtil

public SparkUtil()

Method Detail

initLauncher

public static org.apache.spark.launcher.SparkLauncher initLauncher(String propertiesFile)
                                                            throws URISyntaxException

TODO: Need to fix a bunch of hardcoded stuff here eg: history server, spark distro.

Throws:

URISyntaxException

getDefaultConf

public static org.apache.spark.SparkConf getDefaultConf(String appName,
                                                        org.apache.hudi.common.util.Option<String> sparkMaster)

Get the default spark configuration.

Parameters:

appName - - Spark application name

sparkMaster - - Spark master node name

Returns:

Spark configuration

initJavaSparkContext

public static org.apache.spark.api.java.JavaSparkContext initJavaSparkContext(String name)

initJavaSparkContext

public static org.apache.spark.api.java.JavaSparkContext initJavaSparkContext(String name,
                                                                              org.apache.hudi.common.util.Option<String> master,
                                                                              org.apache.hudi.common.util.Option<String> executorMemory)

initJavaSparkContext

public static org.apache.spark.api.java.JavaSparkContext initJavaSparkContext(org.apache.spark.SparkConf sparkConf)