de.viadee.ki.sparkimporter.util

Class SparkImporterUtils

java.lang.Object

de.viadee.ki.sparkimporter.util.SparkImporterUtils

public class SparkImporterUtils
extends Object

Method Summary

Modifier and Type	Method and Description
<T> scala.collection.Seq<T>	asSeq(List<T> values) implemented help method as per https://stackoverflow.com/questions/40741459/scala-collection-seq-doesnt-work-on-java
static SparkImporterUtils	getInstance()
String	md5CecksumOfObject(Object obj)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	removeDuplicatedColumns(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	removeEmptyLinesAfterImport(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset) removes lines with no process instance id
void	writeDatasetToCSV(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet, String subDirectory)
void	writeDatasetToParquet(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet, String subDirectory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getInstance

public static SparkImporterUtils getInstance()

md5CecksumOfObject

public String md5CecksumOfObject(Object obj)
                          throws IOException,
                                 NoSuchAlgorithmException

Throws:

IOException

NoSuchAlgorithmException

writeDatasetToParquet

public void writeDatasetToParquet(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet,
                                  String subDirectory)

writeDatasetToCSV

public void writeDatasetToCSV(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet,
                              String subDirectory)

removeDuplicatedColumns

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> removeDuplicatedColumns(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset)

removeEmptyLinesAfterImport

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> removeEmptyLinesAfterImport(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset)

removes lines with no process instance id

Parameters:

dataset - dataset to be cleaned

Returns:

the cleaned dataset

asSeq

public <T> scala.collection.Seq<T> asSeq(List<T> values)

implemented help method as per https://stackoverflow.com/questions/40741459/scala-collection-seq-doesnt-work-on-java

Type Parameters:

T - Type of objects to be converted

Parameters:

values - List to be convert to Scala Seq

Returns:

the Scala Seq