de.viadee.ki.sparkimporter.processing.steps.importing

Class InitialCleanupStep

java.lang.Object

de.viadee.ki.sparkimporter.processing.steps.importing.InitialCleanupStep

All Implemented Interfaces:

PreprocessingStepInterface

public class InitialCleanupStep
extends Object
implements PreprocessingStepInterface

Constructor Summary

Constructors

Constructor and Description
InitialCleanupStep()

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	runPreprocessingStep(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset, boolean writeStepResultIntoFile, String dataLevel, Map<String,Object> parameters) Defines one processing step and what to do with the data

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InitialCleanupStep

public InitialCleanupStep()

Method Detail

runPreprocessingStep

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> runPreprocessingStep(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataset,
                                                                                   boolean writeStepResultIntoFile,
                                                                                   String dataLevel,
                                                                                   Map<String,Object> parameters)

Description copied from interface: PreprocessingStepInterface

Defines one processing step and what to do with the data

Specified by:

runPreprocessingStep in interface PreprocessingStepInterface

Parameters:

dataset - the incoming dataset for this processing step

Returns:

the resulting dataset of the processing step