de.viadee.ki.sparkimporter.processing.interfaces

Interface PreprocessingStepInterface

All Known Implementing Classes:

AddReducedColumnsToDatasetStep, AddVariablesColumnsStep, AggregateActivityInstancesStep, AggregateProcessInstancesStep, AggregateVariableUpdatesStep, ColumnHashStep, ColumnRemoveStep, ColumnsPreparationStep, CreateColumnsFromJsonStep, DataFilterStep, DetermineVariableTypesStep, FillActivityInstancesHistoryStep, InitialCleanupStep, JsonVariableFilterStep, ReduceColumnsDatasetStep, TypeCastStep, VariableFilterStep, VariableNameMappingStep, VariablesTypeEscalationStep, WriteToDataSinkStep, WriteToDiscStep

public interface PreprocessingStepInterface

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	runPreprocessingStep(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet, boolean writeStepResultIntoFile, String dataLevel, Map<String,Object> parameters) Defines one processing step and what to do with the data

Method Detail

runPreprocessingStep

org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> runPreprocessingStep(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataSet,
                                                                            boolean writeStepResultIntoFile,
                                                                            String dataLevel,
                                                                            Map<String,Object> parameters)

Defines one processing step and what to do with the data

Parameters:

dataSet - the incoming dataset for this processing step

Returns:

the resulting dataset of the processing step