org.apache.hudi.common.engine

Class HoodieLocalEngineContext

java.lang.Object

org.apache.hudi.common.engine.HoodieEngineContext

org.apache.hudi.common.engine.HoodieLocalEngineContext

public final class HoodieLocalEngineContext
extends HoodieEngineContext

A java based engine context, use this implementation on the query engine integrations if needed.

Field Summary

Fields inherited from class org.apache.hudi.common.engine.HoodieEngineContext

taskContextSupplier

Constructor Summary

Constructors

Constructor and Description
HoodieLocalEngineContext(StorageConfiguration<?> conf)
HoodieLocalEngineContext(StorageConfiguration<?> conf, TaskContextSupplier taskContextSupplier)

Method Summary

Modifier and Type	Method and Description
<I,O> O	aggregate(HoodieData<I> data, O zeroValue, Functions.Function2<O,I,O> seqOp, Functions.Function2<O,O,O> combOp) Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".
void	cancelAllJobs()
void	cancelJob(String jobId)
<T> HoodieData<T>	emptyHoodieData()
<I,O> List<O>	flatMap(List<I> data, SerializableFunction<I,Stream<O>> func, int parallelism)
<I> void	foreach(List<I> data, SerializableConsumer<I> consumer, int parallelism)
List<Integer>	getCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey)
Option<String>	getProperty(EngineProperty key)
<I,O> List<O>	map(List<I> data, SerializableFunction<I,O> func, int parallelism)
<I,K,V> Stream<ImmutablePair<K,V>>	mapPartitionsToPairAndReduceByKey(Stream<I> data, SerializablePairFlatMapFunction<Iterator<I>,K,V> flatMapToPairFunc, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
<I,K,V> Map<K,V>	mapToPair(List<I> data, SerializablePairFunction<I,K,V> func, Integer parallelism)
<I,K,V> List<V>	mapToPairAndReduceByKey(List<I> data, SerializablePairFunction<I,K,V> mapToPairFunc, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
HoodieAccumulator	newAccumulator()
<T> HoodieData<T>	parallelize(List<T> data, int parallelism)
void	putCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey, int... ids)
<I,K,V> List<V>	reduceByKey(List<Pair<K,V>> data, SerializableBiFunction<V,V,V> reduceFunc, int parallelism)
List<Integer>	removeCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey)
void	setJobStatus(String activeModule, String activityDescription)
void	setProperty(EngineProperty key, String value)

Methods inherited from class org.apache.hudi.common.engine.HoodieEngineContext

getStorageConf, getTaskContextSupplier, parallelize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieLocalEngineContext

public HoodieLocalEngineContext(StorageConfiguration<?> conf)

HoodieLocalEngineContext

public HoodieLocalEngineContext(StorageConfiguration<?> conf,
                                TaskContextSupplier taskContextSupplier)

Method Detail

newAccumulator

public HoodieAccumulator newAccumulator()

Specified by:

newAccumulator in class HoodieEngineContext

emptyHoodieData

public <T> HoodieData<T> emptyHoodieData()

Specified by:

emptyHoodieData in class HoodieEngineContext

parallelize

public <T> HoodieData<T> parallelize(List<T> data,
                                     int parallelism)

Specified by:

parallelize in class HoodieEngineContext

map

public <I,O> List<O> map(List<I> data,
                         SerializableFunction<I,O> func,
                         int parallelism)

Specified by:

map in class HoodieEngineContext

mapToPairAndReduceByKey

public <I,K,V> List<V> mapToPairAndReduceByKey(List<I> data,
                                               SerializablePairFunction<I,K,V> mapToPairFunc,
                                               SerializableBiFunction<V,V,V> reduceFunc,
                                               int parallelism)

Specified by:

mapToPairAndReduceByKey in class HoodieEngineContext

mapPartitionsToPairAndReduceByKey

public <I,K,V> Stream<ImmutablePair<K,V>> mapPartitionsToPairAndReduceByKey(Stream<I> data,
                                                                            SerializablePairFlatMapFunction<Iterator<I>,K,V> flatMapToPairFunc,
                                                                            SerializableBiFunction<V,V,V> reduceFunc,
                                                                            int parallelism)

Specified by:

mapPartitionsToPairAndReduceByKey in class HoodieEngineContext

reduceByKey

public <I,K,V> List<V> reduceByKey(List<Pair<K,V>> data,
                                   SerializableBiFunction<V,V,V> reduceFunc,
                                   int parallelism)

Specified by:

reduceByKey in class HoodieEngineContext

flatMap

public <I,O> List<O> flatMap(List<I> data,
                             SerializableFunction<I,Stream<O>> func,
                             int parallelism)

Specified by:

flatMap in class HoodieEngineContext

foreach

public <I> void foreach(List<I> data,
                        SerializableConsumer<I> consumer,
                        int parallelism)

Specified by:

foreach in class HoodieEngineContext

mapToPair

public <I,K,V> Map<K,V> mapToPair(List<I> data,
                                  SerializablePairFunction<I,K,V> func,
                                  Integer parallelism)

Specified by:

mapToPair in class HoodieEngineContext

setProperty

public void setProperty(EngineProperty key,
                        String value)

Specified by:

setProperty in class HoodieEngineContext

getProperty

public Option<String> getProperty(EngineProperty key)

Specified by:

getProperty in class HoodieEngineContext

setJobStatus

public void setJobStatus(String activeModule,
                         String activityDescription)

Specified by:

setJobStatus in class HoodieEngineContext

putCachedDataIds

public void putCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey,
                             int... ids)

Specified by:

putCachedDataIds in class HoodieEngineContext

getCachedDataIds

public List<Integer> getCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey)

Specified by:

getCachedDataIds in class HoodieEngineContext

removeCachedDataIds

public List<Integer> removeCachedDataIds(HoodieData.HoodieDataCacheKey cacheKey)

Specified by:

removeCachedDataIds in class HoodieEngineContext

cancelJob

public void cancelJob(String jobId)

Specified by:

cancelJob in class HoodieEngineContext

cancelAllJobs

public void cancelAllJobs()

Specified by:

cancelAllJobs in class HoodieEngineContext

aggregate

public <I,O> O aggregate(HoodieData<I> data,
                         O zeroValue,
                         Functions.Function2<O,I,O> seqOp,
                         Functions.Function2<O,O,O> combOp)

Description copied from class: HoodieEngineContext

Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".

Specified by:

aggregate in class HoodieEngineContext

Type Parameters:

I - input object type

O - output object type

zeroValue - the initial value for the accumulated result of each partition for the seqOp operator

seqOp - function to aggregate the elements of each partition

combOp - function to combine results from different partitions

Returns:

the result of the aggregation