org.apache.hudi.common.util.queue

Class BoundedInMemoryExecutor<I,O,E>

java.lang.Object

org.apache.hudi.common.util.queue.BaseHoodieQueueBasedExecutor<I,O,E>

org.apache.hudi.common.util.queue.BoundedInMemoryExecutor<I,O,E>

All Implemented Interfaces:

HoodieExecutor<E>

public class BoundedInMemoryExecutor<I,O,E>
extends BaseHoodieQueueBasedExecutor<I,O,E>

Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate concurrent execution of these actors communicating through a central bounded queue

Field Summary

Fields inherited from class org.apache.hudi.common.util.queue.BaseHoodieQueueBasedExecutor

consumer, queue

Constructor Summary

Constructors

Constructor and Description
BoundedInMemoryExecutor(long bufferLimitInBytes, int recordSamplingRate, int recordCacheLimit, Iterator<I> inputItr, HoodieConsumer<O,E> consumer, Function<I,O> transformFunction, Runnable preExecuteRunnable)
BoundedInMemoryExecutor(long bufferLimitInBytes, int recordSamplingRate, int recordCacheLimit, List<HoodieProducer<I>> producers, Option<HoodieConsumer<O,E>> consumer, Function<I,O> transformFunction, SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable)
BoundedInMemoryExecutor(long bufferLimitInBytes, Iterator<I> inputItr, HoodieConsumer<O,E> consumer, Function<I,O> transformFunction, Runnable preExecuteRunnable)
BoundedInMemoryExecutor(long bufferLimitInBytes, List<HoodieProducer<I>> producers, Option<HoodieConsumer<O,E>> consumer, Function<I,O> transformFunction, SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable)

Method Summary

Modifier and Type	Method and Description
protected void	doConsume(HoodieMessageQueue<I,O> queue, HoodieConsumer<O,E> consumer)
Iterator<O>	getRecordIterator()

Methods inherited from class org.apache.hudi.common.util.queue.BaseHoodieQueueBasedExecutor

awaitTermination, doProduce, execute, isRunning, setUp, shutdownNow, startProducingAsync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BoundedInMemoryExecutor

public BoundedInMemoryExecutor(long bufferLimitInBytes,
                               Iterator<I> inputItr,
                               HoodieConsumer<O,E> consumer,
                               Function<I,O> transformFunction,
                               Runnable preExecuteRunnable)

BoundedInMemoryExecutor

public BoundedInMemoryExecutor(long bufferLimitInBytes,
                               int recordSamplingRate,
                               int recordCacheLimit,
                               Iterator<I> inputItr,
                               HoodieConsumer<O,E> consumer,
                               Function<I,O> transformFunction,
                               Runnable preExecuteRunnable)

BoundedInMemoryExecutor

public BoundedInMemoryExecutor(long bufferLimitInBytes,
                               List<HoodieProducer<I>> producers,
                               Option<HoodieConsumer<O,E>> consumer,
                               Function<I,O> transformFunction,
                               SizeEstimator<O> sizeEstimator,
                               Runnable preExecuteRunnable)

BoundedInMemoryExecutor

public BoundedInMemoryExecutor(long bufferLimitInBytes,
                               int recordSamplingRate,
                               int recordCacheLimit,
                               List<HoodieProducer<I>> producers,
                               Option<HoodieConsumer<O,E>> consumer,
                               Function<I,O> transformFunction,
                               SizeEstimator<O> sizeEstimator,
                               Runnable preExecuteRunnable)

Method Detail

doConsume

protected void doConsume(HoodieMessageQueue<I,O> queue,
                         HoodieConsumer<O,E> consumer)

Specified by:

doConsume in class BaseHoodieQueueBasedExecutor<I,O,E>

getRecordIterator

public Iterator<O> getRecordIterator()