org.apache.hudi.client.utils

Class LazyIterableIterator<I,O>

java.lang.Object

org.apache.hudi.client.utils.LazyIterableIterator<I,O>

All Implemented Interfaces:

Iterable<O>, Iterator<O>

Direct Known Subclasses:

HoodieLazyInsertIterable

public abstract class LazyIterableIterator<I,O>
extends Object
implements Iterable<O>, Iterator<O>

(NOTE: Adapted from Apache SystemML) This class is a generic base class for lazy, single pass inputItr classes in order to simplify the implementation of lazy iterators for mapPartitions use cases. Note [SPARK-3369], which gives the reasons for backwards compatibility with regard to the iterable API despite Spark's single pass nature.

Provide a way to obtain a inputItr of type O (output), out of an inputItr of type I (input)

Things to remember: - Assumes Spark calls hasNext() to check for elements, before calling next() to obtain them - Assumes hasNext() gets called atleast once. - Concrete Implementation is responsible for calling inputIterator.next() and doing the processing in computeNext()

Field Summary

Fields

Modifier and Type	Field and Description
protected Iterator<I>	inputItr

Constructor Summary

Constructors

Constructor and Description
LazyIterableIterator(Iterator<I> in)

Method Summary

Modifier and Type	Method and Description
protected abstract O	computeNext() Block computation to be overwritten by sub classes.
protected abstract void	end() Called once, after all elements are processed.
boolean	hasNext()
Iterator<O>	iterator()
O	next()
void	remove()
protected abstract void	start() Called once, before any elements are processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

inputItr

protected Iterator<I> inputItr

Constructor Detail

LazyIterableIterator

public LazyIterableIterator(Iterator<I> in)

Method Detail

start

protected abstract void start()

Called once, before any elements are processed.

computeNext

protected abstract O computeNext()

Block computation to be overwritten by sub classes.

end

protected abstract void end()

Called once, after all elements are processed.

iterator

public Iterator<O> iterator()

Specified by:

iterator in interface Iterable<O>

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<O>

next

public O next()

Specified by:

next in interface Iterator<O>

remove

public void remove()

Specified by:

remove in interface Iterator<O>