org.apache.hudi.common.data

Interface HoodiePairData<K,V>

Type Parameters:

K - type of key.

V - type of value.

All Superinterfaces:

Serializable

All Known Implementing Classes:

HoodieListPairData

public interface HoodiePairData<K,V>
extends Serializable

An abstraction for pairs of key in type K and value in type V to store the reference and do transformation.

Method Summary

Modifier and Type	Method and Description
List<Pair<K,V>>	collectAsList() Collects results of the underlying collection into a List> This is a terminal operation
long	count() Returns number of held pairs
Map<K,Long>	countByKey() Counts the number of pairs grouping them by key
int	deduceNumPartitions()
Object	get()
HoodiePairData<K,Iterable<V>>	groupByKey() Groups the values for each key in the dataset into a single sequence
HoodieData<K>	keys() Returns a HoodieData holding the key from every corresponding pair
<W> HoodiePairData<K,Pair<V,Option<W>>>	leftOuterJoin(HoodiePairData<K,W> other) Performs a left outer join of this dataset against other.
<O> HoodieData<O>	map(SerializableFunction<Pair<K,V>,O> func) Maps key-value pairs of this HoodiePairData container leveraging provided mapper NOTE: That this returns HoodieData and not HoodiePairData
<L,W> HoodiePairData<L,W>	mapToPair(SerializablePairFunction<Pair<K,V>,L,W> mapToPairFunc)
<W> HoodiePairData<K,W>	mapValues(SerializableFunction<V,W> func) Maps values of this HoodiePairData container leveraging provided mapper
void	persist(String cacheConfig) Persists the data (if applicable)
HoodiePairData<K,V>	reduceByKey(SerializableBiFunction<V,V,V> combiner, int parallelism) Reduces original sequence by de-duplicating the pairs w/ the same key, using provided binary operator combiner.
void	unpersist() Un-persists the data (if applicable)
HoodieData<V>	values() Returns a HoodieData holding the value from every corresponding pair

Method Detail

get

Object get()

Returns:

the collection of pairs.

persist

void persist(String cacheConfig)

Persists the data (if applicable)

Parameters:

cacheConfig - config value for caching.

unpersist

void unpersist()

Un-persists the data (if applicable)

keys

HoodieData<K> keys()

Returns a HoodieData holding the key from every corresponding pair

values

HoodieData<V> values()

Returns a HoodieData holding the value from every corresponding pair

count

long count()

Returns number of held pairs

countByKey

Map<K,Long> countByKey()

Counts the number of pairs grouping them by key

groupByKey

HoodiePairData<K,Iterable<V>> groupByKey()

Groups the values for each key in the dataset into a single sequence

reduceByKey

HoodiePairData<K,V> reduceByKey(SerializableBiFunction<V,V,V> combiner,
                                int parallelism)

Reduces original sequence by de-duplicating the pairs w/ the same key, using provided binary operator combiner. Returns an instance of HoodiePairData holding the "de-duplicated" pairs, ie only pairs with unique keys.

Parameters:

combiner - method to combine values of the pairs with the same key

parallelism - target parallelism (if applicable)

map

<O> HoodieData<O> map(SerializableFunction<Pair<K,V>,O> func)

Maps key-value pairs of this HoodiePairData container leveraging provided mapper NOTE: That this returns HoodieData and not HoodiePairData

mapValues

<W> HoodiePairData<K,W> mapValues(SerializableFunction<V,W> func)

Maps values of this HoodiePairData container leveraging provided mapper

mapToPair

<L,W> HoodiePairData<L,W> mapToPair(SerializablePairFunction<Pair<K,V>,L,W> mapToPairFunc)

Type Parameters:

L - new key type.

W - new value type.

Parameters:

mapToPairFunc - serializable map function to generate another pair.

Returns:

containing the result. Actual execution may be deferred.

leftOuterJoin

<W> HoodiePairData<K,Pair<V,Option<W>>> leftOuterJoin(HoodiePairData<K,W> other)

Performs a left outer join of this dataset against other. For each element (k, v) in this, the resulting HoodiePairData will either contain all pairs (k, (v, Some(w))) for every w in the other, or the pair (k, (v, None)) if no elements in other have the pair w/ a key k

Type Parameters:

W - value type of the other HoodiePairData

Parameters:

other - the other HoodiePairData

Returns:

containing the result of the left outer join

collectAsList

List<Pair<K,V>> collectAsList()

Collects results of the underlying collection into a List> This is a terminal operation

deduceNumPartitions

int deduceNumPartitions()

Returns:

the deduce number of shuffle partitions