HoodieListPairData (hudi-common 1.0.0-beta2 API)

java.lang.Object
- org.apache.hudi.common.data.HoodieBaseListData<Pair<K,V>>
- - org.apache.hudi.common.data.HoodieListPairData<K,V>

Type Parameters:

K - type of the key in the pair

V - type of the value in the pair

All Implemented Interfaces:

Serializable, HoodiePairData<K,V>
```
public class HoodieListPairData<K,V>
extends HoodieBaseListData<Pair<K,V>>
implements HoodiePairData<K,V>
```
In-memory implementation of HoodiePairData holding internally a Stream of Pairs. HoodieListData can have either of the 2 execution semantics:
1. Eager: with every operation being executed right away
2. Lazy: with every operation being "stacked up", with it execution postponed until "terminal" operation is invoked
NOTE: This is an in-memory counterpart for HoodieJavaPairRDD, and it strives to provide similar semantic as RDD container -- all intermediate (non-terminal, not de-referencing the stream like "collect", "groupBy", etc) operations are executed *lazily*. This allows to make sure that compute/memory churn is minimal since only necessary computations will ultimately be performed. Please note, however, that while RDD container allows the same collection to be de-referenced more than once (ie terminal operation invoked more than once), HoodieListData allows that only when instantiated w/ an eager execution semantic.
See Also:

Serialized Form

Field Summary
- Fields inherited from class org.apache.hudi.common.data.HoodieBaseListData
  data, lazy

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`List<Pair<K,V>>`	`collectAsList()` Collects results of the underlying collection into a `List>` This is a terminal operation
`long`	`count()` Returns number of held pairs
`Map<K,Long>`	`countByKey()` Counts the number of pairs grouping them by key
`int`	`deduceNumPartitions()`
`static <K,V> HoodieListPairData<K,V>`	`eager(List<Pair<K,V>> data)`
`static <K,V> HoodieListPairData<K,V>`	`eager(Map<K,List<V>> data)`
`<W> HoodiePairData<K,W>`	`flatMapValues(SerializableFunction<V,Iterator<W>> func)`
`List<Pair<K,V>>`	`get()`
`HoodiePairData<K,Iterable<V>>`	`groupByKey()` Groups the values for each key in the dataset into a single sequence
`HoodieData<K>`	`keys()` Returns a `HoodieData` holding the key from every corresponding pair
`static <K,V> HoodieListPairData<K,V>`	`lazy(List<Pair<K,V>> data)`
`static <K,V> HoodieListPairData<K,V>`	`lazy(Map<K,List<V>> data)`
`<W> HoodiePairData<K,Pair<V,Option<W>>>`	`leftOuterJoin(HoodiePairData<K,W> other)` Performs a left outer join of this dataset against `other`.
`<O> HoodieData<O>`	`map(SerializableFunction<Pair<K,V>,O> func)` Maps key-value pairs of this `HoodiePairData` container leveraging provided mapper NOTE: That this returns `HoodieData` and not `HoodiePairData`
`<L,W> HoodiePairData<L,W>`	`mapToPair(SerializablePairFunction<Pair<K,V>,L,W> mapToPairFunc)`
`<W> HoodiePairData<K,W>`	`mapValues(SerializableFunction<V,W> func)` Maps values of this `HoodiePairData` container leveraging provided mapper
`void`	`persist(String cacheConfig)` Persists the data (if applicable)
`HoodiePairData<K,V>`	`reduceByKey(SerializableBiFunction<V,V,V> combiner, int parallelism)` Reduces original sequence by de-duplicating the pairs w/ the same key, using provided binary operator `combiner`.
`void`	`unpersist()` Un-persists the data (if applicable)
`HoodieData<V>`	`values()` Returns a `HoodieData` holding the value from every corresponding pair

Methods inherited from class org.apache.hudi.common.data.HoodieBaseListData
asStream, isEmpty

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - get
```
public List<Pair<K,V>> get()
```
    Specified by:
    
    get in interface HoodiePairData<K,V>
    
    Returns:
    
    the collection of pairs.
  - persist
```
public void persist(String cacheConfig)
```
    Description copied from interface: HoodiePairData
    
    Persists the data (if applicable)
    
    Specified by:
    
    persist in interface HoodiePairData<K,V>
    
    Parameters:
    
    cacheConfig - config value for caching.
  - unpersist
```
public void unpersist()
```
    Description copied from interface: HoodiePairData
    
    Un-persists the data (if applicable)
    
    Specified by:
    
    unpersist in interface HoodiePairData<K,V>
  - keys
```
public HoodieData<K> keys()
```
    Description copied from interface: HoodiePairData
    
    Returns a HoodieData holding the key from every corresponding pair
    
    Specified by:
    
    keys in interface HoodiePairData<K,V>
  - values
```
public HoodieData<V> values()
```
    Description copied from interface: HoodiePairData
    
    Returns a HoodieData holding the value from every corresponding pair
    
    Specified by:
    
    values in interface HoodiePairData<K,V>
  - countByKey
```
public Map<K,Long> countByKey()
```
    Description copied from interface: HoodiePairData
    
    Counts the number of pairs grouping them by key
    
    Specified by:
    
    countByKey in interface HoodiePairData<K,V>
  - groupByKey
```
public HoodiePairData<K,Iterable<V>> groupByKey()
```
    Description copied from interface: HoodiePairData
    
    Groups the values for each key in the dataset into a single sequence
    
    Specified by:
    
    groupByKey in interface HoodiePairData<K,V>
  - reduceByKey
```
public HoodiePairData<K,V> reduceByKey(SerializableBiFunction<V,V,V> combiner,
                                       int parallelism)
```
    Description copied from interface: HoodiePairData
    
    Reduces original sequence by de-duplicating the pairs w/ the same key, using provided binary operator combiner. Returns an instance of HoodiePairData holding the "de-duplicated" pairs, ie only pairs with unique keys.
    
    Specified by:
    
    reduceByKey in interface HoodiePairData<K,V>
    
    Parameters:
    
    combiner - method to combine values of the pairs with the same key
    
    parallelism - target parallelism (if applicable)
  - map
```
public <O> HoodieData<O> map(SerializableFunction<Pair<K,V>,O> func)
```
    Description copied from interface: HoodiePairData
    
    Maps key-value pairs of this HoodiePairData container leveraging provided mapper NOTE: That this returns HoodieData and not HoodiePairData
    
    Specified by:
    
    map in interface HoodiePairData<K,V>
  - mapValues
```
public <W> HoodiePairData<K,W> mapValues(SerializableFunction<V,W> func)
```
    Description copied from interface: HoodiePairData
    
    Maps values of this HoodiePairData container leveraging provided mapper
    
    Specified by:
    
    mapValues in interface HoodiePairData<K,V>
  - flatMapValues
```
public <W> HoodiePairData<K,W> flatMapValues(SerializableFunction<V,Iterator<W>> func)
```
  - mapToPair
```
public <L,W> HoodiePairData<L,W> mapToPair(SerializablePairFunction<Pair<K,V>,L,W> mapToPairFunc)
```
    Specified by:
    
    mapToPair in interface HoodiePairData<K,V>
    
    Type Parameters:
    
    L - new key type.
    
    W - new value type.
    
    Parameters:
    
    mapToPairFunc - serializable map function to generate another pair.
    
    Returns:
    
    containing the result. Actual execution may be deferred.
  - leftOuterJoin
```
public <W> HoodiePairData<K,Pair<V,Option<W>>> leftOuterJoin(HoodiePairData<K,W> other)
```
    Description copied from interface: HoodiePairData
    
    Performs a left outer join of this dataset against other. For each element (k, v) in this, the resulting HoodiePairData will either contain all pairs (k, (v, Some(w))) for every w in the other, or the pair (k, (v, None)) if no elements in other have the pair w/ a key k
    
    Specified by:
    
    leftOuterJoin in interface HoodiePairData<K,V>
    
    Type Parameters:
    
    W - value type of the other HoodiePairData
    
    Parameters:
    
    other - the other HoodiePairData
    
    Returns:
    
    containing the result of the left outer join
  - count
```
public long count()
```
    Description copied from interface: HoodiePairData
    
    Returns number of held pairs
    
    Specified by:
    
    count in interface HoodiePairData<K,V>
    
    Overrides:
    
    count in class HoodieBaseListData<Pair<K,V>>
  - collectAsList
```
public List<Pair<K,V>> collectAsList()
```
    Description copied from interface: HoodiePairData
    
    Collects results of the underlying collection into a List> This is a terminal operation
    
    Specified by:
    
    collectAsList in interface HoodiePairData<K,V>
    
    Overrides:
    
    collectAsList in class HoodieBaseListData<Pair<K,V>>
  - deduceNumPartitions
```
public int deduceNumPartitions()
```
    Specified by:
    
    deduceNumPartitions in interface HoodiePairData<K,V>
    
    Returns:
    
    the deduce number of shuffle partitions
  - lazy
```
public static <K,V> HoodieListPairData<K,V> lazy(List<Pair<K,V>> data)
```
  - eager
```
public static <K,V> HoodieListPairData<K,V> eager(List<Pair<K,V>> data)
```
  - lazy
```
public static <K,V> HoodieListPairData<K,V> lazy(Map<K,List<V>> data)
```
  - eager
```
public static <K,V> HoodieListPairData<K,V> eager(Map<K,List<V>> data)
```

Class HoodieListPairData<K,V>

Field Summary

Fields inherited from class org.apache.hudi.common.data.HoodieBaseListData

Method Summary

Methods inherited from class org.apache.hudi.common.data.HoodieBaseListData

Methods inherited from class java.lang.Object

Method Detail

get

persist

unpersist

keys

values

countByKey

groupByKey

reduceByKey

map

mapValues

flatMapValues

mapToPair

leftOuterJoin

count

collectAsList

deduceNumPartitions

lazy

eager

lazy

eager