org.apache.hudi.index.simple

Class HoodieGlobalSimpleIndex<T extends HoodieRecordPayload<T>>

java.lang.Object

org.apache.hudi.index.HoodieIndex<T,Object,Object,Object>

org.apache.hudi.index.simple.HoodieSimpleIndex<T>

org.apache.hudi.index.simple.HoodieGlobalSimpleIndex<T>

Type Parameters:

T -

All Implemented Interfaces:

Serializable

public class HoodieGlobalSimpleIndex<T extends HoodieRecordPayload<T>>
extends HoodieSimpleIndex<T>

A global simple index which reads interested fields(record key and partition path) from base files and joins with incoming records to find the tagged location.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.index.HoodieIndex

HoodieIndex.IndexType

Field Summary

Fields inherited from class org.apache.hudi.index.HoodieIndex

config

Constructor Summary

Constructors

Constructor and Description
HoodieGlobalSimpleIndex(HoodieWriteConfig config, Option<BaseKeyGenerator> keyGeneratorOpt)

Method Summary

Modifier and Type	Method and Description
protected HoodiePairData<HoodieKey,HoodieRecordLocation>	fetchAllRecordLocations(HoodieEngineContext context, HoodieTable hoodieTable, int parallelism) Fetch record locations for passed in HoodieKeys.
protected List<Pair<String,HoodieBaseFile>>	getAllBaseFilesInTable(HoodieEngineContext context, HoodieTable hoodieTable) Load all files for all partitions as pair data.
boolean	isGlobal() An index is `global` if HoodieKey to fileID mapping, does not depend on the `partitionPath`.
HoodieData<HoodieRecord<T>>	tagLocation(HoodieData<HoodieRecord<T>> records, HoodieEngineContext context, HoodieTable hoodieTable) Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).
protected HoodieData<HoodieRecord<T>>	tagLocationInternal(HoodieData<HoodieRecord<T>> inputRecords, HoodieEngineContext context, HoodieTable hoodieTable) Tags records location for incoming records.

Methods inherited from class org.apache.hudi.index.simple.HoodieSimpleIndex

canIndexLogFiles, fetchRecordLocations, fetchRecordLocationsForAffectedPartitions, isImplicitWithStorage, rollbackCommit, updateLocation

Methods inherited from class org.apache.hudi.index.HoodieIndex

close, tagLocation, updateLocation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieGlobalSimpleIndex

public HoodieGlobalSimpleIndex(HoodieWriteConfig config,
                               Option<BaseKeyGenerator> keyGeneratorOpt)

Method Detail

tagLocation

public HoodieData<HoodieRecord<T>> tagLocation(HoodieData<HoodieRecord<T>> records,
                                               HoodieEngineContext context,
                                               HoodieTable hoodieTable)

Description copied from class: HoodieIndex

Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).

Overrides:

tagLocation in class HoodieSimpleIndex<T extends HoodieRecordPayload<T>>

tagLocationInternal

protected HoodieData<HoodieRecord<T>> tagLocationInternal(HoodieData<HoodieRecord<T>> inputRecords,
                                                          HoodieEngineContext context,
                                                          HoodieTable hoodieTable)

Tags records location for incoming records.

Overrides:

tagLocationInternal in class HoodieSimpleIndex<T extends HoodieRecordPayload<T>>

Parameters:

inputRecords - HoodieData of incoming records

context - instance of HoodieEngineContext to use

hoodieTable - instance of HoodieTable to use

Returns:

HoodieData of records with record locations set

fetchAllRecordLocations

protected HoodiePairData<HoodieKey,HoodieRecordLocation> fetchAllRecordLocations(HoodieEngineContext context,
                                                                                 HoodieTable hoodieTable,
                                                                                 int parallelism)

Fetch record locations for passed in HoodieKeys.

Parameters:

context - instance of HoodieEngineContext to use

hoodieTable - instance of HoodieTable of interest

parallelism - parallelism to use

Returns:

HoodiePairData of HoodieKey and HoodieRecordLocation

getAllBaseFilesInTable

protected List<Pair<String,HoodieBaseFile>> getAllBaseFilesInTable(HoodieEngineContext context,
                                                                   HoodieTable hoodieTable)

Load all files for all partitions as pair data.

isGlobal

public boolean isGlobal()

Description copied from class: HoodieIndex

An index is `global` if HoodieKey to fileID mapping, does not depend on the `partitionPath`. Such an implementation is able to obtain the same mapping, for two hoodie keys with same `recordKey` but different `partitionPath`

Overrides:

isGlobal in class HoodieSimpleIndex<T extends HoodieRecordPayload<T>>

Returns:

whether or not, the index implementation is global in nature