org.apache.hudi.source

Class FileIndex

java.lang.Object

org.apache.hudi.source.FileIndex

public class FileIndex
extends Object

A file index which supports listing files efficiently through metadata table.

It caches the partition paths to avoid redundant look up.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileIndex.Builder Builder for FileIndex.

Method Summary

Modifier and Type	Method and Description
static FileIndex.Builder	builder() Returns the builder.
org.apache.hadoop.fs.FileStatus[]	getFilesInPartitions() Returns all the file statuses under the table base path.
List<String>	getOrBuildPartitionPaths() Returns all the relative partition paths.
List<Map<String,String>>	getPartitions(List<String> partitionKeys, String defaultParName, boolean hivePartition) Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value.
static HoodieMetadataConfig	metadataConfig(org.apache.flink.configuration.Configuration conf)
void	reset() Reset the state of the file index.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static FileIndex.Builder builder()

Returns the builder.

getPartitions

public List<Map<String,String>> getPartitions(List<String> partitionKeys,
                                              String defaultParName,
                                              boolean hivePartition)

Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value. For example, say there is a file path with partition keys [key1, key2, key3]:

   -- file:/// ... key1=val1/key2=val2/key3=val3
   -- file:/// ... key1=val4/key2=val5/key3=val6
 

The return list should be [{key1:val1, key2:val2, key3:val3}, {key1:val4, key2:val5, key3:val6}].

Parameters:

partitionKeys - The partition key list

defaultParName - The default partition name for nulls

hivePartition - Whether the partition path is in Hive style

getFilesInPartitions

public org.apache.hadoop.fs.FileStatus[] getFilesInPartitions()

Returns all the file statuses under the table base path.

reset

@VisibleForTesting
public void reset()

Reset the state of the file index.

getOrBuildPartitionPaths

public List<String> getOrBuildPartitionPaths()

Returns all the relative partition paths.

The partition paths are cached once invoked.

metadataConfig

public static HoodieMetadataConfig metadataConfig(org.apache.flink.configuration.Configuration conf)