org.apache.hudi.source

Class FileIndex

java.lang.Object

org.apache.hudi.source.FileIndex

public class FileIndex
extends Object

A file index which supports listing files efficiently through metadata table.

It caches the partition paths to avoid redundant look up.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FileStatus[]	getFilesInPartitions() Returns all the file statuses under the table base path.
List<String>	getOrBuildPartitionPaths() Returns all the relative partition paths.
List<Map<String,String>>	getPartitions(List<String> partitionKeys, String defaultParName, boolean hivePartition) Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value.
static FileIndex	instance(org.apache.hadoop.fs.Path path, org.apache.flink.configuration.Configuration conf, org.apache.flink.table.types.logical.RowType rowType)
void	reset() Reset the state of the file index.
void	setFilters(List<org.apache.flink.table.expressions.ResolvedExpression> filters) Sets up pushed down filters.
void	setPartitionPaths(Set<String> partitionPaths) Sets up explicit partition paths for pruning.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

instance

public static FileIndex instance(org.apache.hadoop.fs.Path path,
                                 org.apache.flink.configuration.Configuration conf,
                                 org.apache.flink.table.types.logical.RowType rowType)

getPartitions

public List<Map<String,String>> getPartitions(List<String> partitionKeys,
                                              String defaultParName,
                                              boolean hivePartition)

Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value. For example, say there is a file path with partition keys [key1, key2, key3]:

   -- file:/// ... key1=val1/key2=val2/key3=val3
   -- file:/// ... key1=val4/key2=val5/key3=val6
 

The return list should be [{key1:val1, key2:val2, key3:val3}, {key1:val4, key2:val5, key3:val6}].

Parameters:

partitionKeys - The partition key list

defaultParName - The default partition name for nulls

hivePartition - Whether the partition path is in Hive style

getFilesInPartitions

public org.apache.hadoop.fs.FileStatus[] getFilesInPartitions()

Returns all the file statuses under the table base path.

reset

@VisibleForTesting
public void reset()

Reset the state of the file index.

setPartitionPaths

public void setPartitionPaths(@Nullable
                              Set<String> partitionPaths)

Sets up explicit partition paths for pruning.

setFilters

public void setFilters(List<org.apache.flink.table.expressions.ResolvedExpression> filters)

Sets up pushed down filters.

getOrBuildPartitionPaths

public List<String> getOrBuildPartitionPaths()

Returns all the relative partition paths.

The partition paths are cached once invoked.