org.apache.hudi.table.format

Class FilePathUtils

java.lang.Object

org.apache.hudi.table.format.FilePathUtils

public class FilePathUtils
extends Object

Reference the Flink PartitionPathUtils but supports simple partition path besides the Hive style.

Constructor Summary

Constructors

Constructor and Description
FilePathUtils()

Method Summary

Modifier and Type	Method and Description
static String[]	extractHivePartitionFields(org.apache.flink.configuration.Configuration conf) Extracts the hive sync partition fields with given configuration.
static String[]	extractPartitionKeys(org.apache.flink.configuration.Configuration conf) Extracts the partition keys with given configuration.
static LinkedHashMap<String,String>	extractPartitionKeyValues(org.apache.hadoop.fs.Path currPath, boolean hivePartition, String[] partitionKeys) Generates partition key value mapping from path.
static String	generatePartitionPath(LinkedHashMap<String,String> partitionKVs, boolean hivePartition, boolean sepSuffix) Make partition path from partition spec.
static org.apache.hadoop.fs.FileStatus[]	getFileStatusRecursively(org.apache.hadoop.fs.Path path, int expectLevel, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileStatus[]	getFileStatusRecursively(org.apache.hadoop.fs.Path path, int expectLevel, org.apache.hadoop.fs.FileSystem fs)
static List<Map<String,String>>	getPartitions(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf, List<String> partitionKeys, String defaultParName, boolean hivePartition) Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value.
static org.apache.hadoop.fs.Path[]	getReadPaths(org.apache.hadoop.fs.Path path, org.apache.flink.configuration.Configuration conf, org.apache.hadoop.conf.Configuration hadoopConf, List<String> partitionKeys) Returns all the file paths that is the parents of the data files.
static boolean	isHiveStylePartitioning(String path)
static org.apache.hadoop.fs.Path[]	partitionPath2ReadPath(org.apache.hadoop.fs.Path path, List<String> partitionKeys, List<Map<String,String>> partitionPaths, boolean hivePartition) Transforms the given partition key value mapping to read paths.
static List<org.apache.flink.api.java.tuple.Tuple2<LinkedHashMap<String,String>,org.apache.hadoop.fs.Path>>	searchPartKeyValueAndPaths(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean hivePartition, String[] partitionKeys) Search all partitions in this path.
static org.apache.flink.core.fs.Path	toFlinkPath(org.apache.hadoop.fs.Path path) Transforms the Hadoop path to Flink path.
static org.apache.flink.core.fs.Path[]	toFlinkPaths(org.apache.hadoop.fs.Path[] paths) Transforms the array of Hadoop paths to Flink paths.
static Set<String>	toRelativePartitionPaths(List<String> partitionKeys, List<Map<String,String>> partitionPaths, boolean hivePartition) Transforms the given partition key value mapping to relative partition paths.
static String	unescapePathName(String path)
static LinkedHashMap<String,String>	validateAndReorderPartitions(Map<String,String> partitionKVs, List<String> partitionKeys) Reorder the partition key value mapping based on the given partition keys sequence.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FilePathUtils

public FilePathUtils()

Method Detail

generatePartitionPath

public static String generatePartitionPath(LinkedHashMap<String,String> partitionKVs,
                                           boolean hivePartition,
                                           boolean sepSuffix)

Make partition path from partition spec.

Parameters:

partitionKVs - The partition key value mapping

hivePartition - Whether the partition path is with Hive style, e.g. {partition key} = {partition value}

sepSuffix - Whether to append the path separator as suffix

Returns:

an escaped, valid partition name

extractPartitionKeyValues

public static LinkedHashMap<String,String> extractPartitionKeyValues(org.apache.hadoop.fs.Path currPath,
                                                                     boolean hivePartition,
                                                                     String[] partitionKeys)

Generates partition key value mapping from path.

Parameters:

currPath - Partition file path

hivePartition - Whether the partition path is with Hive style

partitionKeys - Partition keys

Returns:

Sequential partition specs.

unescapePathName

public static String unescapePathName(String path)

searchPartKeyValueAndPaths

public static List<org.apache.flink.api.java.tuple.Tuple2<LinkedHashMap<String,String>,org.apache.hadoop.fs.Path>> searchPartKeyValueAndPaths(org.apache.hadoop.fs.FileSystem fs,
                                                                                                                                              org.apache.hadoop.fs.Path path,
                                                                                                                                              boolean hivePartition,
                                                                                                                                              String[] partitionKeys)

Search all partitions in this path.

Parameters:

fs - File system

path - Search path

hivePartition - Whether the partition path is with Hive style

partitionKeys - Partition keys

Returns:

all partition key value mapping in sequence of the given path

getFileStatusRecursively

public static org.apache.hadoop.fs.FileStatus[] getFileStatusRecursively(org.apache.hadoop.fs.Path path,
                                                                         int expectLevel,
                                                                         org.apache.hadoop.conf.Configuration conf)

getFileStatusRecursively

public static org.apache.hadoop.fs.FileStatus[] getFileStatusRecursively(org.apache.hadoop.fs.Path path,
                                                                         int expectLevel,
                                                                         org.apache.hadoop.fs.FileSystem fs)

getPartitions

public static List<Map<String,String>> getPartitions(org.apache.hadoop.fs.Path path,
                                                     org.apache.hadoop.conf.Configuration hadoopConf,
                                                     List<String> partitionKeys,
                                                     String defaultParName,
                                                     boolean hivePartition)

Returns the partition path key and values as a list of map, each map item in the list is a mapping of the partition key name to its actual partition value. For example, say there is a file path with partition keys [key1, key2, key3]:

   -- file:/// ... key1=val1/key2=val2/key3=val3
   -- file:/// ... key1=val4/key2=val5/key3=val6
 

The return list should be [{key1:val1, key2:val2, key3:val3}, {key1:val4, key2:val5, key3:val6}].

Parameters:

path - The base path

hadoopConf - The hadoop configuration

partitionKeys - The partition key list

defaultParName - The default partition name for nulls

hivePartition - Whether the partition path is in Hive style

validateAndReorderPartitions

public static LinkedHashMap<String,String> validateAndReorderPartitions(Map<String,String> partitionKVs,
                                                                        List<String> partitionKeys)

Reorder the partition key value mapping based on the given partition keys sequence.

Parameters:

partitionKVs - The partition key and value mapping

partitionKeys - The partition key list

getReadPaths

public static org.apache.hadoop.fs.Path[] getReadPaths(org.apache.hadoop.fs.Path path,
                                                       org.apache.flink.configuration.Configuration conf,
                                                       org.apache.hadoop.conf.Configuration hadoopConf,
                                                       List<String> partitionKeys)

Returns all the file paths that is the parents of the data files.

Parameters:

path - The base path

conf - The Flink configuration

hadoopConf - The hadoop configuration

partitionKeys - The partition key list

partitionPath2ReadPath

public static org.apache.hadoop.fs.Path[] partitionPath2ReadPath(org.apache.hadoop.fs.Path path,
                                                                 List<String> partitionKeys,
                                                                 List<Map<String,String>> partitionPaths,
                                                                 boolean hivePartition)

Transforms the given partition key value mapping to read paths.

Parameters:

path - The base path

partitionKeys - The partition key list

partitionPaths - The partition key value mapping

hivePartition - Whether the partition path is in Hive style

See Also:

getReadPaths(org.apache.hadoop.fs.Path, org.apache.flink.configuration.Configuration, org.apache.hadoop.conf.Configuration, java.util.List<java.lang.String>)

toRelativePartitionPaths

public static Set<String> toRelativePartitionPaths(List<String> partitionKeys,
                                                   List<Map<String,String>> partitionPaths,
                                                   boolean hivePartition)

Transforms the given partition key value mapping to relative partition paths.

Parameters:

partitionKeys - The partition key list

partitionPaths - The partition key value mapping

hivePartition - Whether the partition path is in Hive style

See Also:

getReadPaths(org.apache.hadoop.fs.Path, org.apache.flink.configuration.Configuration, org.apache.hadoop.conf.Configuration, java.util.List<java.lang.String>)

toFlinkPaths

public static org.apache.flink.core.fs.Path[] toFlinkPaths(org.apache.hadoop.fs.Path[] paths)

Transforms the array of Hadoop paths to Flink paths.

toFlinkPath

public static org.apache.flink.core.fs.Path toFlinkPath(org.apache.hadoop.fs.Path path)

Transforms the Hadoop path to Flink path.

extractPartitionKeys

public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf)

Extracts the partition keys with given configuration.

Parameters:

conf - The flink configuration

Returns:

array of the partition fields

extractHivePartitionFields

public static String[] extractHivePartitionFields(org.apache.flink.configuration.Configuration conf)

Extracts the hive sync partition fields with given configuration.

Parameters:

conf - The flink configuration

Returns:

array of the hive partition fields

isHiveStylePartitioning

public static boolean isHiveStylePartitioning(String path)