org.apache.drill.exec.store.parquet

Class Metadata

java.lang.Object

org.apache.drill.exec.store.parquet.Metadata

public class Metadata
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Metadata.ColumnMetadata
static class	Metadata.ColumnMetadata_v1 A struct that contains the metadata for a column in a parquet file
static class	Metadata.ColumnMetadata_v2 A struct that contains the metadata for a column in a parquet file
static class	Metadata.ColumnMetadata_v3 A struct that contains the metadata for a column in a parquet file
static class	Metadata.ColumnTypeMetadata_v2
static class	Metadata.ColumnTypeMetadata_v3
static class	Metadata.ParquetFileMetadata
static class	Metadata.ParquetFileMetadata_v1 Struct which contains the metadata for a single parquet file
static class	Metadata.ParquetFileMetadata_v2 Struct which contains the metadata for a single parquet file
static class	Metadata.ParquetFileMetadata_v3 Struct which contains the metadata for a single parquet file
static class	Metadata.ParquetTableMetadata_v1
static class	Metadata.ParquetTableMetadata_v2 Struct which contains the metadata for an entire parquet directory structure
static class	Metadata.ParquetTableMetadata_v3
static class	Metadata.ParquetTableMetadataBase Basic class for parquet metadata.
static class	Metadata.ParquetTableMetadataDirs
static class	Metadata.RowGroupMetadata
static class	Metadata.RowGroupMetadata_v1 A struct that contains the metadata for a parquet row group
static class	Metadata.RowGroupMetadata_v2 A struct that contains the metadata for a parquet row group
static class	Metadata.RowGroupMetadata_v3 A struct that contains the metadata for a parquet row group

Field Summary

Fields

Modifier and Type	Field and Description
static String	METADATA_DIRECTORIES_FILENAME
static String	METADATA_FILENAME
static String[]	OLD_METADATA_FILENAMES

Method Summary

Methods

Modifier and Type	Method and Description
static void	createMeta(org.apache.hadoop.fs.FileSystem fs, String path, ParquetFormatConfig formatConfig) Create the parquet metadata file for the directory at the given path, and for any subdirectories
static Metadata.ParquetTableMetadata_v3	getParquetTableMetadata(org.apache.hadoop.fs.FileSystem fs, List<org.apache.hadoop.fs.FileStatus> fileStatuses, ParquetFormatConfig formatConfig) Get the parquet metadata for a list of parquet files
static Metadata.ParquetTableMetadata_v3	getParquetTableMetadata(org.apache.hadoop.fs.FileSystem fs, String path, ParquetFormatConfig formatConfig) Get the parquet metadata for the parquet files in the given directory, including those in subdirectories
static Metadata.ParquetTableMetadataBase	readBlockMeta(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, MetadataContext metaContext, ParquetFormatConfig formatConfig) Get the parquet metadata for the table by reading the metadata file
static Metadata.ParquetTableMetadataDirs	readMetadataDirs(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, MetadataContext metaContext, ParquetFormatConfig formatConfig) Get the parquet metadata for all subdirectories by reading the metadata file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OLD_METADATA_FILENAMES

public static final String[] OLD_METADATA_FILENAMES

METADATA_FILENAME

public static final String METADATA_FILENAME

See Also:

Constant Field Values

METADATA_DIRECTORIES_FILENAME

public static final String METADATA_DIRECTORIES_FILENAME

See Also:

Constant Field Values

Method Detail

createMeta

public static void createMeta(org.apache.hadoop.fs.FileSystem fs,
              String path,
              ParquetFormatConfig formatConfig)
                       throws IOException

Create the parquet metadata file for the directory at the given path, and for any subdirectories

Parameters:

fs -

path -

Throws:

IOException

getParquetTableMetadata

public static Metadata.ParquetTableMetadata_v3 getParquetTableMetadata(org.apache.hadoop.fs.FileSystem fs,
                                                       String path,
                                                       ParquetFormatConfig formatConfig)
                                                                throws IOException

Get the parquet metadata for the parquet files in the given directory, including those in subdirectories

Parameters:

fs -

path -

Returns:

Throws:

IOException

getParquetTableMetadata

public static Metadata.ParquetTableMetadata_v3 getParquetTableMetadata(org.apache.hadoop.fs.FileSystem fs,
                                                       List<org.apache.hadoop.fs.FileStatus> fileStatuses,
                                                       ParquetFormatConfig formatConfig)
                                                                throws IOException

Get the parquet metadata for a list of parquet files

Parameters:

fs -

fileStatuses -

Returns:

Throws:

IOException

readBlockMeta

@Nullable
public static Metadata.ParquetTableMetadataBase readBlockMeta(org.apache.hadoop.fs.FileSystem fs,
                                                       org.apache.hadoop.fs.Path path,
                                                       MetadataContext metaContext,
                                                       ParquetFormatConfig formatConfig)

Get the parquet metadata for the table by reading the metadata file

Parameters:

fs - current file system

path - The path to the metadata file, located in the directory that contains the parquet files

metaContext - metadata context

formatConfig - parquet format plugin configs

Returns:

parquet table metadata. Null if metadata cache is missing, unsupported or corrupted

readMetadataDirs

@Nullable
public static Metadata.ParquetTableMetadataDirs readMetadataDirs(org.apache.hadoop.fs.FileSystem fs,
                                                          org.apache.hadoop.fs.Path path,
                                                          MetadataContext metaContext,
                                                          ParquetFormatConfig formatConfig)

Get the parquet metadata for all subdirectories by reading the metadata file

Parameters:

fs - current file system

path - The path to the metadata file, located in the directory that contains the parquet files

metaContext - metadata context

formatConfig - parquet format plugin configs

Returns:

parquet metadata for a directory. Null if metadata cache is missing, unsupported or corrupted