org.apache.hudi.common.util

Class ParquetUtils

java.lang.Object

org.apache.hudi.common.util.FileFormatUtils

org.apache.hudi.common.util.ParquetUtils

public class ParquetUtils
extends FileFormatUtils

Utility functions involving with parquet.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.util.FileFormatUtils

FileFormatUtils.HoodieKeyIterator

Constructor Summary

Constructors

Constructor and Description
ParquetUtils()

Method Summary

Modifier and Type	Method and Description
ClosableIterator<Pair<HoodieKey,Long>>	fetchRecordKeysWithPositions(HoodieStorage storage, StoragePath filePath) Fetch HoodieKeys with row positions from the given parquet file.
ClosableIterator<Pair<HoodieKey,Long>>	fetchRecordKeysWithPositions(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt, Option<String> partitionPath) Fetch HoodieKeys with row positions from the given parquet file.
Set<Pair<String,Long>>	filterRowKeys(HoodieStorage storage, StoragePath filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given parquet file.
static org.apache.parquet.hadoop.metadata.CompressionCodecName	getCompressionCodecName(String codecName)
HoodieFileFormat	getFormat()
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath)
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt, Option<String> partitionPath) Returns a closable iterator for reading the given parquet file.
long	getRowCount(HoodieStorage storage, StoragePath filePath) Returns the number of records in the parquet file.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath) NOTE: This literally reads the entire file contents, thus should be used with caution.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath, org.apache.avro.Schema schema)
org.apache.avro.Schema	readAvroSchema(HoodieStorage storage, StoragePath filePath)
List<HoodieColumnRangeMetadata<Comparable>>	readColumnStatsFromMetadata(HoodieStorage storage, StoragePath filePath, List<String> columnList)
Map<String,String>	readFooter(HoodieStorage storage, boolean required, StoragePath filePath, String... footerNames)
static org.apache.parquet.hadoop.metadata.ParquetMetadata	readMetadata(HoodieStorage storage, StoragePath parquetFilePath)
org.apache.parquet.schema.MessageType	readSchema(HoodieStorage storage, StoragePath parquetFilePath) Get the schema of the given parquet file.
ByteArrayOutputStream	serializeRecordsToLogBlock(HoodieStorage storage, List<HoodieRecord> records, org.apache.avro.Schema writerSchema, org.apache.avro.Schema readerSchema, String keyFieldName, Map<String,String> paramsMap)
void	writeMetaFile(HoodieStorage storage, StoragePath filePath, Properties props)

Methods inherited from class org.apache.hudi.common.util.FileFormatUtils

getColumnRangeInPartition, readBloomFilterFromMetadata, readMinMaxRecordKeys, readRowKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetUtils

public ParquetUtils()

Method Detail

filterRowKeys

public Set<Pair<String,Long>> filterRowKeys(HoodieStorage storage,
                                            StoragePath filePath,
                                            Set<String> filter)

Read the rowKey list matching the given filter, from the given parquet file. If the filter is empty, then this will return all the rowkeys and corresponding positions.

Specified by:

filterRowKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

filter - record keys filter

Returns:

Set Set of pairs of row key and position matching candidateRecordKeys

readMetadata

public static org.apache.parquet.hadoop.metadata.ParquetMetadata readMetadata(HoodieStorage storage,
                                                                              StoragePath parquetFilePath)

getCompressionCodecName

public static org.apache.parquet.hadoop.metadata.CompressionCodecName getCompressionCodecName(String codecName)

Parameters:

codecName - codec name in String.

Returns:

CompressionCodecName Enum.

fetchRecordKeysWithPositions

public ClosableIterator<Pair<HoodieKey,Long>> fetchRecordKeysWithPositions(HoodieStorage storage,
                                                                           StoragePath filePath)

Fetch HoodieKeys with row positions from the given parquet file.

Specified by:

fetchRecordKeysWithPositions in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

Returns:

List of pairs of HoodieKey and row position fetched from the parquet file

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath)

Specified by:

getHoodieKeyIterator in class FileFormatUtils

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath,
                                                        Option<BaseKeyGenerator> keyGeneratorOpt,
                                                        Option<String> partitionPath)

Returns a closable iterator for reading the given parquet file.

Specified by:

getHoodieKeyIterator in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path

keyGeneratorOpt - instance of KeyGenerator

partitionPath - optional partition path for the file, if provided only the record key is read from the file

Returns:

ClosableIterator of HoodieKeys for reading the parquet file

fetchRecordKeysWithPositions

public ClosableIterator<Pair<HoodieKey,Long>> fetchRecordKeysWithPositions(HoodieStorage storage,
                                                                           StoragePath filePath,
                                                                           Option<BaseKeyGenerator> keyGeneratorOpt,
                                                                           Option<String> partitionPath)

Fetch HoodieKeys with row positions from the given parquet file.

Specified by:

fetchRecordKeysWithPositions in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

keyGeneratorOpt - instance of KeyGenerator.

partitionPath - optional partition path for the file, if provided only the record key is read from the file

Returns:

List of pairs of HoodieKey and row position fetched from the parquet file

readSchema

public org.apache.parquet.schema.MessageType readSchema(HoodieStorage storage,
                                                        StoragePath parquetFilePath)

Get the schema of the given parquet file.

readFooter

public Map<String,String> readFooter(HoodieStorage storage,
                                     boolean required,
                                     StoragePath filePath,
                                     String... footerNames)

Specified by:

readFooter in class FileFormatUtils

readAvroSchema

public org.apache.avro.Schema readAvroSchema(HoodieStorage storage,
                                             StoragePath filePath)

Specified by:

readAvroSchema in class FileFormatUtils

readColumnStatsFromMetadata

public List<HoodieColumnRangeMetadata<Comparable>> readColumnStatsFromMetadata(HoodieStorage storage,
                                                                               StoragePath filePath,
                                                                               List<String> columnList)

Specified by:

readColumnStatsFromMetadata in class FileFormatUtils

getFormat

public HoodieFileFormat getFormat()

Specified by:

getFormat in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath,
                                                                   org.apache.avro.Schema schema)

Specified by:

readAvroRecords in class FileFormatUtils

getRowCount

public long getRowCount(HoodieStorage storage,
                        StoragePath filePath)

Returns the number of records in the parquet file.

Specified by:

getRowCount in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - path of the file

writeMetaFile

public void writeMetaFile(HoodieStorage storage,
                          StoragePath filePath,
                          Properties props)
                   throws IOException

Specified by:

writeMetaFile in class FileFormatUtils

Throws:

IOException

serializeRecordsToLogBlock

public ByteArrayOutputStream serializeRecordsToLogBlock(HoodieStorage storage,
                                                        List<HoodieRecord> records,
                                                        org.apache.avro.Schema writerSchema,
                                                        org.apache.avro.Schema readerSchema,
                                                        String keyFieldName,
                                                        Map<String,String> paramsMap)
                                                 throws IOException

Specified by:

serializeRecordsToLogBlock in class FileFormatUtils

Throws:

IOException