org.apache.drill.exec.store.parquet

Class ParquetReaderUtility

java.lang.Object

org.apache.drill.exec.store.parquet.ParquetReaderUtility

public class ParquetReaderUtility
extends Object

Utility class where we can capture common logic between the two parquet readers

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParquetReaderUtility.DateCorruptionStatus For most recently created parquet files, we can determine if we have corrupted dates (see DRILL-4203) based on the file metadata.
static class	ParquetReaderUtility.NanoTimeUtils Utilities for converting from parquet INT96 binary (impala, hive timestamp) to date time value.

Field Summary

Fields

Modifier and Type	Field and Description
static long	CORRECT_CORRUPT_DATE_SHIFT All old parquet files (which haven't "is.date.correct=true" or "parquet-writer.version" properties in metadata) have a corrupt date shift: 4881176L days or 2 * 2440588L
static int	DATE_CORRUPTION_THRESHOLD The year 5000 (or 1106685 day from Unix epoch) is chosen as the threshold for auto-detecting date corruption.
static int	DRILL_WRITER_VERSION_STD_DATE_FORMAT Version 2 (and later) of the Drill Parquet writer uses the date format described in the Parquet spec.
static long	JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH Number of days between Julian day epoch (January 1, 4713 BC) and Unix day epoch (January 1, 1970).

Constructor Summary

Constructors

Constructor and Description
ParquetReaderUtility()

Method Summary

Methods

Modifier and Type	Method and Description
static int	autoCorrectCorruptedDate(int corruptedDate)
static void	checkDecimalTypeEnabled(OptionManager options)
static ParquetReaderUtility.DateCorruptionStatus	checkForCorruptDateValuesInStatistics(org.apache.parquet.hadoop.metadata.ParquetMetadata footer, List<SchemaPath> columns, boolean autoCorrectCorruptDates) Detect corrupt date values by looking at the min/max values in the metadata.
static void	correctBinaryInMetadataCache(Metadata.ParquetTableMetadataBase parquetTableMetadata) Checks assigns byte arrays to min/max values obtained from the deserialized string for BINARY.
static void	correctDatesInMetadataCache(Metadata.ParquetTableMetadataBase parquetTableMetadata)
static ParquetReaderUtility.DateCorruptionStatus	detectCorruptDates(org.apache.parquet.hadoop.metadata.ParquetMetadata footer, List<SchemaPath> columns, boolean autoCorrectCorruptDates) Check for corrupted dates in a parquet file.
static Map<String,org.apache.parquet.format.SchemaElement>	getColNameToSchemaElementMapping(org.apache.parquet.hadoop.metadata.ParquetMetadata footer)
static int	getIntFromLEBytes(byte[] input, int start)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH

public static final long JULIAN_DAY_NUMBER_FOR_UNIX_EPOCH

Number of days between Julian day epoch (January 1, 4713 BC) and Unix day epoch (January 1, 1970). The value of this constant is 2440588L.

See Also:

Constant Field Values

CORRECT_CORRUPT_DATE_SHIFT

public static final long CORRECT_CORRUPT_DATE_SHIFT

All old parquet files (which haven't "is.date.correct=true" or "parquet-writer.version" properties in metadata) have a corrupt date shift: 4881176L days or 2 * 2440588L

See Also:

Constant Field Values

DATE_CORRUPTION_THRESHOLD

public static final int DATE_CORRUPTION_THRESHOLD

The year 5000 (or 1106685 day from Unix epoch) is chosen as the threshold for auto-detecting date corruption. This balances two possible cases of bad auto-correction. External tools writing dates in the future will not be shifted unless they are past this threshold (and we cannot identify them as external files based on the metadata). On the other hand, historical dates written with Drill wouldn't risk being incorrectly shifted unless they were something like 10,000 years in the past.

DRILL_WRITER_VERSION_STD_DATE_FORMAT

public static final int DRILL_WRITER_VERSION_STD_DATE_FORMAT

Version 2 (and later) of the Drill Parquet writer uses the date format described in the Parquet spec. Prior versions had dates formatted with CORRECT_CORRUPT_DATE_SHIFT

See Also:

Constant Field Values

Constructor Detail

ParquetReaderUtility

public ParquetReaderUtility()

Method Detail

checkDecimalTypeEnabled

public static void checkDecimalTypeEnabled(OptionManager options)

getIntFromLEBytes

public static int getIntFromLEBytes(byte[] input,
                    int start)

getColNameToSchemaElementMapping

public static Map<String,org.apache.parquet.format.SchemaElement> getColNameToSchemaElementMapping(org.apache.parquet.hadoop.metadata.ParquetMetadata footer)

autoCorrectCorruptedDate

public static int autoCorrectCorruptedDate(int corruptedDate)

correctDatesInMetadataCache

public static void correctDatesInMetadataCache(Metadata.ParquetTableMetadataBase parquetTableMetadata)

correctBinaryInMetadataCache

public static void correctBinaryInMetadataCache(Metadata.ParquetTableMetadataBase parquetTableMetadata)

Checks assigns byte arrays to min/max values obtained from the deserialized string for BINARY.

Parameters:

parquetTableMetadata - table metadata that should be corrected

detectCorruptDates

public static ParquetReaderUtility.DateCorruptionStatus detectCorruptDates(org.apache.parquet.hadoop.metadata.ParquetMetadata footer,
                                                           List<SchemaPath> columns,
                                                           boolean autoCorrectCorruptDates)

Check for corrupted dates in a parquet file. See Drill-4203

checkForCorruptDateValuesInStatistics

public static ParquetReaderUtility.DateCorruptionStatus checkForCorruptDateValuesInStatistics(org.apache.parquet.hadoop.metadata.ParquetMetadata footer,
                                                                              List<SchemaPath> columns,
                                                                              boolean autoCorrectCorruptDates)

Detect corrupt date values by looking at the min/max values in the metadata. This should only be used when a file does not have enough metadata to determine if the data was written with an external tool or an older version of Drill (ParquetRecordWriter.WRITER_VERSION_PROPERTY < DRILL_WRITER_VERSION_STD_DATE_FORMAT) This method only checks the first Row Group, because Drill has only ever written a single Row Group per file.

Parameters:

footer -

columns -

autoCorrectCorruptDates - user setting to allow enabling/disabling of auto-correction of corrupt dates. There are some rare cases (storing dates thousands of years into the future, with tools other than Drill writing files) that would result in the date values being "corrected" into bad values.