org.apache.drill.exec.store.parquet.columnreaders

Class ParquetRecordReader

java.lang.Object

org.apache.drill.exec.store.AbstractRecordReader

org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader

All Implemented Interfaces:

AutoCloseable, RecordReader

public class ParquetRecordReader
extends AbstractRecordReader

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParquetRecordReader.Metric

Field Summary

Fields

Modifier and Type	Field and Description
int	bufferedReadSize
protected static List<SchemaPath>	DEFAULT_COLS_TO_READ
static byte[]	endBitMasks
boolean	enforceTotalSize
static int	PARQUET_PAGE_MAX_SIZE
ParquetReaderStats	parquetReaderStats
long	readQueueSize
static byte[]	startBitMasks
boolean	useAsyncColReader
boolean	useAsyncPageReader
boolean	useBufferedReader
boolean	useFadvise

Fields inherited from class org.apache.drill.exec.store.AbstractRecordReader

DEFAULT_TEXT_COLS_TO_READ

Fields inherited from interface org.apache.drill.exec.store.RecordReader

ALLOCATOR_INITIAL_RESERVATION, ALLOCATOR_MAX_RESERVATION

Constructor Summary

Constructors

Constructor and Description
ParquetRecordReader(FragmentContext fragmentContext, long batchSize, long numRecordsToRead, String path, int rowGroupIndex, org.apache.hadoop.fs.FileSystem fs, org.apache.parquet.hadoop.CodecFactory codecFactory, org.apache.parquet.hadoop.metadata.ParquetMetadata footer, List<SchemaPath> columns, ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)
ParquetRecordReader(FragmentContext fragmentContext, String path, int rowGroupIndex, org.apache.hadoop.fs.FileSystem fs, org.apache.parquet.hadoop.CodecFactory codecFactory, org.apache.parquet.hadoop.metadata.ParquetMetadata footer, List<SchemaPath> columns, ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)
ParquetRecordReader(FragmentContext fragmentContext, String path, int rowGroupIndex, long numRecordsToRead, org.apache.hadoop.fs.FileSystem fs, org.apache.parquet.hadoop.CodecFactory codecFactory, org.apache.parquet.hadoop.metadata.ParquetMetadata footer, List<SchemaPath> columns, ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)

Method Summary

Methods

Modifier and Type	Method and Description
void	allocate(Map<String,ValueVector> vectorMap)
void	close()
long	getBatchSize()
int	getBitWidthAllFixedFields()
org.apache.parquet.hadoop.CodecFactory	getCodecFactory()
ParquetReaderUtility.DateCorruptionStatus	getDateCorruptionStatus() Flag indicating if the old non-standard data format appears in this file, see DRILL-4203.
protected List<SchemaPath>	getDefaultColumnsToRead()
org.apache.hadoop.fs.FileSystem	getFileSystem()
FragmentContext	getFragmentContext()
org.apache.hadoop.fs.Path	getHadoopPath()
OperatorContext	getOperatorContext()
int	getRowGroupIndex()
protected DrillRuntimeException	handleException(String s, Exception e)
int	next() Read the next record batch from the file using the reader and read state created previously.
void	setup(OperatorContext operatorContext, OutputMutator output) Prepare the Parquet reader.

Methods inherited from class org.apache.drill.exec.store.AbstractRecordReader

getColumns, isSkipQuery, isStarQuery, setColumns, toString, transformColumns

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_COLS_TO_READ

protected static final List<SchemaPath> DEFAULT_COLS_TO_READ

PARQUET_PAGE_MAX_SIZE

public static final int PARQUET_PAGE_MAX_SIZE

See Also:

Constant Field Values

endBitMasks

public static final byte[] endBitMasks

startBitMasks

public static final byte[] startBitMasks

useAsyncColReader

public boolean useAsyncColReader

useAsyncPageReader

public boolean useAsyncPageReader

useBufferedReader

public boolean useBufferedReader

bufferedReadSize

public int bufferedReadSize

useFadvise

public boolean useFadvise

enforceTotalSize

public boolean enforceTotalSize

readQueueSize

public long readQueueSize

parquetReaderStats

public ParquetReaderStats parquetReaderStats

Constructor Detail

ParquetRecordReader

public ParquetRecordReader(FragmentContext fragmentContext,
                   String path,
                   int rowGroupIndex,
                   long numRecordsToRead,
                   org.apache.hadoop.fs.FileSystem fs,
                   org.apache.parquet.hadoop.CodecFactory codecFactory,
                   org.apache.parquet.hadoop.metadata.ParquetMetadata footer,
                   List<SchemaPath> columns,
                   ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)
                    throws ExecutionSetupException

Throws:

ExecutionSetupException

ParquetRecordReader

public ParquetRecordReader(FragmentContext fragmentContext,
                   String path,
                   int rowGroupIndex,
                   org.apache.hadoop.fs.FileSystem fs,
                   org.apache.parquet.hadoop.CodecFactory codecFactory,
                   org.apache.parquet.hadoop.metadata.ParquetMetadata footer,
                   List<SchemaPath> columns,
                   ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)
                    throws ExecutionSetupException

Throws:

ExecutionSetupException

ParquetRecordReader

public ParquetRecordReader(FragmentContext fragmentContext,
                   long batchSize,
                   long numRecordsToRead,
                   String path,
                   int rowGroupIndex,
                   org.apache.hadoop.fs.FileSystem fs,
                   org.apache.parquet.hadoop.CodecFactory codecFactory,
                   org.apache.parquet.hadoop.metadata.ParquetMetadata footer,
                   List<SchemaPath> columns,
                   ParquetReaderUtility.DateCorruptionStatus dateCorruptionStatus)
                    throws ExecutionSetupException

Throws:

ExecutionSetupException

Method Detail

getDateCorruptionStatus

public ParquetReaderUtility.DateCorruptionStatus getDateCorruptionStatus()

Flag indicating if the old non-standard data format appears in this file, see DRILL-4203.

Returns:

true if the dates are corrupted and need to be corrected

getCodecFactory

public org.apache.parquet.hadoop.CodecFactory getCodecFactory()

getHadoopPath

public org.apache.hadoop.fs.Path getHadoopPath()

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

getRowGroupIndex

public int getRowGroupIndex()

getBitWidthAllFixedFields

public int getBitWidthAllFixedFields()

getBatchSize

public long getBatchSize()

getOperatorContext

public OperatorContext getOperatorContext()

getFragmentContext

public FragmentContext getFragmentContext()

setup

public void setup(OperatorContext operatorContext,
         OutputMutator output)
           throws ExecutionSetupException

Prepare the Parquet reader. First determine the set of columns to read (the schema for this read.) Then, create a state object to track the read across calls to the reader next() method. Finally, create one of three readers to read batches depending on whether this scan is for only fixed-width fields, contains at least one variable-width field, or is a "mock" scan consisting only of null fields (fields in the SELECT clause but not in the Parquet file.)

Parameters:

operatorContext - operator context for the reader

output - The place where output for a particular scan should be written. The record reader is responsible for mutating the set of schema values for that particular record.

Throws:

ExecutionSetupException

handleException

protected DrillRuntimeException handleException(String s,
                                    Exception e)

allocate

public void allocate(Map<String,ValueVector> vectorMap)
              throws OutOfMemoryException

Specified by:

allocate in interface RecordReader

Overrides:

allocate in class AbstractRecordReader

Throws:

OutOfMemoryException

next

public int next()

Read the next record batch from the file using the reader and read state created previously.

Returns:

The number of additional records added to the output.

close

public void close()

getDefaultColumnsToRead

protected List<SchemaPath> getDefaultColumnsToRead()

Overrides:

getDefaultColumnsToRead in class AbstractRecordReader