org.apache.drill.exec.store.parquet

Class ParquetGroupScan

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractGroupScan

org.apache.drill.exec.physical.base.AbstractFileGroupScan

org.apache.drill.exec.store.parquet.ParquetGroupScan

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, FileGroupScan, FragmentLeaf, GroupScan, HasAffinity, Leaf, PhysicalOperator, Scan

public class ParquetGroupScan
extends AbstractFileGroupScan

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParquetGroupScan.RowGroupInfo

Field Summary

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation

Fields inherited from interface org.apache.drill.exec.physical.base.GroupScan

ALL_COLUMNS, NO_COLUMN_STATS

Constructor Summary

Constructors

Constructor and Description
ParquetGroupScan(String userName, FileSelection selection, ParquetFormatPlugin formatPlugin, List<SchemaPath> columns)
ParquetGroupScan(String userName, FileSelection selection, ParquetFormatPlugin formatPlugin, List<SchemaPath> columns, LogicalExpression filter)
ParquetGroupScan(String userName, List<ReadEntryWithPath> entries, StoragePluginConfig storageConfig, FormatPluginConfig formatConfig, StoragePluginRegistry engineRegistry, List<SchemaPath> columns, String selectionRoot, String cacheFileRoot, LogicalExpression filter)

Method Summary

Methods

Modifier and Type	Method and Description
void	applyAssignments(List<CoordinationProtos.DrillbitEndpoint> incomingEndpoints)
GroupScan	applyFilter(LogicalExpression filterExpr, UdfUtilities udfUtilities, FunctionImplementationRegistry functionImplementationRegistry, OptionManager optionManager)
GroupScan	applyLimit(long maxRecords) By default, return null to indicate rowcount based prune is not supported.
boolean	canPushdownProjects(List<SchemaPath> columns) GroupScan should check the list of columns, and see if it could support all the columns in the list.
ParquetGroupScan	clone(FileSelection selection)
ParquetGroupScan	clone(FileSelection selection, long maxRecords)
GroupScan	clone(List<SchemaPath> columns) Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .
List<SchemaPath>	getColumns()
long	getColumnValueCount(SchemaPath column) Return column value count for the specified column.
String	getDigest() Returns a signature of the GroupScan which should usually be composed of all its attributes which could describe it uniquely.
StoragePluginConfig	getEngineConfig()
List<ReadEntryWithPath>	getEntries()
Collection<String>	getFiles() Returns a collection of file names associated with this GroupScan.
Set<String>	getFileSet()
LogicalExpression	getFilter()
ParquetFormatConfig	getFormatConfig()
int	getMaxParallelizationWidth()
PhysicalOperator	getNewWithChildren(List<PhysicalOperator> children) Regenerate with this node with a new set of children.
List<EndpointAffinity>	getOperatorAffinity() Calculates the affinity each endpoint has for this scan, by adding up the affinity each endpoint has for each rowGroup
List<SchemaPath>	getPartitionColumns() Returns a list of columns that can be used for partition pruning
ScanStats	getScanStats()
String	getSelectionRoot()
ParquetRowGroupScan	getSpecificScan(int minorFragmentId)
static TypeProtos.MajorType	getType(org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName type, org.apache.parquet.schema.OriginalType originalType, int scale, int precision) Builds major type using given OriginalType originalType or PrimitiveTypeName type.
TypeProtos.MajorType	getTypeForColumn(SchemaPath schemaPath)
boolean	hasFiles() Return true if this GroupScan can return its selection as a list of file names (retrieved by getFiles()).
void	modifyFileSelection(FileSelection selection)
void	populatePruningVector(ValueVector v, int index, SchemaPath column, String file)
void	setFilter(LogicalExpression filter)
boolean	supportsLimitPushdown() Default is not to support limit pushdown.
String	toString()

Methods inherited from class org.apache.drill.exec.physical.base.AbstractFileGroupScan

supportsPartitionFilterPushdown

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScan

accept, enforceWidth, getDistributionAffinity, getInitialAllocation, getMaxAllocation, getMinParallelizationWidth, getOperatorType, getScanStats, isExecutable, iterator

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.GroupScan

enforceWidth, getMinParallelizationWidth, getScanStats

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getOperatorType, getSVMode, getUserName, isBufferedOperator, isExecutable, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

iterator

Methods inherited from interface org.apache.drill.exec.physical.base.HasAffinity

getDistributionAffinity

Constructor Detail

ParquetGroupScan

public ParquetGroupScan(String userName,
                List<ReadEntryWithPath> entries,
                StoragePluginConfig storageConfig,
                FormatPluginConfig formatConfig,
                StoragePluginRegistry engineRegistry,
                List<SchemaPath> columns,
                String selectionRoot,
                String cacheFileRoot,
                LogicalExpression filter)
                 throws IOException,
                        ExecutionSetupException

Throws:

IOException

ExecutionSetupException

ParquetGroupScan

public ParquetGroupScan(String userName,
                FileSelection selection,
                ParquetFormatPlugin formatPlugin,
                List<SchemaPath> columns)
                 throws IOException

Throws:

IOException

ParquetGroupScan

public ParquetGroupScan(String userName,
                FileSelection selection,
                ParquetFormatPlugin formatPlugin,
                List<SchemaPath> columns,
                LogicalExpression filter)
                 throws IOException

Throws:

IOException

Method Detail

getEntries

public List<ReadEntryWithPath> getEntries()

getFormatConfig

public ParquetFormatConfig getFormatConfig()

getEngineConfig

public StoragePluginConfig getEngineConfig()

getSelectionRoot

public String getSelectionRoot()

getFileSet

public Set<String> getFileSet()

getFilter

public LogicalExpression getFilter()

setFilter

public void setFilter(LogicalExpression filter)

hasFiles

public boolean hasFiles()

Description copied from interface: GroupScan

Return true if this GroupScan can return its selection as a list of file names (retrieved by getFiles()).

Specified by:

hasFiles in interface GroupScan

Overrides:

hasFiles in class AbstractGroupScan

getFiles

public Collection<String> getFiles()

Description copied from interface: GroupScan

Returns a collection of file names associated with this GroupScan. This should be called after checking hasFiles(). If this GroupScan cannot provide file names, it returns null.

Specified by:

getFiles in interface GroupScan

Overrides:

getFiles in class AbstractGroupScan

getType

public static TypeProtos.MajorType getType(org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName type,
                           org.apache.parquet.schema.OriginalType originalType,
                           int scale,
                           int precision)

Builds major type using given OriginalType originalType or PrimitiveTypeName type. For DECIMAL will be returned major type with scale and precision.

Parameters:

type - parquet primitive type

originalType - parquet original type

scale - type scale (used for DECIMAL type)

precision - type precision (used for DECIMAL type)

Returns:

major type

modifyFileSelection

public void modifyFileSelection(FileSelection selection)

Specified by:

modifyFileSelection in interface FileGroupScan

Overrides:

modifyFileSelection in class AbstractFileGroupScan

getTypeForColumn

public TypeProtos.MajorType getTypeForColumn(SchemaPath schemaPath)

populatePruningVector

public void populatePruningVector(ValueVector v,
                         int index,
                         SchemaPath column,
                         String file)

getOperatorAffinity

public List<EndpointAffinity> getOperatorAffinity()

Calculates the affinity each endpoint has for this scan, by adding up the affinity each endpoint has for each rowGroup

Specified by:

getOperatorAffinity in interface HasAffinity

Overrides:

getOperatorAffinity in class AbstractGroupScan

Returns:

a list of EndpointAffinity objects

applyAssignments

public void applyAssignments(List<CoordinationProtos.DrillbitEndpoint> incomingEndpoints)
                      throws PhysicalOperatorSetupException

Throws:

PhysicalOperatorSetupException

getSpecificScan

public ParquetRowGroupScan getSpecificScan(int minorFragmentId)

getMaxParallelizationWidth

public int getMaxParallelizationWidth()

getColumns

public List<SchemaPath> getColumns()

getScanStats

public ScanStats getScanStats()

Overrides:

getScanStats in class AbstractGroupScan

getNewWithChildren

public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children)

Description copied from interface: PhysicalOperator

Regenerate with this node with a new set of children. This is used in the case of materialization or optimization.

Returns:

getDigest

public String getDigest()

Description copied from interface: GroupScan

Returns a signature of the GroupScan which should usually be composed of all its attributes which could describe it uniquely.

toString

public String toString()

Overrides:

toString in class Object

clone

public GroupScan clone(List<SchemaPath> columns)

Description copied from interface: GroupScan

Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .

Specified by:

clone in interface GroupScan

Overrides:

clone in class AbstractGroupScan

clone

public ParquetGroupScan clone(FileSelection selection)
                       throws IOException

Specified by:

clone in interface FileGroupScan

Overrides:

clone in class AbstractFileGroupScan

Throws:

IOException

clone

public ParquetGroupScan clone(FileSelection selection,
                     long maxRecords)
                       throws IOException

Throws:

IOException

supportsLimitPushdown

public boolean supportsLimitPushdown()

Description copied from class: AbstractGroupScan

Default is not to support limit pushdown.

Specified by:

supportsLimitPushdown in interface GroupScan

Overrides:

supportsLimitPushdown in class AbstractGroupScan

applyLimit

public GroupScan applyLimit(long maxRecords)

Description copied from class: AbstractGroupScan

By default, return null to indicate rowcount based prune is not supported. Each groupscan subclass should override, if it supports rowcount based prune.

Specified by:

applyLimit in interface GroupScan

Overrides:

applyLimit in class AbstractGroupScan

Parameters:

maxRecords - : the number of rows requested from group scan.

Returns:

a new instance of group scan if the prune is successful. null when either if row-based prune is not supported, or if prune is not successful.

canPushdownProjects

public boolean canPushdownProjects(List<SchemaPath> columns)

Description copied from interface: GroupScan

GroupScan should check the list of columns, and see if it could support all the columns in the list.

Specified by:

canPushdownProjects in interface GroupScan

Overrides:

canPushdownProjects in class AbstractGroupScan

getColumnValueCount

public long getColumnValueCount(SchemaPath column)

Return column value count for the specified column. If does not contain such column, return 0.

Specified by:

getColumnValueCount in interface GroupScan

Overrides:

getColumnValueCount in class AbstractGroupScan

getPartitionColumns

public List<SchemaPath> getPartitionColumns()

Description copied from interface: GroupScan

Returns a list of columns that can be used for partition pruning

Specified by:

getPartitionColumns in interface GroupScan

Overrides:

getPartitionColumns in class AbstractGroupScan

applyFilter

public GroupScan applyFilter(LogicalExpression filterExpr,
                    UdfUtilities udfUtilities,
                    FunctionImplementationRegistry functionImplementationRegistry,
                    OptionManager optionManager)