org.apache.drill.exec.physical.impl.spill

Class RecordBatchSizer

java.lang.Object

org.apache.drill.exec.physical.impl.spill.RecordBatchSizer

public class RecordBatchSizer
extends Object

Given a record batch or vector container, determines the actual memory consumed by each column, the average row, and the entire record batch.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RecordBatchSizer.ColumnSize Column size information.

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_VECTOR_SIZE
int	maxSize Maximum width of a column; used for memory estimation in case of Varchars
int	nullableCount Count the nullable columns; used for memory estimation

Constructor Summary

Constructors

Constructor and Description
RecordBatchSizer(RecordBatch batch)
RecordBatchSizer(VectorAccessible va) Create empirical metadata for a record batch given a vector accessible (basically, an iterator over the vectors in the batch.)
RecordBatchSizer(VectorAccessible va, SelectionVector2 sv2) Create empirical metadata for a record batch given a vector accessible (basically, an iterator over the vectors in the batch) along with a selection vector for those records.

Method Summary

Methods

Modifier and Type	Method and Description
long	actualSize()
void	applySv2()
int	avgDensity()
VectorInitializer	buildVectorInitializer() The column size information gathered here represents empirically-derived schema metadata.
List<RecordBatchSizer.ColumnSize>	columns()
int	grossRowWidth()
boolean	hasSv2()
int	maxAvgColumnSize()
int	netRowWidth()
int	netRowWidthCap50() Compute the "real" width of the row, taking into account each varchar column size (historically capped at 50, and rounded up to power of 2 to match drill buf allocation) and null marking columns.
long	netSize()
int	rowCount()
static int	safeDivide(long num, long denom)
int	stdRowWidth()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MAX_VECTOR_SIZE

public static final int MAX_VECTOR_SIZE

maxSize

public int maxSize

Maximum width of a column; used for memory estimation in case of Varchars

nullableCount

public int nullableCount

Count the nullable columns; used for memory estimation

Constructor Detail

RecordBatchSizer

public RecordBatchSizer(RecordBatch batch)

RecordBatchSizer

public RecordBatchSizer(VectorAccessible va)

Create empirical metadata for a record batch given a vector accessible (basically, an iterator over the vectors in the batch.)

Parameters:

va - iterator over the batch's vectors

RecordBatchSizer

public RecordBatchSizer(VectorAccessible va,
                SelectionVector2 sv2)

Create empirical metadata for a record batch given a vector accessible (basically, an iterator over the vectors in the batch) along with a selection vector for those records. The selection vector is used to pad the estimated row width with the extra two bytes needed per record. The selection vector memory is added to the total memory consumed by this batch.

Parameters:

va - iterator over the batch's vectors

sv2 - selection vector associated with this batch

Method Detail

applySv2

public void applySv2()

safeDivide

public static int safeDivide(long num,
             long denom)

rowCount

public int rowCount()

stdRowWidth

public int stdRowWidth()

grossRowWidth

public int grossRowWidth()

netRowWidth

public int netRowWidth()

columns

public List<RecordBatchSizer.ColumnSize> columns()

netRowWidthCap50

public int netRowWidthCap50()

Compute the "real" width of the row, taking into account each varchar column size (historically capped at 50, and rounded up to power of 2 to match drill buf allocation) and null marking columns.

Returns:

"real" width of the row

actualSize

public long actualSize()

hasSv2

public boolean hasSv2()

avgDensity

public int avgDensity()

netSize

public long netSize()

maxAvgColumnSize

public int maxAvgColumnSize()

toString

public String toString()

Overrides:

toString in class Object

buildVectorInitializer

public VectorInitializer buildVectorInitializer()

The column size information gathered here represents empirically-derived schema metadata. Use that metadata to create an instance of a class that allocates memory for new vectors based on the observed size information. The caller provides the row count; the size information here provides column widths and the number of elements in each array.