org.apache.hadoop.hive.ql.io.orc

Class OrcFlowFileWriter.TreeWriter

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter.TreeWriter

Direct Known Subclasses:

OrcFlowFileWriter.BinaryTreeWriter, OrcFlowFileWriter.BooleanTreeWriter, OrcFlowFileWriter.ByteTreeWriter, OrcFlowFileWriter.DateTreeWriter, OrcFlowFileWriter.DecimalTreeWriter, OrcFlowFileWriter.DoubleTreeWriter, OrcFlowFileWriter.FloatTreeWriter, OrcFlowFileWriter.IntegerTreeWriter, OrcFlowFileWriter.ListTreeWriter, OrcFlowFileWriter.MapTreeWriter, OrcFlowFileWriter.StringTreeWriter, OrcFlowFileWriter.StructTreeWriter, OrcFlowFileWriter.TimestampTreeWriter, OrcFlowFileWriter.UnionTreeWriter

Enclosing class:

OrcFlowFileWriter

private abstract static class OrcFlowFileWriter.TreeWriter
extends Object

The parent class of all of the writers for each column. Each column is written by an instance of this class. The compound types (struct, list, map, and union) have children tree writers that write the children types.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.hive.ql.io.filters.BloomFilterIO	bloomFilter
private org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder	bloomFilterEntry
private org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.Builder	bloomFilterIndex
private org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream	bloomFilterStream
protected OrcFlowFileWriter.TreeWriter[]	childrenWriters
protected boolean	createBloomFilter
private org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl	fileStatistics
private boolean	foundNulls
protected int	id
protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl	indexStatistics
protected org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector	inspector
private boolean	isCompressed
private org.apache.hadoop.hive.ql.io.orc.BitFieldWriter	isPresent
private org.apache.hadoop.hive.ql.io.orc.OutStream	isPresentOutStream
private org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder	rowIndex
private org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder	rowIndexEntry
protected OrcFlowFileWriter.RowIndexPositionRecorder	rowIndexPosition
private org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream	rowIndexStream
protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl	stripeColStatistics
private List<org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder>	stripeStatsBuilders

Constructor Summary

Constructors

Constructor and Description
TreeWriter(int columnId, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector, OrcFlowFileWriter.StreamFactory streamFactory, boolean nullable) Create a tree writer.

Method Summary

Modifier and Type	Method and Description
(package private) void	addBloomFilterEntry()
(package private) org.apache.hadoop.hive.ql.io.orc.IntegerWriter	createIntegerWriter(org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream output, boolean signed, boolean isDirectV2, OrcFlowFileWriter.StreamFactory writer)
(package private) void	createRowIndexEntry() Create a row index entry with the previous location and the current index statistics.
(package private) long	estimateMemory() Estimate how much memory the writer is consuming excluding the streams.
(package private) OrcFlowFileWriter.TreeWriter[]	getChildrenWriters()
(package private) org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding	getEncoding() Get the encoding for this column.
protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl	getFileStatistics()
protected org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder	getRowIndex()
protected org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder	getRowIndexEntry()
protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl	getStripeStatistics()
(package private) boolean	isNewWriteFormat(OrcFlowFileWriter.StreamFactory writer)
(package private) void	recordPosition(org.apache.hadoop.hive.ql.io.orc.PositionRecorder recorder) Record the current position in each of this column's streams.
private void	removeIsPresentPositions()
(package private) void	write(Object obj) Add a new value to the column.
(package private) void	writeBatch(org.apache.hadoop.hive.ql.exec.vector.ColumnVector vector, int offset, int length) Write the values from the given vector from offset for length elements.
(package private) void	writeRootBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch, int offset, int length) Handle the top level object write.
(package private) void	writeStripe(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder builder, int requiredIndexEntries) Write the stripe out to the file.
private void	writeStripeStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder builder, OrcFlowFileWriter.TreeWriter treeWriter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

id

protected final int id

inspector

protected final org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector

isPresent

private final org.apache.hadoop.hive.ql.io.orc.BitFieldWriter isPresent

isCompressed

private final boolean isCompressed

indexStatistics

protected final org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl indexStatistics

stripeColStatistics

protected final org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl stripeColStatistics

fileStatistics

private final org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl fileStatistics

childrenWriters

protected OrcFlowFileWriter.TreeWriter[] childrenWriters

rowIndexPosition

protected final OrcFlowFileWriter.RowIndexPositionRecorder rowIndexPosition

rowIndex

private final org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder rowIndex

rowIndexEntry

private final org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder rowIndexEntry

rowIndexStream

private final org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream rowIndexStream

bloomFilterStream

private final org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream bloomFilterStream

bloomFilter

protected final org.apache.hadoop.hive.ql.io.filters.BloomFilterIO bloomFilter

createBloomFilter

protected final boolean createBloomFilter

bloomFilterIndex

private final org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.Builder bloomFilterIndex

bloomFilterEntry

private final org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder bloomFilterEntry

foundNulls

private boolean foundNulls

isPresentOutStream

private org.apache.hadoop.hive.ql.io.orc.OutStream isPresentOutStream

stripeStatsBuilders

private final List<org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder> stripeStatsBuilders

Constructor Detail

TreeWriter

TreeWriter(int columnId,
           org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector,
           OrcFlowFileWriter.StreamFactory streamFactory,
           boolean nullable)
    throws IOException

Create a tree writer.

Parameters:

columnId - the column id of the column to write

inspector - the object inspector to use

streamFactory - limited access to the Writer's data.

nullable - can the value be null?

Throws:

IOException - if an error occurs during creation

Method Detail

getRowIndex

protected org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder getRowIndex()

getStripeStatistics

protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl getStripeStatistics()

getFileStatistics

protected org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl getFileStatistics()

getRowIndexEntry

protected org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder getRowIndexEntry()

createIntegerWriter

org.apache.hadoop.hive.ql.io.orc.IntegerWriter createIntegerWriter(org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream output,
                                                                   boolean signed,
                                                                   boolean isDirectV2,
                                                                   OrcFlowFileWriter.StreamFactory writer)

isNewWriteFormat

boolean isNewWriteFormat(OrcFlowFileWriter.StreamFactory writer)

write

void write(Object obj)
    throws IOException

Add a new value to the column.

Parameters:

obj - The value to write to the column

Throws:

IOException - if an error occurs during add

removeIsPresentPositions

private void removeIsPresentPositions()

writeStripe

void writeStripe(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder builder,
                 int requiredIndexEntries)
          throws IOException

Write the stripe out to the file.

Parameters:

builder - the stripe footer that contains the information about the layout of the stripe. The TreeWriter is required to update the footer with its information.

requiredIndexEntries - the number of index entries that are required. this is to check to make sure the row index is well formed.

Throws:

IOException - if an error occurs during write

writeStripeStatistics

private void writeStripeStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder builder,
                                   OrcFlowFileWriter.TreeWriter treeWriter)

getChildrenWriters

OrcFlowFileWriter.TreeWriter[] getChildrenWriters()

getEncoding

org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getEncoding()

Get the encoding for this column.

Returns:

the information about the encoding of this column

createRowIndexEntry

void createRowIndexEntry()
                  throws IOException

Create a row index entry with the previous location and the current index statistics. Also merges the index statistics into the file statistics before they are cleared. Finally, it records the start of the next index and ensures all of the children columns also create an entry.

Throws:

IOException - if an error occurs during create

addBloomFilterEntry

void addBloomFilterEntry()

recordPosition

void recordPosition(org.apache.hadoop.hive.ql.io.orc.PositionRecorder recorder)
             throws IOException

Record the current position in each of this column's streams.

Parameters:

recorder - where should the locations be recorded

Throws:

IOException - if an error occurs during position recording

estimateMemory

long estimateMemory()

Estimate how much memory the writer is consuming excluding the streams.

Returns:

the number of bytes.

writeRootBatch

void writeRootBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch,
                    int offset,
                    int length)
             throws IOException

Handle the top level object write. This default method is used for all types except structs, which are the typical case. VectorizedRowBatch assumes the top level object is a struct, so we use the first column for all other types.

Parameters:

batch - the batch to write from

offset - the row to start on

length - the number of rows to write

Throws:

IOException - if an error occurs during write

writeBatch

void writeBatch(org.apache.hadoop.hive.ql.exec.vector.ColumnVector vector,
                int offset,
                int length)
         throws IOException

Write the values from the given vector from offset for length elements.

Parameters:

vector - the vector to write from

offset - the first value from the vector to write

length - the number of values from the vector to write

Throws:

IOException - if an error occurs during write