org.apache.hadoop.hive.ql.io.orc

Class OrcFlowFileWriter.StringTreeWriter

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter.TreeWriter

org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter.StringTreeWriter

Direct Known Subclasses:

OrcFlowFileWriter.CharTreeWriter, OrcFlowFileWriter.VarcharTreeWriter

Enclosing class:

OrcFlowFileWriter

private static class OrcFlowFileWriter.StringTreeWriter
extends OrcFlowFileWriter.TreeWriter

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	buildIndex
private org.apache.hadoop.hive.ql.io.orc.StringRedBlackTree	dictionary
private float	dictionaryKeySizeThreshold
private org.apache.hadoop.hive.ql.io.orc.IntegerWriter	directLengthOutput
private org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream	directStreamOutput
private boolean	doneDictionaryCheck
private static int	INITIAL_DICTIONARY_SIZE
private boolean	isDirectV2
private org.apache.hadoop.hive.ql.io.orc.IntegerWriter	lengthOutput
private List<Long>	rowIndexValueCount
private org.apache.hadoop.hive.ql.io.orc.IntegerWriter	rowOutput
private org.apache.hadoop.hive.ql.io.orc.DynamicIntArray	rows
private List<org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry>	savedRowIndex
private boolean	strideDictionaryCheck
private org.apache.hadoop.hive.ql.io.orc.OutStream	stringOutput
private boolean	useDictionaryEncoding

Fields inherited from class org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter.TreeWriter

bloomFilter, childrenWriters, createBloomFilter, id, indexStatistics, inspector, rowIndexPosition, stripeColStatistics

Constructor Summary

Constructors

Constructor and Description
StringTreeWriter(int columnId, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector, OrcFlowFileWriter.StreamFactory writer, boolean nullable)

Method Summary

Modifier and Type	Method and Description
private boolean	checkDictionaryEncoding()
(package private) void	createRowIndexEntry() This method doesn't call the super method, because unlike most of the other TreeWriters, this one can't record the position in the streams until the stripe is being flushed.
(package private) long	estimateMemory() Estimate how much memory the writer is consuming excluding the streams.
private void	flushDictionary()
(package private) org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding	getEncoding() Get the encoding for this column.
(package private) org.apache.hadoop.io.Text	getTextValue(Object obj) Method to retrieve text values from the value object, which can be overridden by subclasses.
private void	recordDirectStreamPosition()
(package private) void	write(Object obj) Add a new value to the column.
(package private) void	writeStripe(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder builder, int requiredIndexEntries) Write the stripe out to the file.

Methods inherited from class org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter.TreeWriter

addBloomFilterEntry, createIntegerWriter, getChildrenWriters, getFileStatistics, getRowIndex, getRowIndexEntry, getStripeStatistics, isNewWriteFormat, recordPosition, writeBatch, writeRootBatch

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INITIAL_DICTIONARY_SIZE

private static final int INITIAL_DICTIONARY_SIZE

See Also:

Constant Field Values

stringOutput

private final org.apache.hadoop.hive.ql.io.orc.OutStream stringOutput

lengthOutput

private final org.apache.hadoop.hive.ql.io.orc.IntegerWriter lengthOutput

rowOutput

private final org.apache.hadoop.hive.ql.io.orc.IntegerWriter rowOutput

dictionary

private final org.apache.hadoop.hive.ql.io.orc.StringRedBlackTree dictionary

rows

private final org.apache.hadoop.hive.ql.io.orc.DynamicIntArray rows

directStreamOutput

private final org.apache.hadoop.hive.ql.io.orc.PositionedOutputStream directStreamOutput

directLengthOutput

private final org.apache.hadoop.hive.ql.io.orc.IntegerWriter directLengthOutput

savedRowIndex

private final List<org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry> savedRowIndex

buildIndex

private final boolean buildIndex

rowIndexValueCount

private final List<Long> rowIndexValueCount

dictionaryKeySizeThreshold

private final float dictionaryKeySizeThreshold

useDictionaryEncoding

private boolean useDictionaryEncoding

isDirectV2

private boolean isDirectV2

doneDictionaryCheck

private boolean doneDictionaryCheck

strideDictionaryCheck

private final boolean strideDictionaryCheck

Constructor Detail

StringTreeWriter

StringTreeWriter(int columnId,
                 org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector,
                 OrcFlowFileWriter.StreamFactory writer,
                 boolean nullable)
          throws IOException

Throws:

IOException

Method Detail

getTextValue

org.apache.hadoop.io.Text getTextValue(Object obj)

Method to retrieve text values from the value object, which can be overridden by subclasses.

Parameters:

obj - value

Returns:

Text text value from obj

write

void write(Object obj)
    throws IOException

Description copied from class: OrcFlowFileWriter.TreeWriter

Add a new value to the column.

Overrides:

write in class OrcFlowFileWriter.TreeWriter

Parameters:

obj - The value to write to the column

Throws:

IOException - if an error occurs during add

checkDictionaryEncoding

private boolean checkDictionaryEncoding()

writeStripe

void writeStripe(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder builder,
                 int requiredIndexEntries)
          throws IOException

Description copied from class: OrcFlowFileWriter.TreeWriter

Write the stripe out to the file.

Overrides:

writeStripe in class OrcFlowFileWriter.TreeWriter

Parameters:

builder - the stripe footer that contains the information about the layout of the stripe. The TreeWriter is required to update the footer with its information.

requiredIndexEntries - the number of index entries that are required. this is to check to make sure the row index is well formed.

Throws:

IOException - if an error occurs during write

flushDictionary

private void flushDictionary()
                      throws IOException

Throws:

IOException

getEncoding

org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getEncoding()

Description copied from class: OrcFlowFileWriter.TreeWriter

Get the encoding for this column.

Overrides:

getEncoding in class OrcFlowFileWriter.TreeWriter

Returns:

the information about the encoding of this column

createRowIndexEntry

void createRowIndexEntry()
                  throws IOException

This method doesn't call the super method, because unlike most of the other TreeWriters, this one can't record the position in the streams until the stripe is being flushed. Therefore it saves all of the entries and augments them with the final information as the stripe is written.

Overrides:

createRowIndexEntry in class OrcFlowFileWriter.TreeWriter

Throws:

IOException - if an error occurs on create

recordDirectStreamPosition

private void recordDirectStreamPosition()
                                 throws IOException

Throws:

IOException

estimateMemory

long estimateMemory()

Description copied from class: OrcFlowFileWriter.TreeWriter

Estimate how much memory the writer is consuming excluding the streams.

Overrides:

estimateMemory in class OrcFlowFileWriter.TreeWriter

Returns:

the number of bytes.