org.apache.hadoop.hive.ql.io.orc

Class OrcFlowFileWriter

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.OrcFlowFileWriter

All Implemented Interfaces:

org.apache.hadoop.hive.ql.io.orc.MemoryManager.Callback, org.apache.hadoop.hive.ql.io.orc.Writer

public class OrcFlowFileWriter
extends Object
implements org.apache.hadoop.hive.ql.io.orc.Writer, org.apache.hadoop.hive.ql.io.orc.MemoryManager.Callback

An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams.

This class is synchronized so that multi-threaded access is ok. In particular, because the MemoryManager is shared between writers, this class assumes that checkMemory may be called from a separate thread.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	OrcFlowFileWriter.BinaryTreeWriter
private static class	OrcFlowFileWriter.BooleanTreeWriter
private class	OrcFlowFileWriter.BufferedStream This class is used to hold the contents of streams as they are buffered.
private static class	OrcFlowFileWriter.ByteTreeWriter
private static class	OrcFlowFileWriter.CharTreeWriter Under the covers, char is written to ORC the same way as string.
private static class	OrcFlowFileWriter.DateTreeWriter
private static class	OrcFlowFileWriter.DecimalTreeWriter
private class	OrcFlowFileWriter.DirectStream An output receiver that writes the ByteBuffers to the output stream as they are received.
private static class	OrcFlowFileWriter.DoubleTreeWriter
private static class	OrcFlowFileWriter.FloatTreeWriter
private static class	OrcFlowFileWriter.IntegerTreeWriter
private static class	OrcFlowFileWriter.ListTreeWriter
private static class	OrcFlowFileWriter.MapTreeWriter
private static class	OrcFlowFileWriter.RowIndexPositionRecorder
private class	OrcFlowFileWriter.StreamFactory Interface from the Writer to the TreeWriters.
private static class	OrcFlowFileWriter.StringTreeWriter
private static class	OrcFlowFileWriter.StructTreeWriter
private static class	OrcFlowFileWriter.TimestampTreeWriter
private static class	OrcFlowFileWriter.TreeWriter The parent class of all of the writers for each column.
private static class	OrcFlowFileWriter.UnionTreeWriter
private static class	OrcFlowFileWriter.VarcharTreeWriter Under the covers, varchar is written to ORC the same way as string.

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	addBlockPadding
private long	adjustedStripeSize
(package private) static String	BASE_TIMESTAMP_STRING
private long	blockSize
private boolean[]	bloomFilterColumns
private double	bloomFilterFpp
private int	bufferSize
private boolean	buildIndex
private org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterCallback	callback
private org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterContext	callbackContext
private org.apache.hadoop.hive.ql.io.orc.CompressionCodec	codec
private static int	COLUMN_COUNT_THRESHOLD
private int	columnCount
private org.apache.hadoop.hive.ql.io.orc.CompressionKind	compress
private org.apache.hadoop.hive.ql.io.orc.OrcFile.CompressionStrategy	compressionStrategy
private org.apache.hadoop.conf.Configuration	conf
private long	defaultStripeSize
private org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy	encodingStrategy
private OutputStream	flowFileOutputStream
private static int	HDFS_BUFFER_SIZE
private long	headerLength
private static org.apache.commons.logging.Log	LOG
private org.apache.hadoop.hive.ql.io.orc.MemoryManager	memoryManager
(package private) static int	MILLIS_PER_SECOND
private static int	MIN_ROW_INDEX_STRIDE
private float	paddingTolerance
private org.apache.hadoop.fs.Path	path
private com.google.protobuf.CodedOutputStream	protobufWriter
private long	rawDataSize
private ByteCountingOutputStream	rawWriter
private long	rowCount
private int	rowIndexStride
private int	rowsInIndex
private long	rowsInStripe
private OrcFlowFileWriter.StreamFactory	streamFactory
private Map<org.apache.hadoop.hive.ql.io.orc.StreamName,OrcFlowFileWriter.BufferedStream>	streams
private List<org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation>	stripes
private int	stripesAtLastFlush
private OrcFlowFileWriter.TreeWriter	treeWriter
private Map<String,com.google.protobuf.ByteString>	userMetadata
private org.apache.hadoop.hive.ql.io.orc.OrcFile.Version	version
private org.apache.hadoop.hive.ql.io.orc.OutStream	writer

Constructor Summary

Constructors

Constructor and Description

OrcFlowFileWriter(OutputStream flowFileOutputStream, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector, long stripeSize, org.apache.hadoop.hive.ql.io.orc.CompressionKind compress, int bufferSize, int rowIndexStride, org.apache.hadoop.hive.ql.io.orc.MemoryManager memoryManager, boolean addBlockPadding, org.apache.hadoop.hive.ql.io.orc.OrcFile.Version version, org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterCallback callback, org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy encodingStrategy, org.apache.hadoop.hive.ql.io.orc.OrcFile.CompressionStrategy compressionStrategy, float paddingTolerance, long blockSizeValue, String bloomFilterColumnNames, double bloomFilterFpp)

Method Summary

Modifier and Type	Method and Description
void	addRow(Object row)
void	addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
void	addUserMetadata(String name, ByteBuffer value)
void	appendStripe(byte[] stripe, int offset, int length, org.apache.hadoop.hive.ql.io.orc.StripeInformation stripeInfo, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics stripeStatistics)
void	appendUserMetadata(List<org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem> userMetadata)
boolean	checkMemory(double newScale)
void	close()
private long	computeRawDataSize()
static org.apache.hadoop.hive.ql.io.orc.CompressionCodec	createCodec(org.apache.hadoop.hive.ql.io.orc.CompressionKind kind)
private void	createRowIndexEntry()
private static OrcFlowFileWriter.TreeWriter	createTreeWriter(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector, OrcFlowFileWriter.StreamFactory streamFactory, boolean nullable)
private long	estimateStripeSize()
private void	flushStripe()
private List<OrcFlowFileWriter.TreeWriter>	getAllColumnTreeWriters(OrcFlowFileWriter.TreeWriter rootTreeWriter)
private void	getAllColumnTreeWritersImpl(OrcFlowFileWriter.TreeWriter tw, List<OrcFlowFileWriter.TreeWriter> result)
private int	getClosestBufferSize(int estBufferSize, int bs)
private String	getColumnNamesFromInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector)
(package private) int	getEstimatedBufferSize(int bs)
(package private) int	getEstimatedBufferSize(String colNames, int bs)
private long	getMemoryAvailableForORC()
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
private long	getRawDataSizeFromInspectors(OrcFlowFileWriter.TreeWriter child, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector oi)
private long	getRawDataSizeFromPrimitives(OrcFlowFileWriter.TreeWriter child, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector oi)
OutputStream	getStream()
private void	updateFileStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics stripeStatistics)
private org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind	writeCompressionKind(org.apache.hadoop.hive.ql.io.orc.CompressionKind kind)
private void	writeFileStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder builder, OrcFlowFileWriter.TreeWriter writer)
private int	writeFooter(long bodyLength)
long	writeIntermediateFooter()
private int	writeMetadata()
private int	writePostScript(int footerLength, int metadataLength)
private static void	writeTypes(org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder builder, OrcFlowFileWriter.TreeWriter treeWriter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

private static final org.apache.commons.logging.Log LOG

HDFS_BUFFER_SIZE

private static final int HDFS_BUFFER_SIZE

See Also:

Constant Field Values

MIN_ROW_INDEX_STRIDE

private static final int MIN_ROW_INDEX_STRIDE

See Also:

Constant Field Values

COLUMN_COUNT_THRESHOLD

private static final int COLUMN_COUNT_THRESHOLD

See Also:

Constant Field Values

path

private final org.apache.hadoop.fs.Path path

defaultStripeSize

private final long defaultStripeSize

adjustedStripeSize

private long adjustedStripeSize

rowIndexStride

private final int rowIndexStride

compress

private final org.apache.hadoop.hive.ql.io.orc.CompressionKind compress

codec

private final org.apache.hadoop.hive.ql.io.orc.CompressionCodec codec

addBlockPadding

private final boolean addBlockPadding

bufferSize

private final int bufferSize

blockSize

private final long blockSize

paddingTolerance

private final float paddingTolerance

streams

private final Map<org.apache.hadoop.hive.ql.io.orc.StreamName,OrcFlowFileWriter.BufferedStream> streams

flowFileOutputStream

private final OutputStream flowFileOutputStream

rawWriter

private ByteCountingOutputStream rawWriter

writer

private org.apache.hadoop.hive.ql.io.orc.OutStream writer

protobufWriter

private com.google.protobuf.CodedOutputStream protobufWriter

headerLength

private long headerLength

columnCount

private int columnCount

rowCount

private long rowCount

rowsInStripe

private long rowsInStripe

rawDataSize

private long rawDataSize

rowsInIndex

private int rowsInIndex

stripesAtLastFlush

private int stripesAtLastFlush

stripes

private final List<org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation> stripes

userMetadata

private final Map<String,com.google.protobuf.ByteString> userMetadata

streamFactory

private final OrcFlowFileWriter.StreamFactory streamFactory

treeWriter

private final OrcFlowFileWriter.TreeWriter treeWriter

buildIndex

private final boolean buildIndex

memoryManager

private final org.apache.hadoop.hive.ql.io.orc.MemoryManager memoryManager

version

private final org.apache.hadoop.hive.ql.io.orc.OrcFile.Version version

conf

private final org.apache.hadoop.conf.Configuration conf

callback

private final org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterCallback callback

callbackContext

private final org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterContext callbackContext

encodingStrategy

private final org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy encodingStrategy

compressionStrategy

private final org.apache.hadoop.hive.ql.io.orc.OrcFile.CompressionStrategy compressionStrategy

bloomFilterColumns

private final boolean[] bloomFilterColumns

bloomFilterFpp

private final double bloomFilterFpp

MILLIS_PER_SECOND

static final int MILLIS_PER_SECOND

See Also:

Constant Field Values

BASE_TIMESTAMP_STRING

static final String BASE_TIMESTAMP_STRING

See Also:

Constant Field Values

Constructor Detail

OrcFlowFileWriter

public OrcFlowFileWriter(OutputStream flowFileOutputStream,
                         org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.conf.Configuration conf,
                         org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector,
                         long stripeSize,
                         org.apache.hadoop.hive.ql.io.orc.CompressionKind compress,
                         int bufferSize,
                         int rowIndexStride,
                         org.apache.hadoop.hive.ql.io.orc.MemoryManager memoryManager,
                         boolean addBlockPadding,
                         org.apache.hadoop.hive.ql.io.orc.OrcFile.Version version,
                         org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterCallback callback,
                         org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy encodingStrategy,
                         org.apache.hadoop.hive.ql.io.orc.OrcFile.CompressionStrategy compressionStrategy,
                         float paddingTolerance,
                         long blockSizeValue,
                         String bloomFilterColumnNames,
                         double bloomFilterFpp)
                  throws IOException

Throws:

IOException

Method Detail

getColumnNamesFromInspector

private String getColumnNamesFromInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector)

getEstimatedBufferSize

int getEstimatedBufferSize(int bs)

getEstimatedBufferSize

int getEstimatedBufferSize(String colNames,
                           int bs)

getClosestBufferSize

private int getClosestBufferSize(int estBufferSize,
                                 int bs)

getMemoryAvailableForORC

private long getMemoryAvailableForORC()

createCodec

public static org.apache.hadoop.hive.ql.io.orc.CompressionCodec createCodec(org.apache.hadoop.hive.ql.io.orc.CompressionKind kind)

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

Specified by:

checkMemory in interface org.apache.hadoop.hive.ql.io.orc.MemoryManager.Callback

Throws:

IOException

createTreeWriter

private static OrcFlowFileWriter.TreeWriter createTreeWriter(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector inspector,
                                                             OrcFlowFileWriter.StreamFactory streamFactory,
                                                             boolean nullable)
                                                      throws IOException

Throws:

IOException

writeTypes

private static void writeTypes(org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder builder,
                               OrcFlowFileWriter.TreeWriter treeWriter)

getStream

public OutputStream getStream()
                       throws IOException

Throws:

IOException

createRowIndexEntry

private void createRowIndexEntry()
                          throws IOException

Throws:

IOException

flushStripe

private void flushStripe()
                  throws IOException

Throws:

IOException

computeRawDataSize

private long computeRawDataSize()

getRawDataSizeFromInspectors

private long getRawDataSizeFromInspectors(OrcFlowFileWriter.TreeWriter child,
                                          org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector oi)

getRawDataSizeFromPrimitives

private long getRawDataSizeFromPrimitives(OrcFlowFileWriter.TreeWriter child,
                                          org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector oi)

writeCompressionKind

private org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind writeCompressionKind(org.apache.hadoop.hive.ql.io.orc.CompressionKind kind)

writeFileStatistics

private void writeFileStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder builder,
                                 OrcFlowFileWriter.TreeWriter writer)
                          throws IOException

Throws:

IOException

writeMetadata

private int writeMetadata()
                   throws IOException

Throws:

IOException

writeFooter

private int writeFooter(long bodyLength)
                 throws IOException

Throws:

IOException

writePostScript

private int writePostScript(int footerLength,
                            int metadataLength)
                     throws IOException

Throws:

IOException

estimateStripeSize

private long estimateStripeSize()

addUserMetadata

public void addUserMetadata(String name,
                            ByteBuffer value)

Specified by:

addUserMetadata in interface org.apache.hadoop.hive.ql.io.orc.Writer

addRow

public void addRow(Object row)
            throws IOException

Specified by:

addRow in interface org.apache.hadoop.hive.ql.io.orc.Writer

Throws:

IOException

addRowBatch

public void addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
                 throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface org.apache.hadoop.hive.ql.io.orc.Writer

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

Specified by:

getRawDataSize in interface org.apache.hadoop.hive.ql.io.orc.Writer

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

Specified by:

getNumberOfRows in interface org.apache.hadoop.hive.ql.io.orc.Writer

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

Specified by:

writeIntermediateFooter in interface org.apache.hadoop.hive.ql.io.orc.Writer

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         org.apache.hadoop.hive.ql.io.orc.StripeInformation stripeInfo,
                         org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

Specified by:

appendStripe in interface org.apache.hadoop.hive.ql.io.orc.Writer

Throws:

IOException

updateFileStatistics

private void updateFileStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics stripeStatistics)

getAllColumnTreeWriters

private List<OrcFlowFileWriter.TreeWriter> getAllColumnTreeWriters(OrcFlowFileWriter.TreeWriter rootTreeWriter)

getAllColumnTreeWritersImpl

private void getAllColumnTreeWritersImpl(OrcFlowFileWriter.TreeWriter tw,
                                         List<OrcFlowFileWriter.TreeWriter> result)

appendUserMetadata

public void appendUserMetadata(List<org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem> userMetadata)

Specified by:

appendUserMetadata in interface org.apache.hadoop.hive.ql.io.orc.Writer