org.apache.tajo.storage

Class StorageUtil

java.lang.Object

org.apache.tajo.storage.StorageConstants

org.apache.tajo.storage.StorageUtil

public class StorageUtil
extends StorageConstants

Field Summary

Fields

Modifier and Type	Field and Description
static TajoConf	TAJO_CONF

Fields inherited from class org.apache.tajo.storage.StorageConstants

AVRO_SCHEMA_LITERAL, AVRO_SCHEMA_URL, COMPRESSION_CODEC, COMPRESSION_TYPE, CSVFILE_SERDE, DEFAULT_BINARY_SERDE, DEFAULT_BLOCK_SIZE, DEFAULT_FIELD_DELIMITER, DEFAULT_ORC_BUFFER_SIZE, DEFAULT_ORC_COMPRESSION_KIND, DEFAULT_ORC_MAX_MERGE_DISTANCE, DEFAULT_ORC_ROW_INDEX_STRIDE, DEFAULT_ORC_STRIPE_SIZE, DEFAULT_PAGE_SIZE, DEFAULT_PARTITION_NAME, DEFAULT_TEXT_ERROR_TOLERANCE_MAXNUM, DEFAULT_TEXT_SERDE, DEFAULT_TEXT_SERDE_CLASS, INSERT_DIRECTLY, LOCAL_FS_URI, ORC_BUFFER_SIZE, ORC_COMPRESSION, ORC_MAX_MERGE_DISTANCE, ORC_ROW_INDEX_STRIDE, ORC_STRIPE_SIZE, PARQUET_DEFAULT_BLOCK_SIZE, PARQUET_DEFAULT_COMPRESSION_CODEC_NAME, PARQUET_DEFAULT_IS_DICTIONARY_ENABLED, PARQUET_DEFAULT_IS_VALIDATION_ENABLED, PARQUET_DEFAULT_PAGE_SIZE, RCFILE_NULL, RCFILE_SERDE, SEQUENCEFILE_DELIMITER, SEQUENCEFILE_NULL, SEQUENCEFILE_SERDE, SHUFFLE_TYPE, TEXT_DELIMITER, TEXT_ERROR_TOLERANCE_MAXNUM, TEXT_NULL, TEXT_SERDE_CLASS, TEXT_SKIP_HEADER_LINE, TIMEZONE

Constructor Summary

Constructors

Constructor and Description
StorageUtil()

Method Summary

Methods

Modifier and Type	Method and Description
static void	closeBuffer(ByteBuffer buffer)
static org.apache.hadoop.fs.Path	concatPath(org.apache.hadoop.fs.Path parent, String... childs)
static org.apache.hadoop.fs.Path	concatPath(String parent, String... childs)
static int	getColByteSize(Column col)
static int	getMaxFileSequence(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Written files can be one of two forms: "part-[0-9]*-[0-9]*" or "part-[0-9]*-[0-9]*-[0-9]*".
static int	readFully(InputStream is, byte[] buffer, int offset, int length)
static void	skipFully(DataInput in, int len) Similar to readFully().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TAJO_CONF

public static final TajoConf TAJO_CONF

Constructor Detail

StorageUtil

public StorageUtil()

Method Detail

getColByteSize

public static int getColByteSize(Column col)

concatPath

public static org.apache.hadoop.fs.Path concatPath(String parent,
                                   String... childs)

concatPath

public static org.apache.hadoop.fs.Path concatPath(org.apache.hadoop.fs.Path parent,
                                   String... childs)

getMaxFileSequence

public static int getMaxFileSequence(org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.fs.Path path,
                     boolean recursive)
                              throws IOException

Written files can be one of two forms: "part-[0-9]*-[0-9]*" or "part-[0-9]*-[0-9]*-[0-9]*". This method finds the maximum sequence number from existing data files through the above patterns. If it cannot find any matched file or the maximum number, it will return -1.

Parameters:

fs -

path -

recursive -

Returns:

The maximum sequence number

Throws:

IOException

closeBuffer

public static void closeBuffer(ByteBuffer buffer)

readFully

public static int readFully(InputStream is,
            byte[] buffer,
            int offset,
            int length)
                     throws IOException

Throws:

IOException

skipFully

public static void skipFully(DataInput in,
             int len)
                      throws IOException

Similar to readFully(). Skips bytes in a loop.

Parameters:

in - The DataInput to skip bytes from

len - number of bytes to skip.

Throws:

IOException - if it could not skip requested number of bytes for any reason (including EOF)