org.apache.tajo.storage

Class Tablespace

java.lang.Object

org.apache.tajo.storage.Tablespace

public abstract class Tablespace
extends Object

Tablespace manages the functions of storing and reading data. Tablespace is a abstract class. For supporting such as HDFS, HBASE, a specific Tablespace should be implemented by inheriting this class.

Field Summary

Fields

Modifier and Type	Field and Description
protected TajoConf	conf
protected net.minidev.json.JSONObject	config
protected String	name
protected URI	uri
protected boolean	visible this space is visible or not.

Constructor Summary

Constructors

Constructor and Description
Tablespace(String name, URI uri, net.minidev.json.JSONObject config)

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	close() Release storage manager resource
abstract org.apache.hadoop.fs.Path	commitTable(OverridableConf queryContext, ExecutionBlockId finalEbId, LogicalPlan plan, Schema schema, TableDesc tableDesc) Finalizes result data.
abstract void	createTable(TableDesc tableDesc, boolean ifNotExists) This method is called after executing "CREATE TABLE" statement.
boolean	equals(Object obj)
Appender	getAppender(OverridableConf queryContext, TaskAttemptId taskAttemptId, TableMeta meta, Schema schema, org.apache.hadoop.fs.Path workDir) Returns Appender instance.
Appender	getAppenderForInsertRow(OverridableConf queryContext, TaskAttemptId taskAttemptId, TableMeta meta, Schema schema, org.apache.hadoop.fs.Path workDir)
net.minidev.json.JSONObject	getConfig()
Set<String>	getDependencies()
abstract FormatProperty	getFormatProperty(TableMeta meta)
abstract TupleRange[]	getInsertSortRanges(OverridableConf queryContext, TableDesc tableDesc, Schema inputSchema, SortSpec[] sortSpecs, TupleRange dataRange) It is called by a Repartitioner for range shuffling when the SortRangeType of SortNode is USING_STORAGE_MANAGER.
MetadataProvider	getMetadataProvider()
String	getName()
abstract StorageProperty	getProperty() It returns the storage property.
URI	getRootUri() if StorageProperty.isArbitraryPathAllowed() is true, the storage allows arbitrary path accesses.
Scanner	getScanner(TableMeta meta, Schema schema, Fragment fragment, Schema target) Returns Scanner instance.
Class<? extends Scanner>	getScannerClass(String dataFormat) Return the Scanner class for the DataFormat that is defined in storage-default.xml.
SeekableScanner	getSeekableScanner(TableMeta meta, Schema schema, CatalogProtos.FragmentProto fragment, Schema target) Returns Scanner instance.
abstract List<Fragment>	getSplits(String inputSourceId, TableDesc tableDesc, EvalNode filterCondition) Returns the splits that will serve as input for the scan tasks.
abstract URI	getStagingUri(OverridableConf context, String queryId, TableMeta meta)
abstract URI	getTableUri(String databaseName, String tableName) Get Table URI
abstract long	getTableVolume(TableDesc table, com.google.common.base.Optional<EvalNode> filter)
URI	getUri()
void	init(TajoConf tajoConf) Initialize Tablespace instance.
boolean	isVisible()
int	markAccetablePlanPart(LogicalPlan plan)
URI	prepareStagingSpace(TajoConf conf, String queryId, OverridableConf context, TableMeta meta)
abstract void	prepareTable(LogicalNode node) This method is called before executing 'INSERT' or 'CREATE TABLE as SELECT'.
abstract void	purgeTable(TableDesc tableDesc) This method is called after executing "DROP TABLE" statement with the 'PURGE' option which is the option to delete all the data.
void	rewritePlan(OverridableConf context, LogicalPlan plan) Rewrite the logical plan.
abstract void	rollbackTable(LogicalNode node)
void	setVisible(boolean visible)
protected abstract void	storageInit() Initialize storage manager.
String	toString()
abstract void	verifySchemaToWrite(TableDesc tableDesc, Schema outSchema) It is called after making logical plan.

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

name

protected final String name

uri

protected final URI uri

config

protected final net.minidev.json.JSONObject config

visible

protected boolean visible

this space is visible or not.

conf

protected TajoConf conf

Constructor Detail

Tablespace

public Tablespace(String name,
          URI uri,
          net.minidev.json.JSONObject config)

Method Detail

getConfig

public net.minidev.json.JSONObject getConfig()

setVisible

public void setVisible(boolean visible)

getDependencies

public Set<String> getDependencies()

storageInit

protected abstract void storageInit()
                             throws IOException

Initialize storage manager.

Throws:

IOException

getName

public String getName()

getUri

public URI getUri()

isVisible

public boolean isVisible()

toString

public String toString()

Overrides:

toString in class Object

getTableVolume

public abstract long getTableVolume(TableDesc table,
                  com.google.common.base.Optional<EvalNode> filter)
                             throws UnsupportedException

Throws:

UnsupportedException

getRootUri

public URI getRootUri()

if StorageProperty.isArbitraryPathAllowed() is true, the storage allows arbitrary path accesses. In this case, the storage must provide the root URI.

Returns:

Root URI

See Also:

StorageProperty#isArbitraryPathAllowed}

getTableUri

public abstract URI getTableUri(String databaseName,
              String tableName)

Get Table URI

Parameters:

databaseName - Database name

tableName - Table name

Returns:

Table URI

getSplits

public abstract List<Fragment> getSplits(String inputSourceId,
                       TableDesc tableDesc,
                       @Nullable
                       EvalNode filterCondition)
                                  throws IOException,
                                         TajoException

Returns the splits that will serve as input for the scan tasks. The number of splits matches the number of regions in a table.

Parameters:

inputSourceId - Input source identifier, which can be either relation name or execution block id

tableDesc - The table description for the target data.

filterCondition - filter condition which can prune splits if possible

Returns:

The list of input fragments.

Throws:

IOException

TajoException

getProperty

public abstract StorageProperty getProperty()

It returns the storage property.

Returns:

The storage property

getFormatProperty

public abstract FormatProperty getFormatProperty(TableMeta meta)

close

public abstract void close()

Release storage manager resource

getInsertSortRanges

public abstract TupleRange[] getInsertSortRanges(OverridableConf queryContext,
                               TableDesc tableDesc,
                               Schema inputSchema,
                               SortSpec[] sortSpecs,
                               TupleRange dataRange)
                                          throws IOException

It is called by a Repartitioner for range shuffling when the SortRangeType of SortNode is USING_STORAGE_MANAGER. In general Repartitioner determines the partition range using previous output statistics data. In the special cases, such as HBase Repartitioner uses the result of this method.

Parameters:

queryContext - The current query context which contains query properties.

tableDesc - The table description for the target data.

inputSchema - The input schema

sortSpecs - The sort specification that contains the sort column and sort order.

Returns:

The list of sort ranges.

Throws:

IOException

init

public void init(TajoConf tajoConf)
          throws IOException

Initialize Tablespace instance. It should be called before using.

Parameters:

tajoConf -

Throws:

IOException

getScanner

public Scanner getScanner(TableMeta meta,
                 Schema schema,
                 Fragment fragment,
                 @Nullable
                 Schema target)
                   throws IOException

Returns Scanner instance.

Parameters:

meta - The table meta

schema - The input schema

fragment - The fragment for scanning

target - The output schema

Returns:

Scanner instance

Throws:

IOException

getAppenderForInsertRow

public Appender getAppenderForInsertRow(OverridableConf queryContext,
                               TaskAttemptId taskAttemptId,
                               TableMeta meta,
                               Schema schema,
                               org.apache.hadoop.fs.Path workDir)
                                 throws IOException

Throws:

IOException

getSeekableScanner

public SeekableScanner getSeekableScanner(TableMeta meta,
                                 Schema schema,
                                 CatalogProtos.FragmentProto fragment,
                                 Schema target)
                                   throws IOException

Returns Scanner instance.

Parameters:

meta - The table meta

schema - The input schema

fragment - The fragment for scanning

target - The output schema

Returns:

Scanner instance

Throws:

IOException

getAppender

public Appender getAppender(OverridableConf queryContext,
                   TaskAttemptId taskAttemptId,
                   TableMeta meta,
                   Schema schema,
                   org.apache.hadoop.fs.Path workDir)
                     throws IOException

Returns Appender instance.

Parameters:

queryContext - Query property.

taskAttemptId - Task id.

meta - Table meta data.

schema - Output schema.

workDir - Working directory

Returns:

Appender instance

Throws:

IOException

getScannerClass

public Class<? extends Scanner> getScannerClass(String dataFormat)
                                         throws IOException

Return the Scanner class for the DataFormat that is defined in storage-default.xml.

Parameters:

dataFormat - store type

Returns:

The Scanner class

Throws:

IOException

verifySchemaToWrite

public abstract void verifySchemaToWrite(TableDesc tableDesc,
                       Schema outSchema)
                                  throws TajoException

It is called after making logical plan. Storage manager should verify the schema for inserting.

Parameters:

tableDesc - The table description of insert target.

outSchema - The output schema of select query for inserting.

Throws:

IOException

TajoException

rewritePlan

public void rewritePlan(OverridableConf context,
               LogicalPlan plan)
                 throws TajoException

Rewrite the logical plan. It is assumed that the final plan will be given in this method.

Throws:

TajoException

createTable

public abstract void createTable(TableDesc tableDesc,
               boolean ifNotExists)
                          throws TajoException,
                                 IOException

This method is called after executing "CREATE TABLE" statement. If a storage is a file based storage, a storage manager may create directory.

Parameters:

tableDesc - Table description which is created.

ifNotExists - Creates the table only when the table does not exist.

Throws:

IOException

TajoException

purgeTable

public abstract void purgeTable(TableDesc tableDesc)
                         throws IOException,
                                TajoException

This method is called after executing "DROP TABLE" statement with the 'PURGE' option which is the option to delete all the data.

Parameters:

tableDesc -

Throws:

IOException

TajoException

prepareTable

public abstract void prepareTable(LogicalNode node)
                           throws IOException,
                                  TajoException

This method is called before executing 'INSERT' or 'CREATE TABLE as SELECT'. In general Tajo creates the target table after finishing the final sub-query of CATS. But In the special cases, such as HBase INSERT or CAST query uses the target table information. That kind of the storage should implements the logic related to creating table in this method.

Parameters:

node - The child node of the root node.

Throws:

IOException

TajoException

commitTable

public abstract org.apache.hadoop.fs.Path commitTable(OverridableConf queryContext,
                                    ExecutionBlockId finalEbId,
                                    LogicalPlan plan,
                                    Schema schema,
                                    TableDesc tableDesc)
                                               throws IOException

Finalizes result data. Tajo stores result data in the staging directory. If the query fails, clean up the staging directory. Otherwise the query is successful, move to the final directory from the staging directory.

Parameters:

queryContext - The query property

finalEbId - The final execution block id

plan - The query plan

schema - The final output schema

tableDesc - The description of the target table

Returns:

Saved path

Throws:

IOException

rollbackTable

public abstract void rollbackTable(LogicalNode node)
                            throws IOException,
                                   TajoException

Throws:

IOException

TajoException

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

getStagingUri

public abstract URI getStagingUri(OverridableConf context,
                String queryId,
                TableMeta meta)
                           throws IOException

Throws:

IOException

prepareStagingSpace

public URI prepareStagingSpace(TajoConf conf,
                      String queryId,
                      OverridableConf context,
                      TableMeta meta)
                        throws IOException

Throws:

IOException

getMetadataProvider

public MetadataProvider getMetadataProvider()

markAccetablePlanPart

public int markAccetablePlanPart(LogicalPlan plan)