org.apache.hadoop.hive.metastore.txn

Class TxnUtils

java.lang.Object

org.apache.hadoop.hive.metastore.txn.TxnUtils

public class TxnUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
TxnUtils()

Method Summary

Modifier and Type	Method and Description
static List<Integer>	buildQueryWithINClause(org.apache.hadoop.conf.Configuration conf, List<String> queries, StringBuilder prefix, StringBuilder suffix, Collection<Long> inValues, String inColumn, boolean addParens, boolean notIn) Build a query (or queries if one query is too big but only for the case of 'IN' composite clause.
static List<Integer>	buildQueryWithINClauseStrings(org.apache.hadoop.conf.Configuration conf, List<String> queries, StringBuilder prefix, StringBuilder suffix, List<String> inList, String inColumn, boolean addParens, boolean notIn) Build a query (or queries if one query is too big but only for the case of 'IN' composite clause.
static String	createInsertPreparedStmt(String tableName, List<String> columnNames)
static String	createUpdatePreparedStmt(String tableName, List<String> columnNames, List<String> conditionKeys)
static ValidCompactorWriteIdList	createValidCompactWriteIdList(TableValidWriteIds tableValidWriteIds) Transform a TableValidWriteIds to a ValidCompactorWriteIdList.
static ValidTxnList	createValidTxnListForCleaner(GetOpenTxnsResponse txns, long minOpenTxn, boolean isAbortCleanup) Returns a valid txn list for cleaner.
static CompactionType	dbCompactionType2ThriftType(char dbValue)
static List<Integer>	executeQueriesInBatch(Statement stmt, List<String> queries, int batchSize)
static void	executeQueriesInBatchNoCount(DatabaseProduct dbProduct, Statement stmt, List<String> queries, int batchSize) Calls queries in batch, but does not return affected row numbers.
static String	findUserToRunAs(String location, Table t, org.apache.hadoop.conf.Configuration conf) Determine which user to run an operation as.
static long	generateTemporaryId()
static String[]	getDbTableName(String fullTableName)
static String	getEpochFn(DatabaseProduct dbProduct) Get database specific function which returns the milliseconds value after the epoch.
static String	getFullTableName(String dbName, String tableName) Should produce the result as <dbName>.<tableName>.
static SQLException	getSqlException(Throwable ex) Checks is the passed exception, or any of the root (cause) exceptions are an instance of SQLException.
static TxnStore	getTxnStore(org.apache.hadoop.conf.Configuration conf) Get an instance of the TxnStore that is appropriate for this store
static boolean	isAcidTable(Map<String,String> parameters)
static boolean	isAcidTable(Table table) Should produce the same result as org.apache.hadoop.hive.ql.io.AcidUtils#isAcidTable.
static boolean	isInsertOnlyTable(Table table)
static boolean	isTableSoftDeleteEnabled(Table table, boolean isSoftDelete)
static boolean	isTransactionalTable(Map<String,String> parameters)
static boolean	isTransactionalTable(Table table) Note, users are responsible for using the correct TxnManager.
static boolean	isValidTxn(long txnId)
static String	normalizePartitionCase(String s)
static String	nvl(String input) A helper method to return SQL's 'IS NULL' clause whenever input is NULL.
static void	raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid) Used to raise an informative error when the caller expected a txn in a particular TxnStatus but found it in some other status
static void	seedTxnSequence(Connection conn, org.apache.hadoop.conf.Configuration conf, Statement stmt, long seedTxnId) Restarts the txnId sequence with the given seed value.
static Character	thriftCompactionType2DbType(CompactionType ct)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TxnUtils

public TxnUtils()

Method Detail

createValidTxnListForCleaner

public static ValidTxnList createValidTxnListForCleaner(GetOpenTxnsResponse txns,
                                                        long minOpenTxn,
                                                        boolean isAbortCleanup)

Returns a valid txn list for cleaner.

Parameters:

txns - Response containing open txns list.

minOpenTxn - Minimum open txn which is min open write txn on the table in the case of abort cleanup.

isAbortCleanup - Whether the request is for abort cleanup.

Returns:

a valid txn list

createValidCompactWriteIdList

public static ValidCompactorWriteIdList createValidCompactWriteIdList(TableValidWriteIds tableValidWriteIds)

Transform a TableValidWriteIds to a ValidCompactorWriteIdList. This assumes that the caller intends to compact the files, and thus treats only open transactions/write ids as invalid. Additionally any writeId > highestOpenWriteId is also invalid. This is to avoid creating something like delta_17_120 where writeId 80, for example, is still open.

Parameters:

tableValidWriteIds - table write id list from the metastore

Returns:

a valid write id list.

getTxnStore

public static TxnStore getTxnStore(org.apache.hadoop.conf.Configuration conf)

Get an instance of the TxnStore that is appropriate for this store

Parameters:

conf - configuration

Returns:

txn store

isTransactionalTable

public static boolean isTransactionalTable(Table table)

Note, users are responsible for using the correct TxnManager. We do not look at SessionState.get().getTxnMgr().supportsAcid() here Should produce the same result as org.apache.hadoop.hive.ql.io.AcidUtils#isTransactionalTable.

Returns:

true if table is a transactional table, false otherwise

isTransactionalTable

public static boolean isTransactionalTable(Map<String,String> parameters)

isAcidTable

public static boolean isAcidTable(Table table)

Should produce the same result as org.apache.hadoop.hive.ql.io.AcidUtils#isAcidTable.

isAcidTable

public static boolean isAcidTable(Map<String,String> parameters)

isInsertOnlyTable

public static boolean isInsertOnlyTable(Table table)

isTableSoftDeleteEnabled

public static boolean isTableSoftDeleteEnabled(Table table,
                                               boolean isSoftDelete)

getFullTableName

public static String getFullTableName(String dbName,
                                      String tableName)

Should produce the result as <dbName>.<tableName>.

getDbTableName

public static String[] getDbTableName(String fullTableName)

buildQueryWithINClause

public static List<Integer> buildQueryWithINClause(org.apache.hadoop.conf.Configuration conf,
                                                   List<String> queries,
                                                   StringBuilder prefix,
                                                   StringBuilder suffix,
                                                   Collection<Long> inValues,
                                                   String inColumn,
                                                   boolean addParens,
                                                   boolean notIn)

Build a query (or queries if one query is too big but only for the case of 'IN' composite clause. For the case of 'NOT IN' clauses, multiple queries change the semantics of the intended query. E.g., Let's assume that input "inValues" parameter has [5, 6] and that _DIRECT_SQL_MAX_QUERY_LENGTH_ configuration parameter only allows one value in a 'NOT IN' clause, Then having two delete statements changes the semantics of the intended SQL statement. I.e. 'delete from T where a not in (5)' and 'delete from T where a not in (6)' sequence is not equal to 'delete from T where a not in (5, 6)'.) with one or multiple 'IN' or 'NOT IN' clauses with the given input parameters. Note that this method currently support only single column for IN/NOT IN clauses and that only covers OR-based composite 'IN' clause and AND-based composite 'NOT IN' clause. For example, for 'IN' clause case, the method will build a query with OR. E.g., "id in (1,2,3) OR id in (4,5,6)". For 'NOT IN' case, NOT IN list is broken into multiple 'NOT IN" clauses connected by AND. Note that, in this method, "a composite 'IN' clause" is defined as "a list of multiple 'IN' clauses in a query".

Parameters:

queries - OUT: Array of query strings

prefix - IN: Part of the query that comes before IN list

suffix - IN: Part of the query that comes after IN list

inValues - IN: Collection containing IN clause values

inColumn - IN: single column name of IN list operator

addParens - IN: add a pair of parenthesis outside the IN lists e.g. "(id in (1,2,3) OR id in (4,5,6))"

notIn - IN: is this for building a 'NOT IN' composite clause?

Returns:

OUT: a list of the count of IN list values that are in each of the corresponding queries

buildQueryWithINClauseStrings

public static List<Integer> buildQueryWithINClauseStrings(org.apache.hadoop.conf.Configuration conf,
                                                          List<String> queries,
                                                          StringBuilder prefix,
                                                          StringBuilder suffix,
                                                          List<String> inList,
                                                          String inColumn,
                                                          boolean addParens,
                                                          boolean notIn)

Build a query (or queries if one query is too big but only for the case of 'IN' composite clause. For the case of 'NOT IN' clauses, multiple queries change the semantics of the intended query. E.g., Let's assume that input "inList" parameter has [5, 6] and that _DIRECT_SQL_MAX_QUERY_LENGTH_ configuration parameter only allows one value in a 'NOT IN' clause, Then having two delete statements changes the semantics of the inteneded SQL statement. I.e. 'delete from T where a not in (5)' and 'delete from T where a not in (6)' sequence is not equal to 'delete from T where a not in (5, 6)'.) with one or multiple 'IN' or 'NOT IN' clauses with the given input parameters. Note that this method currently support only single column for IN/NOT IN clauses and that only covers OR-based composite 'IN' clause and AND-based composite 'NOT IN' clause. For example, for 'IN' clause case, the method will build a query with OR. E.g., "id in (1,2,3) OR id in (4,5,6)". For 'NOT IN' case, NOT IN list is broken into multiple 'NOT IN" clauses connected by AND. Note that, in this method, "a composite 'IN' clause" is defined as "a list of multiple 'IN' clauses in a query".

Parameters:

queries - IN-OUT: Array of query strings

prefix - IN: Part of the query that comes before IN list

suffix - IN: Part of the query that comes after IN list

inList - IN: the list with IN list values

inColumn - IN: single column name of IN list operator

addParens - IN: add a pair of parenthesis outside the IN lists e.g. "(id in (1,2,3) OR id in (4,5,6))"

notIn - IN: is this for building a 'NOT IN' composite clause?

Returns:

OUT: a list of the count of IN list values that are in each of the corresponding queries

getEpochFn

public static String getEpochFn(DatabaseProduct dbProduct)
                         throws MetaException

Get database specific function which returns the milliseconds value after the epoch.

Parameters:

dbProduct - The type of the db which is used

Throws:

MetaException - For unknown database type.

executeQueriesInBatchNoCount

public static void executeQueriesInBatchNoCount(DatabaseProduct dbProduct,
                                                Statement stmt,
                                                List<String> queries,
                                                int batchSize)
                                         throws SQLException

Calls queries in batch, but does not return affected row numbers. Same as executeQueriesInBatch, with the only difference when the db is Oracle. In this case it is called as an anonymous stored procedure instead of batching, since batching is not optimized. See: https://docs.oracle.com/cd/E11882_01/java.112/e16548/oraperf.htm#JJDBC28752

Parameters:

dbProduct - The type of the db which is used

stmt - Statement which will be used for batching and execution.

queries - List of sql queries to execute in a Statement batch.

batchSize - maximum number of queries in a single batch

Throws:

SQLException - Thrown if an execution error occurs.

executeQueriesInBatch

public static List<Integer> executeQueriesInBatch(Statement stmt,
                                                  List<String> queries,
                                                  int batchSize)
                                           throws SQLException

Parameters:

stmt - Statement which will be used for batching and execution.

queries - List of sql queries to execute in a Statement batch.

batchSize - maximum number of queries in a single batch

Returns:

A list with the number of rows affected by each query in queries.

Throws:

SQLException - Thrown if an execution error occurs.

seedTxnSequence

public static void seedTxnSequence(Connection conn,
                                   org.apache.hadoop.conf.Configuration conf,
                                   Statement stmt,
                                   long seedTxnId)
                            throws SQLException

Restarts the txnId sequence with the given seed value. It is the responsibility of the caller to not set the sequence backward.

Parameters:

conn - database connection

stmt - sql statement

seedTxnId - the seed value for the sequence

Throws:

SQLException - ex

findUserToRunAs

public static String findUserToRunAs(String location,
                                     Table t,
                                     org.apache.hadoop.conf.Configuration conf)
                              throws IOException,
                                     InterruptedException

Determine which user to run an operation as. If metastore.compactor.run.as.user is set, that user will be returned; if not: the the owner of the directory to be compacted. It is asserted that either the user running the hive metastore or the table owner must be able to stat the directory and determine the owner.

Parameters:

location - directory that will be read or written to.

t - metastore table object

Returns:

metastore.compactor.run.as.user value; or if that is not set: username of the owner of the location.

Throws:

IOException - if neither the hive metastore user nor the table owner can stat the location.

InterruptedException

dbCompactionType2ThriftType

public static CompactionType dbCompactionType2ThriftType(char dbValue)
                                                  throws SQLException

Throws:

SQLException

thriftCompactionType2DbType

public static Character thriftCompactionType2DbType(CompactionType ct)
                                             throws MetaException

Throws:

MetaException

nvl

public static String nvl(String input)

A helper method to return SQL's 'IS NULL' clause whenever input is NULL.

Parameters:

input - A string to be compared to null.

Returns:

String

normalizePartitionCase

public static String normalizePartitionCase(String s)

generateTemporaryId

public static long generateTemporaryId()

isValidTxn

public static boolean isValidTxn(long txnId)

raiseTxnUnexpectedState

public static void raiseTxnUnexpectedState(TxnStatus actualStatus,
                                           long txnid)
                                    throws NoSuchTxnException,
                                           TxnAbortedException

Used to raise an informative error when the caller expected a txn in a particular TxnStatus but found it in some other status

Throws:

NoSuchTxnException

TxnAbortedException

getSqlException

public static SQLException getSqlException(Throwable ex)
                                    throws IllegalArgumentException

Checks is the passed exception, or any of the root (cause) exceptions are an instance of SQLException. Returns with the found SQLException or throws an IllegalArgumentException if no SQLException found in the chain.

Parameters:

ex - The exception to check

Returns:

Returns with the SQLException found in the exception chain.

Throws:

IllegalArgumentException - Thrown if there is no SQLException in the exception chain

createUpdatePreparedStmt

public static String createUpdatePreparedStmt(String tableName,
                                              List<String> columnNames,
                                              List<String> conditionKeys)

createInsertPreparedStmt

public static String createInsertPreparedStmt(String tableName,
                                              List<String> columnNames)