org.apache.hadoop.hive.metastore.txn

Class TxnHandler

java.lang.Object

org.apache.hadoop.hive.metastore.txn.TxnHandler

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, TxnStore, TxnStore.MutexAPI

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public abstract class TxnHandler
extends Object
implements TxnStore, TxnStore.MutexAPI

A handler to answer transaction related calls that come into the metastore server.

Note on log messages: Please include txnid:X and lockid info using JavaUtils.txnIdToString(long) and JavaUtils.lockIdToString(long) in all messages. The txnid:X and lockid:Y matches how Thrift object toString() methods are generated, so keeping the format consistent makes grep'ing the logs much easier.

Note on HIVE_LOCKS.hl_last_heartbeat. For locks that are part of transaction, we set this 0 (would rather set it to NULL but Currently the DB schema has this NOT NULL) and only update/read heartbeat from corresponding transaction in TXNS.

In general there can be multiple metastores where this logic can execute, thus the DB is used to ensure proper mutexing of operations. Select ... For Update (or equivalent: either MsSql with(updlock) or actual Update stmt) is used to properly sequence operations. Most notably: 1. various sequence IDs are generated with aid of this mutex 2. ensuring that each (Hive) Transaction state is transitioned atomically. Transaction state includes its actual state (Open, Aborted) as well as it's lock list/component list. Thus all per transaction ops, either start by update/delete of the relevant TXNS row or do S4U on that row. This allows almost all operations to run at READ_COMMITTED and minimizes DB deadlocks. 3. checkLock() - this is mutexted entirely since we must ensure that while we check if some lock can be granted, no other (strictly speaking "earlier") lock can change state.

The exception to this is Derby which doesn't support proper S4U. Derby is always running embedded (this is the only supported configuration for Derby) in the same JVM as HiveMetaStoreHandler thus we use JVM wide lock to properly sequnce the operations.

If we ever decide to run remote Derby server, according to https://db.apache.org/derby/docs/10.0/manuals/develop/develop78.html all transactions will be seriazlied, so that would also work though has not been tested.

General design note: It's imperative that any operation on a txn (e.g. commit), ensure (atomically) that this txn is still valid and active. In the code this is usually achieved at the same time the txn record is locked for some operation.

Note on retry logic: Metastore has retry logic in both RetryingMetaStoreClient and RetryingHMSHandler. The retry logic there is very generic and is not aware whether the operations are idempotent or not. (This is separate from retry logic here in TxnHander which can/does retry DB errors intelligently). The worst case is when an op here issues a successful commit against the RDBMS but the calling stack doesn't receive the ack and retries. (If an op fails before commit, it's trivially idempotent) Thus the ops here need to be made idempotent as much as possible or the metstore call stack should have logic not to retry. There are RetrySemantics annotations to document the behavior.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TxnHandler.ConfVars

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.metastore.txn.TxnStore

TxnStore.MUTEX_KEY, TxnStore.MutexAPI

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.metastore.txn.TxnStore.MutexAPI

TxnStore.MutexAPI.LockHandle

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf Number of consecutive deadlocks we have seen
static TxnHandler.ConfVars	ConfVars
protected static DataSource	connPoolCompactor
protected static DatabaseProduct	dbProduct
protected MultiDataSourceJdbcResource	jdbcResource
protected static long	openTxnTimeOutMillis
protected static SQLGenerator	sqlGenerator
protected List<TransactionalMetaStoreEventListener>	transactionalListeners

Fields inherited from interface org.apache.hadoop.hive.metastore.txn.TxnStore

ABORT_TXN_CLEANUP_TYPE, ABORTED_RESPONSE, ABORTED_STATE, CLEANING_RESPONSE, COMPACTION_STATES, DID_NOT_INITIATE, DID_NOT_INITIATE_RESPONSE, FAILED_RESPONSE, FAILED_STATE, INITIATED_RESPONSE, INITIATED_STATE, MAJOR_TYPE, MINOR_TYPE, POOL_COMPACTOR, POOL_MUTEX, POOL_TX, READY_FOR_CLEANING, REBALANCE_TYPE, REFUSED_RESPONSE, REFUSED_STATE, SUCCEEDED_RESPONSE, SUCCEEDED_STATE, TIMED_OUT_TXN_ABORT_BATCH_SIZE, TXN_KEY_START, WORKING_RESPONSE, WORKING_STATE

Constructor Summary

Constructors

Constructor and Description
TxnHandler()

Method Summary

Modifier and Type	Method and Description
AbortCompactResponse	abortCompactions(AbortCompactionRequest reqst) Abort (rollback) a compaction.
void	abortTxn(AbortTxnRequest rqst) Abort (rollback) a transaction.
void	abortTxns(AbortTxnsRequest rqst) Abort (rollback) a list of transactions in one request.
TxnStore.MutexAPI.LockHandle	acquireLock(String key) The key is name of the lock.
void	acquireLock(String key, TxnStore.MutexAPI.LockHandle handle) Same as TxnStore.MutexAPI.acquireLock(String) but takes an already existing handle as input.
void	addDynamicPartitions(AddDynamicPartitions rqst) Retry-by-caller note: This may be retried after dbConn.commit.
void	addWriteIdsToMinHistory(long txnid, Map<String,Long> minOpenWriteIds)
void	addWriteNotificationLog(ListenerEvent acidWriteEvent) Add the ACID write event information to writeNotificationLog table.
AllocateTableWriteIdsResponse	allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) Allocate a write ID for the given table and associate it with a transaction
LockResponse	checkLock(CheckLockRequest rqst) Why doesn't this get a txnid as parameter? The caller should either know the txnid or know there isn't one.
long	cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout)
void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) Clean up corresponding records in metastore tables when corresponding object is dropped, specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc operations.
void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator, long txnId)
void	commitTxn(CommitTxnRequest rqst) Concurrency/isolation notes: This is mutexed with openTxns(OpenTxnRequest) and other commitTxn(CommitTxnRequest) operations using select4update on NEXT_TXN_ID.
CompactionResponse	compact(CompactionRequest rqst) Submit a compaction request into the queue.
void	countOpenTxns() Get the count for open transactions.
org.apache.hadoop.conf.Configuration	getConf()
protected Connection	getDbConn(int isolationLevel, DataSource connPool)
protected Timestamp	getDbTime() Determine the current time, using the RDBMS as a source of truth
MultiDataSourceJdbcResource	getJdbcResourceHolder()
GetLatestCommittedCompactionInfoResponse	getLatestCommittedCompactionInfo(GetLatestCommittedCompactionInfoRequest rqst) We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
long	getLatestTxnIdInConflict(long txnid) Checks if there are transactions that are overlapping, i.e.
Materialization	getMaterializationInvalidationInfo(CreationMetadata creationMetadata, String validTxnListStr) Get invalidation info for the materialization.
MaxAllocatedTableWriteIdResponse	getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) Reads the maximum allocated writeId for the given table
MetricsInfo	getMetricsInfo() Returns ACID metadata related metrics info.
TxnStore.MutexAPI	getMutexAPI()
int	getNumLocks() TODO: remove in future, for testing only, do not use.
GetOpenTxnsResponse	getOpenTxns() Get list of valid transactions.
GetOpenTxnsResponse	getOpenTxns(List<TxnType> excludeTxnTypes) Get list of valid transactions.
GetOpenTxnsInfoResponse	getOpenTxnsInfo() Get information about open transactions.
long	getOpenTxnTimeOutMillis()
SqlRetryHandler	getRetryHandler()
long	getTargetTxnId(String replPolicy, long sourceTxnId)
long	getTxnIdForWriteId(String dbName, String tblName, long writeId)
GetValidWriteIdsResponse	getValidWriteIds(GetValidWriteIdsRequest rqst) Gets the list of valid write ids for the given table wrt to current txn
void	heartbeat(HeartbeatRequest ids) ids should only have txnid or lockid but not both, ideally.
boolean	heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
HeartbeatTxnRangeResponse	heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) Heartbeat a group of transactions together
LockResponse	lock(LockRequest rqst) As much as possible (i.e.
LockResponse	lockMaterializationRebuild(String dbName, String tableName, long txnId)
void	onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName, String newCatName, String newDbName, String newTabName, String newPartName) Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
OpenTxnsResponse	openTxns(OpenTxnRequest rqst) Retry-by-caller note: Worst case, it will leave an open txn which will timeout.
void	performTimeOuts() Isolation Level Notes Plain: RC is OK This will find transactions that have timed out and abort them.
void	performWriteSetGC() WriteSet tracking is used to ensure proper transaction isolation.
void	replTableWriteIdState(ReplTblWriteIdStateRequest rqst) Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
void	seedTxnId(SeedTxnIdRequest rqst) Sets the next txnId to the given value.
void	seedWriteId(SeedTableWriteIdsRequest rqst) Called on conversion of existing table to full acid.
void	setConf(org.apache.hadoop.conf.Configuration conf) This is logically part of c'tor and must be called prior to any other method.
void	setOpenTxnTimeOutMillis(long openTxnTimeOutMillis)
long	setTimeout(long milliseconds) TODO: remove in future, for testing only, do not use.
protected static DataSource	setupJdbcConnectionPool(org.apache.hadoop.conf.Configuration conf, int maxPoolSize)
ShowCompactResponse	showCompact(ShowCompactRequest rqst) Show list of current compactions.
ShowLocksResponse	showLocks(ShowLocksRequest rqst) Get information on current locks.
boolean	submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId)
void	unlock(UnlockRequest rqst) This would have been made simpler if all locks were associated with a txn.
void	updateTransactionStatistics(UpdateTransactionalStatsRequest req)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.metastore.txn.TxnStore

checkFailedCompactions, cleanEmptyAbortedAndCommittedTxns, cleanTxnToWriteIdTable, cleanupRecords, clearCleanerStart, findColumnsWithStats, findMinOpenTxnIdForCleaner, findMinTxnIdSeenOpen, findNextToCompact, findNextToCompact, findPotentialCompactions, findPotentialCompactions, findReadyToClean, findReadyToCleanAborts, getCompactionByTxnId, getCompactionMetricsData, getTopCompactionMetricsDataPerType, markCleaned, markCleanerStart, markCompacted, markFailed, markRefused, purgeCompactionHistory, removeCompactionMetricsData, removeDuplicateCompletedTxnComponents, revokeFromLocalWorkers, revokeTimedoutWorkers, setCleanerRetryRetentionTimeOnError, setHadoopJobId, updateCompactionMetricsData, updateCompactorState

Field Detail

ConfVars

public static final TxnHandler.ConfVars ConfVars

connPoolCompactor

protected static DataSource connPoolCompactor

dbProduct

protected static DatabaseProduct dbProduct

sqlGenerator

protected static SQLGenerator sqlGenerator

openTxnTimeOutMillis

protected static long openTxnTimeOutMillis

conf

protected org.apache.hadoop.conf.Configuration conf

Number of consecutive deadlocks we have seen

transactionalListeners

protected List<TransactionalMetaStoreEventListener> transactionalListeners

jdbcResource

protected MultiDataSourceJdbcResource jdbcResource

Constructor Detail

TxnHandler

public TxnHandler()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

This is logically part of c'tor and must be called prior to any other method. Not physically part of c'tor due to use of reflection

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getRetryHandler

public SqlRetryHandler getRetryHandler()

Specified by:

getRetryHandler in interface TxnStore

Returns:

Returns the SqlRetryHandler instance used by TxnStore.

getJdbcResourceHolder

public MultiDataSourceJdbcResource getJdbcResourceHolder()

Specified by:

getJdbcResourceHolder in interface TxnStore

Returns:

Returns the MultiDataSourceJdbcResource instance used by TxnStore.

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getOpenTxnsInfo

public GetOpenTxnsInfoResponse getOpenTxnsInfo()
                                        throws MetaException

Description copied from interface: TxnStore

Get information about open transactions. This gives extensive information about the transactions rather than just the list of transactions. This should be used when the need is to see information about the transactions (e.g. show transactions).

Specified by:

getOpenTxnsInfo in interface TxnStore

Returns:

information about open transactions

Throws:

MetaException

getOpenTxns

public GetOpenTxnsResponse getOpenTxns()
                                throws MetaException

Description copied from interface: TxnStore

Get list of valid transactions. This gives just the list of transactions that are open.

Specified by:

getOpenTxns in interface TxnStore

Returns:

list of open transactions, as well as a high watermark.

Throws:

MetaException

getOpenTxns

public GetOpenTxnsResponse getOpenTxns(List<TxnType> excludeTxnTypes)
                                throws MetaException

Description copied from interface: TxnStore

Get list of valid transactions. This gives just the list of transactions that are open.

Specified by:

getOpenTxns in interface TxnStore

Parameters:

excludeTxnTypes - : excludes this type of txns while getting the open txns

Returns:

list of open transactions, as well as a high watermark.

Throws:

MetaException

openTxns

public OpenTxnsResponse openTxns(OpenTxnRequest rqst)
                          throws MetaException

Retry-by-caller note: Worst case, it will leave an open txn which will timeout.

Specified by:

openTxns in interface TxnStore

Parameters:

rqst - request to open transactions

Returns:

information on opened transactions

Throws:

MetaException

getOpenTxnTimeOutMillis

public long getOpenTxnTimeOutMillis()

Specified by:

getOpenTxnTimeOutMillis in interface TxnStore

setOpenTxnTimeOutMillis

public void setOpenTxnTimeOutMillis(long openTxnTimeOutMillis)

Specified by:

setOpenTxnTimeOutMillis in interface TxnStore

getTargetTxnId

public long getTargetTxnId(String replPolicy,
                           long sourceTxnId)
                    throws MetaException

Specified by:

getTargetTxnId in interface TxnStore

Throws:

MetaException

abortTxn

public void abortTxn(AbortTxnRequest rqst)
              throws NoSuchTxnException,
                     MetaException,
                     TxnAbortedException

Description copied from interface: TxnStore

Abort (rollback) a transaction.

Specified by:

abortTxn in interface TxnStore

Parameters:

rqst - info on transaction to abort

Throws:

NoSuchTxnException

MetaException

TxnAbortedException

abortTxns

public void abortTxns(AbortTxnsRequest rqst)
               throws MetaException

Description copied from interface: TxnStore

Abort (rollback) a list of transactions in one request.

Specified by:

abortTxns in interface TxnStore

Parameters:

rqst - info on transactions to abort

Throws:

MetaException

commitTxn

public void commitTxn(CommitTxnRequest rqst)
               throws NoSuchTxnException,
                      TxnAbortedException,
                      MetaException

Concurrency/isolation notes: This is mutexed with openTxns(OpenTxnRequest) and other commitTxn(CommitTxnRequest) operations using select4update on NEXT_TXN_ID. Also, mutexes on TXNS table for specific txnid:X see more notes below. In order to prevent lost updates, we need to determine if any 2 transactions overlap. Each txn is viewed as an interval [M,N]. M is the txnid and N is taken from the same NEXT_TXN_ID sequence so that we can compare commit time of txn T with start time of txn S. This sequence can be thought of as a logical time counter. If S.commitTime < T.startTime, T and S do NOT overlap.

Motivating example: Suppose we have multi-statement transactions T and S both of which are attempting x = x + 1 In order to prevent lost update problem, then the non-overlapping txns must lock in the snapshot that they read appropriately. In particular, if txns do not overlap, then one follows the other (assuming they write the same entity), and thus the 2nd must see changes of the 1st. We ensure this by locking in snapshot after openTxns(OpenTxnRequest) call is made (see org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn) and mutexing openTxn() with commit(). In other words, once a S.commit() starts we must ensure that txn T which will be considered a later txn, locks in a snapshot that includes the result of S's commit (assuming no other txns). As a counter example, suppose we have S[3,3] and T[4,4] (commitId=txnid means no other transactions were running in parallel). If T and S both locked in the same snapshot (for example commit of txnid:2, which is possible if commitTxn() and openTxnx() is not mutexed) 'x' would be updated to the same value by both, i.e. lost update.

Specified by:

commitTxn in interface TxnStore

Parameters:

rqst - info on transaction to commit

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

getLatestTxnIdInConflict

public long getLatestTxnIdInConflict(long txnid)
                              throws MetaException

Checks if there are transactions that are overlapping, i.e. conflicting with the current one. Valid only if the caller txn holds an exclusive lock that prevents other txns to make changes to the same tables/partitions. Only considers txn performing update/delete and intentionally ignores inserts.

Specified by:

getLatestTxnIdInConflict in interface TxnStore

Parameters:

txnid -

Returns:

max Id for the conflicting transaction, if any, otherwise -1

Throws:

MetaException

replTableWriteIdState

public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst)
                           throws MetaException

Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.

Specified by:

replTableWriteIdState in interface TxnStore

Parameters:

rqst - info on table/partitions and writeid snapshot to replicate.

Throws:

MetaException

getValidWriteIds

public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst)
                                          throws MetaException

Description copied from interface: TxnStore

Gets the list of valid write ids for the given table wrt to current txn

Specified by:

getValidWriteIds in interface TxnStore

Parameters:

rqst - info on transaction and list of table names associated with given transaction

Throws:

MetaException

allocateTableWriteIds

public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
                                                    throws MetaException

Description copied from interface: TxnStore

Allocate a write ID for the given table and associate it with a transaction

Specified by:

allocateTableWriteIds in interface TxnStore

Parameters:

rqst - info on transaction and table to allocate write id

Throws:

MetaException

getMaxAllocatedTableWrited

public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst)
                                                            throws MetaException

Description copied from interface: TxnStore

Reads the maximum allocated writeId for the given table

Specified by:

getMaxAllocatedTableWrited in interface TxnStore

Parameters:

rqst - table for which the maximum writeId is requested

Returns:

the maximum allocated writeId

Throws:

MetaException

seedWriteId

public void seedWriteId(SeedTableWriteIdsRequest rqst)
                 throws MetaException

Description copied from interface: TxnStore

Called on conversion of existing table to full acid. Sets initial write ID to a high enough value so that we can assign unique ROW__IDs to data in existing files.

Specified by:

seedWriteId in interface TxnStore

Throws:

MetaException

seedTxnId

public void seedTxnId(SeedTxnIdRequest rqst)
               throws MetaException

Description copied from interface: TxnStore

Sets the next txnId to the given value. If the actual txnId is greater it will throw an exception.

Specified by:

seedTxnId in interface TxnStore

Throws:

MetaException

addWriteNotificationLog

public void addWriteNotificationLog(ListenerEvent acidWriteEvent)
                             throws MetaException

Description copied from interface: TxnStore

Add the ACID write event information to writeNotificationLog table.

Specified by:

addWriteNotificationLog in interface TxnStore

Throws:

MetaException

performWriteSetGC

public void performWriteSetGC()
                       throws MetaException

Description copied from interface: TxnStore

WriteSet tracking is used to ensure proper transaction isolation. This method deletes the transaction metadata once it becomes unnecessary.

Specified by:

performWriteSetGC in interface TxnStore

Throws:

MetaException

updateTransactionStatistics

public void updateTransactionStatistics(UpdateTransactionalStatsRequest req)
                                 throws MetaException

Specified by:

updateTransactionStatistics in interface TxnStore

Throws:

MetaException

getMaterializationInvalidationInfo

public Materialization getMaterializationInvalidationInfo(CreationMetadata creationMetadata,
                                                          String validTxnListStr)
                                                   throws MetaException

Get invalidation info for the materialization. Materialization information contains information about whether there was update/delete/compaction operations on the source tables used by the materialization since it was created.

Specified by:

getMaterializationInvalidationInfo in interface TxnStore

Parameters:

creationMetadata - creation metadata for the materialization

validTxnListStr - valid transaction list for snapshot taken for current query

Throws:

MetaException

lockMaterializationRebuild

public LockResponse lockMaterializationRebuild(String dbName,
                                               String tableName,
                                               long txnId)
                                        throws MetaException

Specified by:

lockMaterializationRebuild in interface TxnStore

Throws:

MetaException

heartbeatLockMaterializationRebuild

public boolean heartbeatLockMaterializationRebuild(String dbName,
                                                   String tableName,
                                                   long txnId)
                                            throws MetaException

Specified by:

heartbeatLockMaterializationRebuild in interface TxnStore

Throws:

MetaException

cleanupMaterializationRebuildLocks

public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList,
                                               long timeout)
                                        throws MetaException

Specified by:

cleanupMaterializationRebuildLocks in interface TxnStore

Throws:

MetaException

lock

public LockResponse lock(LockRequest rqst)
                  throws NoSuchTxnException,
                         TxnAbortedException,
                         MetaException

As much as possible (i.e. in absence of retries) we want both operations to be done on the same connection (but separate transactions). Retry-by-caller note: If the call to lock is from a transaction, then in the worst case there will be a duplicate set of locks but both sets will belong to the same txn so they will not conflict with each other. For locks w/o txn context (i.e. read-only query), this may lead to deadlock (at least a long wait). (e.g. 1st call creates locks in LOCK_WAITING mode and response gets lost. Then RetryingMetaStoreClient retries, and enqueues another set of locks in LOCK_WAITING. The 2nd LockResponse is delivered to the DbLockManager, which will keep dong checkLock(CheckLockRequest) until the 1st set of locks times out.

Specified by:

lock in interface TxnStore

Parameters:

rqst - information on the lock to obtain. If the requester is part of a transaction the txn information must be included in the lock request.

Returns:

info on the lock, including whether it was obtained.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

checkLock

public LockResponse checkLock(CheckLockRequest rqst)
                       throws NoSuchTxnException,
                              NoSuchLockException,
                              TxnAbortedException,
                              MetaException

Why doesn't this get a txnid as parameter? The caller should either know the txnid or know there isn't one. Either way getTxnIdFromLockId() will not be needed. This would be a Thrift change.

Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(), in practice more often) which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.

The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired. We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.

checkLock(CheckLockRequest) must run at SERIALIZABLE (make sure some lock we are checking against doesn't move from W to A in another txn) but this method can heartbeat in separate txn at READ_COMMITTED.

Retry-by-caller note: Retryable because checkLock(CheckLockRequest) is

Specified by:

checkLock in interface TxnStore

Parameters:

rqst - info on the lock to check

Returns:

info on the state of the lock

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

unlock

public void unlock(UnlockRequest rqst)
            throws TxnOpenException,
                   MetaException

This would have been made simpler if all locks were associated with a txn. Then only txn needs to be heartbeated, committed, etc. no need for client to track individual locks. When removing locks not associated with txn this potentially conflicts with heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db. since this only removes from HIVE_LOCKS at worst some lock acquire is delayed

Specified by:

unlock in interface TxnStore

Parameters:

rqst - lock to unlock

Throws:

TxnOpenException

MetaException

showLocks

public ShowLocksResponse showLocks(ShowLocksRequest rqst)
                            throws MetaException

Description copied from interface: TxnStore

Get information on current locks.

Specified by:

showLocks in interface TxnStore

Parameters:

rqst - lock information to retrieve

Returns:

lock information.

Throws:

MetaException

heartbeat

public void heartbeat(HeartbeatRequest ids)
               throws NoSuchTxnException,
                      NoSuchLockException,
                      TxnAbortedException,
                      MetaException

ids should only have txnid or lockid but not both, ideally. Currently DBTxnManager.heartbeat() enforces this.

Specified by:

heartbeat in interface TxnStore

Parameters:

ids - lock and/or txn id to heartbeat

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

heartbeatTxnRange

public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
                                            throws MetaException

Description copied from interface: TxnStore

Heartbeat a group of transactions together

Specified by:

heartbeatTxnRange in interface TxnStore

Parameters:

rqst - set of transactions to heartbat

Returns:

info on txns that were heartbeated

Throws:

MetaException

getTxnIdForWriteId

public long getTxnIdForWriteId(String dbName,
                               String tblName,
                               long writeId)
                        throws MetaException

Specified by:

getTxnIdForWriteId in interface TxnStore

Throws:

MetaException

compact

public CompactionResponse compact(CompactionRequest rqst)
                           throws MetaException

Description copied from interface: TxnStore

Submit a compaction request into the queue. This is called when a user manually requests a compaction.

Specified by:

compact in interface TxnStore

Parameters:

rqst - information on what to compact

Returns:

id of the compaction that has been started or existing id if this resource is already scheduled

Throws:

MetaException

submitForCleanup

public boolean submitForCleanup(CompactionRequest rqst,
                                long highestWriteId,
                                long txnId)
                         throws MetaException

Specified by:

submitForCleanup in interface TxnStore

Throws:

MetaException

showCompact

public ShowCompactResponse showCompact(ShowCompactRequest rqst)
                                throws MetaException

Description copied from interface: TxnStore

Show list of current compactions.

Specified by:

showCompact in interface TxnStore

Parameters:

rqst - info on which compactions to show

Returns:

compaction information

Throws:

MetaException

getLatestCommittedCompactionInfo

public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(GetLatestCommittedCompactionInfoRequest rqst)
                                                                          throws MetaException

We assume this is only called by metadata cache server to know if there are new base/delta files should be read. The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because only these two states means there are new files ready to be read.

Specified by:

getLatestCommittedCompactionInfo in interface TxnStore

Parameters:

rqst - info on which compaction to retrieve

Returns:

one latest compaction record for a non partitioned table or one latest record for each partition specified by the request.

Throws:

MetaException

getMetricsInfo

public MetricsInfo getMetricsInfo()
                           throws MetaException

Description copied from interface: TxnStore

Returns ACID metadata related metrics info.

Specified by:

getMetricsInfo in interface TxnStore

Returns:

metrics info object

Throws:

MetaException

addDynamicPartitions

public void addDynamicPartitions(AddDynamicPartitions rqst)
                          throws NoSuchTxnException,
                                 TxnAbortedException,
                                 MetaException

Retry-by-caller note: This may be retried after dbConn.commit. At worst, it will create duplicate entries in TXN_COMPONENTS which won't affect anything. See more comments in commitTxn(CommitTxnRequest)

Specified by:

addDynamicPartitions in interface TxnStore

Parameters:

rqst - dynamic partition info.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

cleanupRecords

public void cleanupRecords(HiveObjectType type,
                           Database db,
                           Table table,
                           Iterator<Partition> partitionIterator,
                           boolean keepTxnToWriteIdMetaData)
                    throws MetaException

Clean up corresponding records in metastore tables when corresponding object is dropped, specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc operations.

HIVE_LOCKS and WS_SET are cleaned up by AcidHouseKeeperService, if turned on

Specified by:

cleanupRecords in interface TxnStore

Throws:

MetaException

cleanupRecords

public void cleanupRecords(HiveObjectType type,
                           Database db,
                           Table table,
                           Iterator<Partition> partitionIterator,
                           long txnId)
                    throws MetaException

Specified by:

cleanupRecords in interface TxnStore

Throws:

MetaException

onRename

public void onRename(String oldCatName,
                     String oldDbName,
                     String oldTabName,
                     String oldPartName,
                     String newCatName,
                     String newDbName,
                     String newTabName,
                     String newPartName)
              throws MetaException

Catalog hasn't been added to transactional tables yet, so it's passed in but not used.

Specified by:

onRename in interface TxnStore

Throws:

MetaException

getNumLocks

public int getNumLocks()

TODO: remove in future, for testing only, do not use.

Specified by:

getNumLocks in interface TxnStore

setTimeout

public long setTimeout(long milliseconds)

TODO: remove in future, for testing only, do not use.

Specified by:

setTimeout in interface TxnStore

getDbConn

protected Connection getDbConn(int isolationLevel,
                               DataSource connPool)
                        throws SQLException

Throws:

SQLException

performTimeOuts

public void performTimeOuts()

Isolation Level Notes Plain: RC is OK This will find transactions that have timed out and abort them. Will also delete locks which are not associated with a transaction and have timed out Tries to keep transactions (against metastore db) small to reduce lock contention.

Specified by:

performTimeOuts in interface TxnStore

countOpenTxns

public void countOpenTxns()
                   throws MetaException

Description copied from interface: TxnStore

Get the count for open transactions.

Specified by:

countOpenTxns in interface TxnStore

Throws:

MetaException

addWriteIdsToMinHistory

public void addWriteIdsToMinHistory(long txnid,
                                    Map<String,Long> minOpenWriteIds)
                             throws MetaException

Specified by:

addWriteIdsToMinHistory in interface TxnStore

Throws:

MetaException

setupJdbcConnectionPool

protected static DataSource setupJdbcConnectionPool(org.apache.hadoop.conf.Configuration conf,
                                                    int maxPoolSize)

getMutexAPI

public TxnStore.MutexAPI getMutexAPI()

Specified by:

getMutexAPI in interface TxnStore

acquireLock

public TxnStore.MutexAPI.LockHandle acquireLock(String key)
                                         throws MetaException

Description copied from interface: TxnStore.MutexAPI

The key is name of the lock. Will acquire an exclusive lock or block. It returns a handle which must be used to release the lock. Each invocation returns a new handle.

Specified by:

acquireLock in interface TxnStore.MutexAPI

Throws:

MetaException

acquireLock

public void acquireLock(String key,
                        TxnStore.MutexAPI.LockHandle handle)
                 throws MetaException

Description copied from interface: TxnStore.MutexAPI

Same as TxnStore.MutexAPI.acquireLock(String) but takes an already existing handle as input. This will associate the lock on key with the same handle. All locks associated with the same handle will be released together.

Specified by:

acquireLock in interface TxnStore.MutexAPI

handle - not NULL

Throws:

MetaException

abortCompactions

public AbortCompactResponse abortCompactions(AbortCompactionRequest reqst)
                                      throws MetaException,
                                             NoSuchCompactionException

Description copied from interface: TxnStore

Abort (rollback) a compaction.

Specified by:

abortCompactions in interface TxnStore

Parameters:

reqst - info on compaction to abort

Returns:

Throws:

MetaException

NoSuchCompactionException

getDbTime

protected Timestamp getDbTime()
                       throws MetaException

Determine the current time, using the RDBMS as a source of truth

Returns:

current time in milliseconds

Throws:

MetaException - if the time cannot be determined