org.apache.hadoop.hive.metastore.txn

Interface TxnStore

All Superinterfaces:

org.apache.hadoop.conf.Configurable

All Known Implementing Classes:

TxnHandler

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public interface TxnStore
extends org.apache.hadoop.conf.Configurable

A handler to answer transaction related calls that come into the metastore server.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	TxnStore.MUTEX_KEY
static interface	TxnStore.MutexAPI This is primarily designed to provide coarse-grained mutex support to operations running inside the Metastore (of which there could be several instances).

Field Summary

Fields

Modifier and Type	Field and Description
static char	ABORT_TXN_CLEANUP_TYPE
static String	ABORTED_RESPONSE
static char	ABORTED_STATE
static String	CLEANING_RESPONSE
static String[]	COMPACTION_STATES
static char	DID_NOT_INITIATE
static String	DID_NOT_INITIATE_RESPONSE
static String	FAILED_RESPONSE
static char	FAILED_STATE
static String	INITIATED_RESPONSE
static char	INITIATED_STATE
static char	MAJOR_TYPE
static char	MINOR_TYPE
static String	POOL_COMPACTOR
static String	POOL_MUTEX
static String	POOL_TX
static char	READY_FOR_CLEANING
static char	REBALANCE_TYPE
static String	REFUSED_RESPONSE
static char	REFUSED_STATE
static String	SUCCEEDED_RESPONSE
static char	SUCCEEDED_STATE
static int	TIMED_OUT_TXN_ABORT_BATCH_SIZE
static String	TXN_KEY_START Prefix for key when committing with a key/value.
static String	WORKING_RESPONSE
static char	WORKING_STATE

Method Summary

Modifier and Type	Method and Description
AbortCompactResponse	abortCompactions(AbortCompactionRequest rqst) Abort (rollback) a compaction.
void	abortTxn(AbortTxnRequest rqst) Abort (rollback) a transaction.
void	abortTxns(AbortTxnsRequest rqst) Abort (rollback) a list of transactions in one request.
void	addDynamicPartitions(AddDynamicPartitions rqst) Add information on a set of dynamic partitions that participated in a transaction.
void	addWriteIdsToMinHistory(long txnId, Map<String,Long> minOpenWriteIds)
void	addWriteNotificationLog(ListenerEvent acidWriteEvent) Add the ACID write event information to writeNotificationLog table.
AllocateTableWriteIdsResponse	allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) Allocate a write ID for the given table and associate it with a transaction
boolean	checkFailedCompactions(CompactionInfo ci) Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled.
LockResponse	checkLock(CheckLockRequest rqst) Check whether a lock has been obtained.
void	cleanEmptyAbortedAndCommittedTxns() Clean up aborted or committed transactions from txns that have no components in txn_components.
void	cleanTxnToWriteIdTable() Clean up entries from TXN_TO_WRITE_ID table less than min_uncommited_txnid as found by min(max(TXNS.txn_id), min(WRITE_SET.WS_COMMIT_ID), min(Aborted TXNS.txn_id)).
long	cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout)
default void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator) Clean up corresponding records in metastore tables.
void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData)
void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator, long txnId)
void	clearCleanerStart(CompactionInfo info) Removes the cleaning start time for a particular compaction
void	commitTxn(CommitTxnRequest rqst) Commit a transaction
CompactionResponse	compact(CompactionRequest rqst) Submit a compaction request into the queue.
void	countOpenTxns() Get the count for open transactions.
List<String>	findColumnsWithStats(CompactionInfo ci) Queries metastore DB directly to find columns in the table which have statistics information.
long	findMinOpenTxnIdForCleaner() Return the currently seen minimum open transaction ID.
long	findMinTxnIdSeenOpen() Deprecated. remove when min_history_level table is dropped
CompactionInfo	findNextToCompact(FindNextCompactRequest rqst) This will grab the next compaction request off of the queue, and assign it to the worker.
CompactionInfo	findNextToCompact(String workerId) Deprecated. Replaced by findNextToCompact(org.apache.hadoop.hive.metastore.api.FindNextCompactRequest)
Set<CompactionInfo>	findPotentialCompactions(int abortedThreshold, long abortedTimeThreshold) This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction.
Set<CompactionInfo>	findPotentialCompactions(int abortedThreshold, long abortedTimeThreshold, long lastChecked)
List<CompactionInfo>	findReadyToClean(long minOpenTxnWaterMark, long retentionTime) Find entries in the queue that are ready to be cleaned.
List<CompactionInfo>	findReadyToCleanAborts(long abortedTimeThreshold, int abortedThreshold) Find the aborted entries in TXN_COMPONENTS which can be used to clean directories belonging to transactions in aborted state.
Optional<CompactionInfo>	getCompactionByTxnId(long txnId) Returns the compaction running in the transaction txnId
CompactionMetricsData	getCompactionMetricsData(String dbName, String tblName, String partitionName, CompactionMetricsData.MetricType type) Returns ACID metrics related info for a specific resource and metric type.
MultiDataSourceJdbcResource	getJdbcResourceHolder()
GetLatestCommittedCompactionInfoResponse	getLatestCommittedCompactionInfo(GetLatestCommittedCompactionInfoRequest rqst) Get one latest record of SUCCEEDED or READY_FOR_CLEANING compaction for a table/partition.
long	getLatestTxnIdInConflict(long txnid)
Materialization	getMaterializationInvalidationInfo(CreationMetadata cm, String validTxnList) Get invalidation info for the materialization.
MaxAllocatedTableWriteIdResponse	getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) Reads the maximum allocated writeId for the given table
MetricsInfo	getMetricsInfo() Returns ACID metadata related metrics info.
TxnStore.MutexAPI	getMutexAPI()
int	getNumLocks()
GetOpenTxnsResponse	getOpenTxns() Get list of valid transactions.
GetOpenTxnsResponse	getOpenTxns(List<TxnType> excludeTxnTypes) Get list of valid transactions.
GetOpenTxnsInfoResponse	getOpenTxnsInfo() Get information about open transactions.
long	getOpenTxnTimeOutMillis()
SqlRetryHandler	getRetryHandler()
long	getTargetTxnId(String replPolicy, long sourceTxnId)
List<CompactionMetricsData>	getTopCompactionMetricsDataPerType(int limit) Returns the top ACID metrics from each type CompactionMetricsData.MetricType
long	getTxnIdForWriteId(String dbName, String tblName, long writeId)
GetValidWriteIdsResponse	getValidWriteIds(GetValidWriteIdsRequest rqst) Gets the list of valid write ids for the given table wrt to current txn
void	heartbeat(HeartbeatRequest ids) Send a heartbeat for a lock or a transaction
boolean	heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
HeartbeatTxnRangeResponse	heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) Heartbeat a group of transactions together
LockResponse	lock(LockRequest rqst) Obtain a lock.
LockResponse	lockMaterializationRebuild(String dbName, String tableName, long txnId)
void	markCleaned(CompactionInfo info) This will remove an entry from the queue after it has been compacted.
void	markCleanerStart(CompactionInfo info) Sets the cleaning start time for a particular compaction
void	markCompacted(CompactionInfo info) This will mark an entry in the queue as compacted and put it in the ready to clean state.
void	markFailed(CompactionInfo info) Mark a compaction entry as failed.
void	markRefused(CompactionInfo info) Mark a compaction as refused (to run).
void	onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName, String newCatName, String newDbName, String newTabName, String newPartName)
OpenTxnsResponse	openTxns(OpenTxnRequest rqst) Open a set of transactions
void	performTimeOuts() Timeout transactions and/or locks.
void	performWriteSetGC() WriteSet tracking is used to ensure proper transaction isolation.
void	purgeCompactionHistory() For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example.
void	removeCompactionMetricsData(String dbName, String tblName, String partitionName, CompactionMetricsData.MetricType type) Remove records from the compaction metrics cache matching the filter criteria passed in as parameters
void	removeDuplicateCompletedTxnComponents() De-duplicate entries from COMPLETED_TXN_COMPONENTS table.
void	replTableWriteIdState(ReplTblWriteIdStateRequest rqst) Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
void	revokeFromLocalWorkers(String hostname) This will take all entries assigned to workers on a host return them to INITIATED state.
void	revokeTimedoutWorkers(long timeout) This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state.
void	seedTxnId(SeedTxnIdRequest rqst) Sets the next txnId to the given value.
void	seedWriteId(SeedTableWriteIdsRequest rqst) Called on conversion of existing table to full acid.
void	setCleanerRetryRetentionTimeOnError(CompactionInfo info) Stores the value of CompactionInfo.retryRetention and CompactionInfo.errorMessage fields of the CompactionInfo either by inserting or updating the fields in the HMS database.
void	setHadoopJobId(String hadoopJobId, long id) Once a ThreadPoolExecutor Worker submits a job to the cluster, it calls this to update the metadata.
void	setOpenTxnTimeOutMillis(long openTxnTimeOutMillis)
long	setTimeout(long milliseconds)
ShowCompactResponse	showCompact(ShowCompactRequest rqst) Show list of current compactions.
ShowLocksResponse	showLocks(ShowLocksRequest rqst) Get information on current locks.
boolean	submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId)
void	unlock(UnlockRequest rqst) Unlock a lock.
boolean	updateCompactionMetricsData(CompactionMetricsData data) Create, update or delete one record in the compaction metrics cache.
void	updateCompactorState(CompactionInfo ci, long compactionTxnId) This updates COMPACTION_QUEUE.
void	updateTransactionStatistics(UpdateTransactionalStatsRequest req)

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

TXN_KEY_START

static final String TXN_KEY_START

Prefix for key when committing with a key/value.

See Also:

Constant Field Values

INITIATED_RESPONSE

static final String INITIATED_RESPONSE

See Also:

Constant Field Values

WORKING_RESPONSE

static final String WORKING_RESPONSE

See Also:

Constant Field Values

CLEANING_RESPONSE

static final String CLEANING_RESPONSE

See Also:

Constant Field Values

FAILED_RESPONSE

static final String FAILED_RESPONSE

See Also:

Constant Field Values

SUCCEEDED_RESPONSE

static final String SUCCEEDED_RESPONSE

See Also:

Constant Field Values

DID_NOT_INITIATE_RESPONSE

static final String DID_NOT_INITIATE_RESPONSE

See Also:

Constant Field Values

REFUSED_RESPONSE

static final String REFUSED_RESPONSE

See Also:

Constant Field Values

ABORTED_RESPONSE

static final String ABORTED_RESPONSE

See Also:

Constant Field Values

INITIATED_STATE

static final char INITIATED_STATE

See Also:

Constant Field Values

WORKING_STATE

static final char WORKING_STATE

See Also:

Constant Field Values

READY_FOR_CLEANING

static final char READY_FOR_CLEANING

See Also:

Constant Field Values

FAILED_STATE

static final char FAILED_STATE

See Also:

Constant Field Values

SUCCEEDED_STATE

static final char SUCCEEDED_STATE

See Also:

Constant Field Values

DID_NOT_INITIATE

static final char DID_NOT_INITIATE

See Also:

Constant Field Values

REFUSED_STATE

static final char REFUSED_STATE

See Also:

Constant Field Values

ABORTED_STATE

static final char ABORTED_STATE

See Also:

Constant Field Values

MAJOR_TYPE

static final char MAJOR_TYPE

See Also:

Constant Field Values

MINOR_TYPE

static final char MINOR_TYPE

See Also:

Constant Field Values

REBALANCE_TYPE

static final char REBALANCE_TYPE

See Also:

Constant Field Values

ABORT_TXN_CLEANUP_TYPE

static final char ABORT_TXN_CLEANUP_TYPE

See Also:

Constant Field Values

COMPACTION_STATES

static final String[] COMPACTION_STATES

TIMED_OUT_TXN_ABORT_BATCH_SIZE

static final int TIMED_OUT_TXN_ABORT_BATCH_SIZE

See Also:

Constant Field Values

POOL_TX

static final String POOL_TX

See Also:

Constant Field Values

POOL_MUTEX

static final String POOL_MUTEX

See Also:

Constant Field Values

POOL_COMPACTOR

static final String POOL_COMPACTOR

See Also:

Constant Field Values

Method Detail

getRetryHandler

SqlRetryHandler getRetryHandler()

Returns:

Returns the SqlRetryHandler instance used by TxnStore.

getJdbcResourceHolder

MultiDataSourceJdbcResource getJdbcResourceHolder()

Returns:

Returns the MultiDataSourceJdbcResource instance used by TxnStore.

getOpenTxnsInfo

@Transactional(value="txnhandler")
GetOpenTxnsInfoResponse getOpenTxnsInfo()
                                                                    throws MetaException

Get information about open transactions. This gives extensive information about the transactions rather than just the list of transactions. This should be used when the need is to see information about the transactions (e.g. show transactions).

Returns:

information about open transactions

Throws:

MetaException

getOpenTxns

@Transactional(value="txnhandler")
GetOpenTxnsResponse getOpenTxns()
                                                            throws MetaException

Get list of valid transactions. This gives just the list of transactions that are open.

Returns:

list of open transactions, as well as a high watermark.

Throws:

MetaException

getOpenTxns

@Transactional(value="txnhandler")
GetOpenTxnsResponse getOpenTxns(List<TxnType> excludeTxnTypes)
                                                            throws MetaException

Get list of valid transactions. This gives just the list of transactions that are open.

Parameters:

excludeTxnTypes - : excludes this type of txns while getting the open txns

Returns:

list of open transactions, as well as a high watermark.

Throws:

MetaException

countOpenTxns

@Transactional(value="txnhandler")
void countOpenTxns()
                                               throws MetaException

Get the count for open transactions.

Throws:

MetaException

openTxns

@Transactional(value="txnhandler",
               noRollbackFor=SqlRetryException.class)
OpenTxnsResponse openTxns(OpenTxnRequest rqst)
                                                                                                            throws MetaException

Open a set of transactions

Parameters:

rqst - request to open transactions

Returns:

information on opened transactions

Throws:

MetaException

getTargetTxnId

@Transactional(value="txnhandler")
long getTargetTxnId(String replPolicy,
                                                       long sourceTxnId)
                                                throws MetaException

Throws:

MetaException

abortTxn

@Transactional(value="txnhandler")
void abortTxn(AbortTxnRequest rqst)
                                          throws NoSuchTxnException,
                                                 MetaException,
                                                 TxnAbortedException

Abort (rollback) a transaction.

Parameters:

rqst - info on transaction to abort

Throws:

NoSuchTxnException

MetaException

TxnAbortedException

abortTxns

@Transactional(value="txnhandler")
void abortTxns(AbortTxnsRequest rqst)
                                           throws NoSuchTxnException,
                                                  MetaException

Abort (rollback) a list of transactions in one request.

Parameters:

rqst - info on transactions to abort

Throws:

NoSuchTxnException

MetaException

commitTxn

@Transactional(value="txnhandler",
               noRollbackFor=org.apache.hadoop.hive.metastore.api.TxnAbortedException.class)
void commitTxn(CommitTxnRequest rqst)
                                                                                                                                        throws NoSuchTxnException,
                                                                                                                                               TxnAbortedException,
                                                                                                                                               MetaException

Commit a transaction

Parameters:

rqst - info on transaction to commit

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

replTableWriteIdState

@Transactional(value="txnhandler")
void replTableWriteIdState(ReplTblWriteIdStateRequest rqst)
                                                       throws MetaException

Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.

Parameters:

rqst - info on table/partitions and writeid snapshot to replicate.

Throws:

MetaException - in case of failure

updateTransactionStatistics

@Transactional(value="txnhandler")
void updateTransactionStatistics(UpdateTransactionalStatsRequest req)
                                                             throws MetaException

Throws:

MetaException

getMaterializationInvalidationInfo

@Transactional(value="txnhandler")
Materialization getMaterializationInvalidationInfo(CreationMetadata cm,
                                                                                      String validTxnList)
                                                                               throws MetaException

Get invalidation info for the materialization. Currently, the materialization information only contains information about whether there was update/delete operations on the source tables used by the materialization since it was created.

Parameters:

cm - creation metadata for the materialization

validTxnList - valid transaction list for snapshot taken for current query

Throws:

MetaException

getTxnIdForWriteId

long getTxnIdForWriteId(String dbName,
                        String tblName,
                        long writeId)
                 throws MetaException

Throws:

MetaException

getLatestTxnIdInConflict

@Transactional(value="txnhandler")
long getLatestTxnIdInConflict(long txnid)
                                                          throws MetaException

Throws:

MetaException

lockMaterializationRebuild

@Transactional(value="txnhandler")
LockResponse lockMaterializationRebuild(String dbName,
                                                                           String tableName,
                                                                           long txnId)
                                                                    throws MetaException

Throws:

MetaException

heartbeatLockMaterializationRebuild

@Transactional(value="txnhandler")
boolean heartbeatLockMaterializationRebuild(String dbName,
                                                                               String tableName,
                                                                               long txnId)
                                                                        throws MetaException

Throws:

MetaException

cleanupMaterializationRebuildLocks

@Transactional(value="txnhandler")
long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList,
                                                                           long timeout)
                                                                    throws MetaException

Throws:

MetaException

getValidWriteIds

@Transactional(value="txnhandler")
GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst)
                                                                      throws NoSuchTxnException,
                                                                             MetaException

Gets the list of valid write ids for the given table wrt to current txn

Parameters:

rqst - info on transaction and list of table names associated with given transaction

Throws:

NoSuchTxnException

MetaException

addWriteIdsToMinHistory

@Transactional(value="txnhandler")
void addWriteIdsToMinHistory(long txnId,
                                                                Map<String,Long> minOpenWriteIds)
                                                         throws MetaException

Throws:

MetaException

allocateTableWriteIds

@Transactional(value="txnhandler")
AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
                                                                                throws NoSuchTxnException,
                                                                                       TxnAbortedException,
                                                                                       MetaException

Allocate a write ID for the given table and associate it with a transaction

Parameters:

rqst - info on transaction and table to allocate write id

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

getMaxAllocatedTableWrited

@Transactional(value="txnhandler")
MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst)
                                                                                        throws MetaException

Reads the maximum allocated writeId for the given table

Parameters:

rqst - table for which the maximum writeId is requested

Returns:

the maximum allocated writeId

Throws:

MetaException

seedWriteId

@Transactional(value="txnhandler")
void seedWriteId(SeedTableWriteIdsRequest rqst)
                                             throws MetaException

Called on conversion of existing table to full acid. Sets initial write ID to a high enough value so that we can assign unique ROW__IDs to data in existing files.

Throws:

MetaException

seedTxnId

@Transactional(value="txnhandler")
void seedTxnId(SeedTxnIdRequest rqst)
                                           throws MetaException

Sets the next txnId to the given value. If the actual txnId is greater it will throw an exception.

Parameters:

rqst -

Throws:

MetaException

lock

LockResponse lock(LockRequest rqst)
           throws NoSuchTxnException,
                  TxnAbortedException,
                  MetaException

Obtain a lock.

Parameters:

rqst - information on the lock to obtain. If the requester is part of a transaction the txn information must be included in the lock request.

Returns:

info on the lock, including whether it was obtained.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

checkLock

@Transactional(value="txnhandler")
LockResponse checkLock(CheckLockRequest rqst)
                                                   throws NoSuchTxnException,
                                                          NoSuchLockException,
                                                          TxnAbortedException,
                                                          MetaException

Check whether a lock has been obtained. This is used after lock(org.apache.hadoop.hive.metastore.api.LockRequest) returned a wait state.

Parameters:

rqst - info on the lock to check

Returns:

info on the state of the lock

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

unlock

@Transactional(value="txnhandler")
void unlock(UnlockRequest rqst)
                                        throws NoSuchLockException,
                                               TxnOpenException,
                                               MetaException

Unlock a lock. It is not legal to call this if the caller is part of a txn. In that case the txn should be committed or aborted instead. (Note someday this will change since multi-statement transactions will allow unlocking in the transaction.)

Parameters:

rqst - lock to unlock

Throws:

NoSuchLockException

TxnOpenException

MetaException

showLocks

@Transactional(value="txnhandler")
ShowLocksResponse showLocks(ShowLocksRequest rqst)
                                                        throws MetaException

Get information on current locks.

Parameters:

rqst - lock information to retrieve

Returns:

lock information.

Throws:

MetaException

heartbeat

@Transactional(value="txnhandler")
void heartbeat(HeartbeatRequest ids)
                                           throws NoSuchTxnException,
                                                  NoSuchLockException,
                                                  TxnAbortedException,
                                                  MetaException

Send a heartbeat for a lock or a transaction

Parameters:

ids - lock and/or txn id to heartbeat

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

heartbeatTxnRange

@Transactional(value="txnhandler")
HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
                                                                        throws MetaException

Heartbeat a group of transactions together

Parameters:

rqst - set of transactions to heartbat

Returns:

info on txns that were heartbeated

Throws:

MetaException

compact

@Transactional(value="txnhandler")
CompactionResponse compact(CompactionRequest rqst)
                                                       throws MetaException

Submit a compaction request into the queue. This is called when a user manually requests a compaction.

Parameters:

rqst - information on what to compact

Returns:

id of the compaction that has been started or existing id if this resource is already scheduled

Throws:

MetaException

submitForCleanup

@Transactional(value="txnhandler")
boolean submitForCleanup(CompactionRequest rqst,
                                                            long highestWriteId,
                                                            long txnId)
                                                     throws MetaException

Throws:

MetaException

showCompact

@Transactional(value="txnhandler")
ShowCompactResponse showCompact(ShowCompactRequest rqst)
                                                            throws MetaException

Show list of current compactions.

Parameters:

rqst - info on which compactions to show

Returns:

compaction information

Throws:

MetaException

abortCompactions

@Transactional(value="txnhandler")
AbortCompactResponse abortCompactions(AbortCompactionRequest rqst)
                                                                  throws NoSuchCompactionException,
                                                                         MetaException

Abort (rollback) a compaction.

Parameters:

rqst - info on compaction to abort

Returns:

Throws:

NoSuchCompactionException

MetaException

getLatestCommittedCompactionInfo

@Transactional(value="txnhandler")
GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(GetLatestCommittedCompactionInfoRequest rqst)
                                                                                                      throws MetaException

Get one latest record of SUCCEEDED or READY_FOR_CLEANING compaction for a table/partition. No checking is done on the dbname, tablename, or partitionname to make sure they refer to valid objects. Is is assumed to be done by the caller. Note that partition names should be supplied with the request for a partitioned table; otherwise, no records will be returned.

Parameters:

rqst - info on which compaction to retrieve

Returns:

one latest compaction record for a non partitioned table or one latest record for each partition specified by the request.

Throws:

MetaException

addDynamicPartitions

@Transactional(value="txnhandler")
void addDynamicPartitions(AddDynamicPartitions rqst)
                                                      throws NoSuchTxnException,
                                                             TxnAbortedException,
                                                             MetaException

Add information on a set of dynamic partitions that participated in a transaction.

Parameters:

rqst - dynamic partition info.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

cleanupRecords

@Transactional(value="txnhandler")
default void cleanupRecords(HiveObjectType type,
                                                               Database db,
                                                               Table table,
                                                               Iterator<Partition> partitionIterator)
                                                        throws MetaException

Clean up corresponding records in metastore tables.

Parameters:

type - Hive object type

db - database object

table - table object

partitionIterator - partition iterator

Throws:

MetaException

cleanupRecords

@Transactional(value="txnhandler")
void cleanupRecords(HiveObjectType type,
                                                       Database db,
                                                       Table table,
                                                       Iterator<Partition> partitionIterator,
                                                       boolean keepTxnToWriteIdMetaData)
                                                throws MetaException

Throws:

MetaException

cleanupRecords

@Transactional(value="txnhandler")
void cleanupRecords(HiveObjectType type,
                                                       Database db,
                                                       Table table,
                                                       Iterator<Partition> partitionIterator,
                                                       long txnId)
                                                throws MetaException

Throws:

MetaException

onRename

@Transactional(value="txnhandler")
void onRename(String oldCatName,
                                                 String oldDbName,
                                                 String oldTabName,
                                                 String oldPartName,
                                                 String newCatName,
                                                 String newDbName,
                                                 String newTabName,
                                                 String newPartName)
                                          throws MetaException

Throws:

MetaException

performTimeOuts

@Transactional(value="txnhandler")
void performTimeOuts()

Timeout transactions and/or locks. This should only be called by the compactor.

findPotentialCompactions

@Transactional(value="compactor")
Set<CompactionInfo> findPotentialCompactions(int abortedThreshold,
                                                                               long abortedTimeThreshold)
                                                                        throws MetaException

This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction. Also, look through txns and txn_components tables for aborted transactions that we should add to the list.

Parameters:

abortedThreshold - number of aborted queries forming a potential compaction request.

abortedTimeThreshold - age of an aborted txn in milliseconds that will trigger a potential compaction request.

Returns:

list of CompactionInfo structs. These will not have id, type, or runAs set since these are only potential compactions not actual ones.

Throws:

MetaException

findPotentialCompactions

@Transactional(value="compactor")
Set<CompactionInfo> findPotentialCompactions(int abortedThreshold,
                                                                               long abortedTimeThreshold,
                                                                               long lastChecked)
                                                                        throws MetaException

Throws:

MetaException

updateCompactorState

@Transactional(value="compactor")
void updateCompactorState(CompactionInfo ci,
                                                            long compactionTxnId)
                                                     throws MetaException

This updates COMPACTION_QUEUE. Set runAs username for the case where the request was generated by the user and so the worker must set this value later. Sets highestWriteId so that cleaner doesn't clean above what compactor has processed. Updates TXN_COMPONENTS so that we know where compactionTxnId was writing to in case it aborts.

Parameters:

compactionTxnId - - txnid in which Compactor is running

Throws:

MetaException

findNextToCompact

@Deprecated
 @Transactional(value="compactor")
CompactionInfo findNextToCompact(String workerId)
                                                                         throws MetaException

Deprecated. Replaced by findNextToCompact(org.apache.hadoop.hive.metastore.api.FindNextCompactRequest)

This will grab the next compaction request off of the queue, and assign it to the worker.

Parameters:

workerId - id of the worker calling this, will be recorded in the db

Returns:

an info element for this compaction request, or null if there is no work to do now.

Throws:

MetaException

findNextToCompact

@Transactional(value="compactor")
CompactionInfo findNextToCompact(FindNextCompactRequest rqst)
                                                            throws MetaException

This will grab the next compaction request off of the queue, and assign it to the worker.

Parameters:

rqst - request to find next compaction to run

Returns:

an info element for next compaction in the queue, or null if there is no work to do now.

Throws:

MetaException

markCompacted

@Transactional(value="compactor")
void markCompacted(CompactionInfo info)
                                              throws MetaException

This will mark an entry in the queue as compacted and put it in the ready to clean state.

Parameters:

info - info on the compaction entry to mark as compacted.

Throws:

MetaException

findReadyToClean

@Transactional(value="compactor")
List<CompactionInfo> findReadyToClean(long minOpenTxnWaterMark,
                                                                        long retentionTime)
                                                                 throws MetaException

Find entries in the queue that are ready to be cleaned.

Parameters:

minOpenTxnWaterMark - Minimum open txnId

retentionTime - Milliseconds to delay the cleaner

Returns:

information on the entry in the queue.

Throws:

MetaException

findReadyToCleanAborts

@Transactional(value="compactor")
List<CompactionInfo> findReadyToCleanAborts(long abortedTimeThreshold,
                                                                              int abortedThreshold)
                                                                       throws MetaException

Find the aborted entries in TXN_COMPONENTS which can be used to clean directories belonging to transactions in aborted state.

Parameters:

abortedTimeThreshold - Age of table/partition's oldest aborted transaction involving a given table or partition that will trigger cleanup.

abortedThreshold - Number of aborted transactions involving a given table or partition that will trigger cleanup.

Returns:

Information of potential abort items that needs to be cleaned.

Throws:

MetaException

markCleanerStart

@Transactional(value="compactor")
void markCleanerStart(CompactionInfo info)
                                                 throws MetaException

Sets the cleaning start time for a particular compaction

Parameters:

info - info on the compaction entry

Throws:

MetaException

clearCleanerStart

@Transactional(value="compactor")
void clearCleanerStart(CompactionInfo info)
                                                  throws MetaException

Removes the cleaning start time for a particular compaction

Parameters:

info - info on the compaction entry

Throws:

MetaException

markCleaned

@Transactional(value="compactor")
void markCleaned(CompactionInfo info)
                                            throws MetaException

This will remove an entry from the queue after it has been compacted.

Parameters:

info - info on the compaction entry to remove

Throws:

MetaException

markFailed

@Transactional(value="compactor")
void markFailed(CompactionInfo info)
                                           throws MetaException

Mark a compaction entry as failed. This will move it to the compaction history queue with a failed status. It will NOT clean up aborted transactions in the table/partition associated with this compaction.

Parameters:

info - information on the compaction that failed.

Throws:

MetaException

markRefused

@Transactional(value="compactor")
void markRefused(CompactionInfo info)
                                            throws MetaException

Mark a compaction as refused (to run). This can happen if a manual compaction is requested by the user, but for some reason, the table is not suitable for compation.

Parameters:

info - compaction job.

Throws:

MetaException

setCleanerRetryRetentionTimeOnError

@Transactional(value="compactor")
void setCleanerRetryRetentionTimeOnError(CompactionInfo info)
                                                                    throws MetaException

Stores the value of CompactionInfo.retryRetention and CompactionInfo.errorMessage fields of the CompactionInfo either by inserting or updating the fields in the HMS database.

Parameters:

info - The CompactionInfo object holding the values.

Throws:

MetaException

cleanTxnToWriteIdTable

@Transactional(value="compactor")
void cleanTxnToWriteIdTable()
                                                       throws MetaException

Clean up entries from TXN_TO_WRITE_ID table less than min_uncommited_txnid as found by min(max(TXNS.txn_id), min(WRITE_SET.WS_COMMIT_ID), min(Aborted TXNS.txn_id)).

Throws:

MetaException

removeDuplicateCompletedTxnComponents

@Transactional(value="compactor")
void removeDuplicateCompletedTxnComponents()
                                                                      throws MetaException

De-duplicate entries from COMPLETED_TXN_COMPONENTS table.

Throws:

MetaException

cleanEmptyAbortedAndCommittedTxns

@Transactional(value="compactor")
void cleanEmptyAbortedAndCommittedTxns()
                                                                  throws MetaException

Clean up aborted or committed transactions from txns that have no components in txn_components. The reason such txns exist can be that no work was done in this txn (e.g. Streaming opened TransactionBatch and abandoned it w/o doing any work) or due to markCleaned(CompactionInfo) being called, or the delete from the txns was delayed because of TXN_OPENTXN_TIMEOUT window.

Throws:

MetaException

revokeFromLocalWorkers

@Transactional(value="compactor")
void revokeFromLocalWorkers(String hostname)
                                                       throws MetaException

This will take all entries assigned to workers on a host return them to INITIATED state. The initiator should use this at start up to clean entries from any workers that were in the middle of compacting when the metastore shutdown. It does not reset entries from worker threads on other hosts as those may still be working.

Parameters:

hostname - Name of this host. It is assumed this prefixes the thread's worker id, so that like hostname% will match the worker id.

Throws:

MetaException

revokeTimedoutWorkers

@Transactional(value="compactor")
void revokeTimedoutWorkers(long timeout)
                                                      throws MetaException

This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state. This should be called by the initiator on start up and occasionally when running to clean up after dead threads. At start up revokeFromLocalWorkers(String) should be called first.

Parameters:

timeout - number of milliseconds since start time that should elapse before a worker is declared dead.

Throws:

MetaException

findColumnsWithStats

@Transactional(value="compactor")
List<String> findColumnsWithStats(CompactionInfo ci)
                                                             throws MetaException

Queries metastore DB directly to find columns in the table which have statistics information. If ci includes partition info then per partition stats info is examined, otherwise table level stats are examined.

Throws:

MetaException

purgeCompactionHistory

@Transactional(value="compactor")
void purgeCompactionHistory()
                                                       throws MetaException

For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example. The idea is to retain the tail (most recent) of the history such that a configurable number of each type of state is present. Any other entries can be purged. This scheme has advantage of always retaining the last failure/success even if it's not recent.

Throws:

MetaException

performWriteSetGC

@Transactional(value="txnhandler")
void performWriteSetGC()
                                                   throws MetaException

WriteSet tracking is used to ensure proper transaction isolation. This method deletes the transaction metadata once it becomes unnecessary.

Throws:

MetaException

checkFailedCompactions

@Transactional(value="compactor")
boolean checkFailedCompactions(CompactionInfo ci)
                                                          throws MetaException

Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled. User initiated compactions do not do this check.

Parameters:

ci - Table or partition to check.

Returns:

true if it is ok to compact, false if there have been too many failures.

Throws:

MetaException

getNumLocks

@Transactional(value="txnhandler")
int getNumLocks()
                                            throws SQLException,
                                                   MetaException

Throws:

SQLException

MetaException

setTimeout

long setTimeout(long milliseconds)

getOpenTxnTimeOutMillis

long getOpenTxnTimeOutMillis()

setOpenTxnTimeOutMillis

void setOpenTxnTimeOutMillis(long openTxnTimeOutMillis)

getMutexAPI

TxnStore.MutexAPI getMutexAPI()

setHadoopJobId

@Transactional(value="compactor")
void setHadoopJobId(String hadoopJobId,
                                                      long id)
                                               throws MetaException

Once a ThreadPoolExecutor Worker submits a job to the cluster, it calls this to update the metadata.

Parameters:

id - CompactionInfo.id

Throws:

MetaException

addWriteNotificationLog

@Transactional(value="compactor")
void addWriteNotificationLog(ListenerEvent acidWriteEvent)
                                                        throws MetaException

Add the ACID write event information to writeNotificationLog table.

Parameters:

acidWriteEvent -

Throws:

MetaException

findMinOpenTxnIdForCleaner

@Transactional(value="compactor")
long findMinOpenTxnIdForCleaner()
                                                           throws MetaException

Return the currently seen minimum open transaction ID.

Returns:

minimum transaction ID

Throws:

MetaException

getCompactionByTxnId

@Transactional(value="compactor")
Optional<CompactionInfo> getCompactionByTxnId(long txnId)
                                                                         throws MetaException

Returns the compaction running in the transaction txnId

Parameters:

txnId - transaction Id

Returns:

compaction info

Throws:

MetaException - ex

findMinTxnIdSeenOpen

@Transactional(value="compactor")
 @Deprecated
long findMinTxnIdSeenOpen()
                                                                  throws MetaException

Deprecated. remove when min_history_level table is dropped

Returns the smallest txnid that could be seen in open state across all active transactions in the system or -1 if there are no active transactions.

Returns:

transaction ID

Throws:

MetaException

getMetricsInfo

@Transactional(value="txnhandler")
MetricsInfo getMetricsInfo()
                                                       throws MetaException

Returns ACID metadata related metrics info.

Returns:

metrics info object

Throws:

MetaException

getCompactionMetricsData

@Transactional(value="compactor")
CompactionMetricsData getCompactionMetricsData(String dbName,
                                                                                 String tblName,
                                                                                 String partitionName,
                                                                                 CompactionMetricsData.MetricType type)
                                                                          throws MetaException

Returns ACID metrics related info for a specific resource and metric type. If no record is found matching the filter criteria, null will be returned.

Parameters:

dbName - name of database, non-null

tblName - name of the table, non-null

partitionName - name of the partition, can be null

type - type of the delta metric, non-null

Returns:

instance of delta metrics info, can be null

Throws:

MetaException

removeCompactionMetricsData

@Transactional(value="compactor")
void removeCompactionMetricsData(String dbName,
                                                                   String tblName,
                                                                   String partitionName,
                                                                   CompactionMetricsData.MetricType type)
                                                            throws MetaException

Remove records from the compaction metrics cache matching the filter criteria passed in as parameters

Parameters:

dbName - name of the database, non-null

tblName - name of the table, non-null

partitionName - name of the partition, non-null

type - type of the delta metric, non-null

Throws:

MetaException

getTopCompactionMetricsDataPerType

@Transactional(value="compactor")
List<CompactionMetricsData> getTopCompactionMetricsDataPerType(int limit)
                                                                                          throws MetaException

Returns the top ACID metrics from each type CompactionMetricsData.MetricType

Parameters:

limit - number of returned records for each type

Returns:

list of metrics, always non-null

Throws:

MetaException

updateCompactionMetricsData

@Transactional(value="compactor")
boolean updateCompactionMetricsData(CompactionMetricsData data)
                                                               throws MetaException

Create, update or delete one record in the compaction metrics cache.

If the metric is not found in the metrics cache, it will be created.

If the metric is found, it will be updated. This operation uses an optimistic locking mechanism, meaning if another operation changed the value of this metric, the update will abort and won't be retried.

If the new metric value is below CompactionMetricsData.getThreshold(), it will be deleted.

Parameters:

data - the object that is used for the operation

Returns:

true, if update finished successfully

Throws:

MetaException