org.apache.nifi.util.hive

Class HiveWriter

java.lang.Object

org.apache.nifi.util.hive.HiveWriter

public class HiveWriter
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static interface	HiveWriter.CallRunner<T> Simple interface whose call method is called by {#callWithTimeout} in a new thread inside a PrivilegedExceptionAction.run() call.
static class	HiveWriter.CommitFailure
static class	HiveWriter.ConnectFailure
static class	HiveWriter.Failure
static class	HiveWriter.TxnBatchFailure
static class	HiveWriter.TxnFailure
static class	HiveWriter.WriteFailure

Field Summary

Fields

Modifier and Type	Field and Description
private long	callTimeout
private ExecutorService	callTimeoutPool
protected boolean	closed
private org.apache.hive.hcatalog.streaming.StreamingConnection	connection
private org.apache.hive.hcatalog.streaming.HiveEndPoint	endPoint
private long	lastUsed
private static org.slf4j.Logger	LOG
private org.apache.hive.hcatalog.streaming.RecordWriter	recordWriter
private int	totalRecords
private org.apache.hive.hcatalog.streaming.TransactionBatch	txnBatch
private Object	txnBatchLock
private int	txnsPerBatch
private org.apache.hadoop.security.UserGroupInformation	ugi

Constructor Summary

Constructors

Constructor and Description
HiveWriter(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint, int txnsPerBatch, boolean autoCreatePartitions, long callTimeout, ExecutorService callTimeoutPool, org.apache.hadoop.security.UserGroupInformation ugi, org.apache.hadoop.hive.conf.HiveConf hiveConf)

Method Summary

Modifier and Type	Method and Description
void	abort() Aborts the current Txn and switches to next Txn.
protected void	abortTxn() Aborts current Txn in the txnBatch.
private <T> T	callWithTimeout(HiveWriter.CallRunner<T> callRunner) Execute the callable on a separate thread and wait for the completion for the specified amount of time in milliseconds.
protected static void	checkAndThrowInterruptedException() If the current thread has been interrupted, then throws an exception.
void	close() Close the Transaction Batch and connection
protected void	closeConnection()
protected void	closeTxnBatch()
protected void	commitTxn()
void	flush(boolean rollToNext) Commits the current Txn if totalRecordsPerTransaction > 0 .
void	flushAndClose() Flush and Close current transactionBatch.
private byte[]	generateRecord(List<String> tuple)
long	getLastUsed()
protected org.apache.hive.hcatalog.streaming.RecordWriter	getRecordWriter(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint, org.apache.hadoop.security.UserGroupInformation ugi, org.apache.hadoop.hive.conf.HiveConf hiveConf)
int	getTotalRecords() Returns totalRecords written so far in a transaction
void	heartBeat() Queues up a heartbeat request on the current and remaining txns using the heartbeatThdPool and returns immediately
protected org.apache.hive.hcatalog.streaming.StreamingConnection	newConnection(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint, boolean autoCreatePartitions, org.apache.hadoop.hive.conf.HiveConf conf, org.apache.hadoop.security.UserGroupInformation ugi)
protected void	nextTxn(boolean rollToNext) if there are remainingTransactions in current txnBatch, begins nextTransactions otherwise creates new txnBatch.
protected org.apache.hive.hcatalog.streaming.TransactionBatch	nextTxnBatch(org.apache.hive.hcatalog.streaming.RecordWriter recordWriter)
String	toString()
void	write(byte[] record) Write the record data to Hive

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

private static final org.slf4j.Logger LOG

endPoint

private final org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint

connection

private final org.apache.hive.hcatalog.streaming.StreamingConnection connection

txnsPerBatch

private final int txnsPerBatch

recordWriter

private final org.apache.hive.hcatalog.streaming.RecordWriter recordWriter

callTimeoutPool

private final ExecutorService callTimeoutPool

callTimeout

private final long callTimeout

txnBatchLock

private final Object txnBatchLock

ugi

private final org.apache.hadoop.security.UserGroupInformation ugi

txnBatch

private org.apache.hive.hcatalog.streaming.TransactionBatch txnBatch

lastUsed

private long lastUsed

closed

protected boolean closed

totalRecords

private int totalRecords

Constructor Detail

HiveWriter

public HiveWriter(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint,
                  int txnsPerBatch,
                  boolean autoCreatePartitions,
                  long callTimeout,
                  ExecutorService callTimeoutPool,
                  org.apache.hadoop.security.UserGroupInformation ugi,
                  org.apache.hadoop.hive.conf.HiveConf hiveConf)
           throws InterruptedException,
                  HiveWriter.ConnectFailure

Throws:

InterruptedException

HiveWriter.ConnectFailure

Method Detail

getRecordWriter

protected org.apache.hive.hcatalog.streaming.RecordWriter getRecordWriter(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint,
                                                                          org.apache.hadoop.security.UserGroupInformation ugi,
                                                                          org.apache.hadoop.hive.conf.HiveConf hiveConf)
                                                                   throws org.apache.hive.hcatalog.streaming.StreamingException,
                                                                          IOException,
                                                                          InterruptedException

Throws:

org.apache.hive.hcatalog.streaming.StreamingException

IOException

InterruptedException

toString

public String toString()

Overrides:

toString in class Object

write

public void write(byte[] record)
           throws HiveWriter.WriteFailure,
                  org.apache.hive.hcatalog.streaming.SerializationError,
                  InterruptedException

Write the record data to Hive

Throws:

IOException - if an error occurs during the write

InterruptedException - if the write operation is interrupted

HiveWriter.WriteFailure

org.apache.hive.hcatalog.streaming.SerializationError

flush

public void flush(boolean rollToNext)
           throws HiveWriter.CommitFailure,
                  HiveWriter.TxnBatchFailure,
                  HiveWriter.TxnFailure,
                  InterruptedException

Commits the current Txn if totalRecordsPerTransaction > 0 . If 'rollToNext' is true, will switch to next Txn in batch or to a new TxnBatch if current Txn batch is exhausted

Throws:

HiveWriter.CommitFailure

HiveWriter.TxnBatchFailure

HiveWriter.TxnFailure

InterruptedException

heartBeat

public void heartBeat()
               throws InterruptedException

Queues up a heartbeat request on the current and remaining txns using the heartbeatThdPool and returns immediately

Throws:

InterruptedException

getTotalRecords

public int getTotalRecords()

Returns totalRecords written so far in a transaction

flushAndClose

public void flushAndClose()
                   throws HiveWriter.TxnBatchFailure,
                          HiveWriter.TxnFailure,
                          HiveWriter.CommitFailure,
                          IOException,
                          InterruptedException

Flush and Close current transactionBatch.

Throws:

HiveWriter.TxnBatchFailure

HiveWriter.TxnFailure

HiveWriter.CommitFailure

IOException

InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException

Close the Transaction Batch and connection

Throws:

IOException - if an error occurs during close

InterruptedException - if the close operation is interrupted

closeConnection

protected void closeConnection()
                        throws InterruptedException

Throws:

InterruptedException

commitTxn

protected void commitTxn()
                  throws HiveWriter.CommitFailure,
                         InterruptedException

Throws:

HiveWriter.CommitFailure

InterruptedException

newConnection

protected org.apache.hive.hcatalog.streaming.StreamingConnection newConnection(org.apache.hive.hcatalog.streaming.HiveEndPoint endPoint,
                                                                               boolean autoCreatePartitions,
                                                                               org.apache.hadoop.hive.conf.HiveConf conf,
                                                                               org.apache.hadoop.security.UserGroupInformation ugi)
                                                                        throws InterruptedException,
                                                                               HiveWriter.ConnectFailure

Throws:

InterruptedException

HiveWriter.ConnectFailure

nextTxnBatch

protected org.apache.hive.hcatalog.streaming.TransactionBatch nextTxnBatch(org.apache.hive.hcatalog.streaming.RecordWriter recordWriter)
                                                                    throws InterruptedException,
                                                                           HiveWriter.TxnBatchFailure

Throws:

InterruptedException

HiveWriter.TxnBatchFailure

closeTxnBatch

protected void closeTxnBatch()
                      throws InterruptedException

Throws:

InterruptedException

abort

public void abort()
           throws org.apache.hive.hcatalog.streaming.StreamingException,
                  HiveWriter.TxnBatchFailure,
                  InterruptedException

Aborts the current Txn and switches to next Txn.

Throws:

org.apache.hive.hcatalog.streaming.StreamingException - if could not get new Transaction Batch, or switch to next Txn

HiveWriter.TxnBatchFailure

InterruptedException

abortTxn

protected void abortTxn()
                 throws InterruptedException

Aborts current Txn in the txnBatch.

Throws:

InterruptedException

nextTxn

protected void nextTxn(boolean rollToNext)
                throws org.apache.hive.hcatalog.streaming.StreamingException,
                       InterruptedException,
                       HiveWriter.TxnBatchFailure

if there are remainingTransactions in current txnBatch, begins nextTransactions otherwise creates new txnBatch.

Parameters:

rollToNext - Whether to roll to the next transaction batch

Throws:

org.apache.hive.hcatalog.streaming.StreamingException

InterruptedException

HiveWriter.TxnBatchFailure

checkAndThrowInterruptedException

protected static void checkAndThrowInterruptedException()
                                                 throws InterruptedException

If the current thread has been interrupted, then throws an exception.

Throws:

InterruptedException - uf the current thread has been interrupted

callWithTimeout

private <T> T callWithTimeout(HiveWriter.CallRunner<T> callRunner)
                       throws TimeoutException,
                              org.apache.hive.hcatalog.streaming.StreamingException,
                              InterruptedException

Execute the callable on a separate thread and wait for the completion for the specified amount of time in milliseconds. In case of timeout cancel the callable and throw an IOException

Throws:

TimeoutException

org.apache.hive.hcatalog.streaming.StreamingException

InterruptedException

getLastUsed

public long getLastUsed()

generateRecord

private byte[] generateRecord(List<String> tuple)