org.apache.hive.storage.jdbc.dao

Class GenericJdbcDatabaseAccessor

java.lang.Object

org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor

All Implemented Interfaces:

DatabaseAccessor

Direct Known Subclasses:

DB2DatabaseAccessor, DerbyDatabaseAccessor, HiveDatabaseAccessor, JethroDatabaseAccessor, MsSqlDatabaseAccessor, MySqlDatabaseAccessor, OracleDatabaseAccessor, PostgresDatabaseAccessor

public class GenericJdbcDatabaseAccessor
extends Object
implements DatabaseAccessor

A data accessor that should in theory work with all JDBC compliant database drivers.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	DBCP_CONFIG_PREFIX
protected DataSource	dbcpDataSource
protected static int	DEFAULT_FETCH_SIZE
protected static org.slf4j.Logger	LOGGER

Constructor Summary

Constructors

Constructor and Description
GenericJdbcDatabaseAccessor()

Method Summary

Modifier and Type	Method and Description
protected String	addBoundaryToQuery(String tableName, String sql, String partitionColumn, String lowerBound, String upperBound)
protected String	addLimitAndOffsetToQuery(String sql, int limit, int offset) Uses generic JDBC escape functions to add a limit and offset clause to a query string
protected String	addLimitToQuery(String sql, int limit)
protected void	cleanupResources(Connection conn, PreparedStatement ps, ResultSet rs)
protected String	constructQuery(String table, String[] columnNames) Constructs the query used as the prepared statement to insert data.
org.apache.commons.lang3.tuple.Pair<String,String>	getBounds(org.apache.hadoop.conf.Configuration conf, String partitionColumn, boolean retrieveMin, boolean retrieveMax)
List<String>	getColumnNames(org.apache.hadoop.conf.Configuration conf)
List<TypeInfo>	getColumnTypes(org.apache.hadoop.conf.Configuration conf) Returns a list of types for the columns in the specified configuration.
protected Properties	getConnectionPoolProperties(org.apache.hadoop.conf.Configuration conf)
protected Properties	getDefaultDBCPProperties()
protected int	getFetchSize(org.apache.hadoop.conf.Configuration conf)
protected String	getMetaDataQuery(String sql)
JdbcRecordIterator	getRecordIterator(org.apache.hadoop.conf.Configuration conf, String partitionColumn, String lowerBound, String upperBound, int limit, int offset)
org.apache.hadoop.mapreduce.RecordWriter	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
int	getTotalNumberOfRecords(org.apache.hadoop.conf.Configuration conf)
protected void	initializeDatabaseConnection(org.apache.hadoop.conf.Configuration conf)
boolean	needColumnQuote()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DBCP_CONFIG_PREFIX

protected static final String DBCP_CONFIG_PREFIX

See Also:

Constant Field Values

DEFAULT_FETCH_SIZE

protected static final int DEFAULT_FETCH_SIZE

See Also:

Constant Field Values

LOGGER

protected static final org.slf4j.Logger LOGGER

dbcpDataSource

protected DataSource dbcpDataSource

Constructor Detail

GenericJdbcDatabaseAccessor

public GenericJdbcDatabaseAccessor()

Method Detail

getColumnNames

public List<String> getColumnNames(org.apache.hadoop.conf.Configuration conf)
                            throws HiveJdbcDatabaseAccessException

Specified by:

getColumnNames in interface DatabaseAccessor

Throws:

HiveJdbcDatabaseAccessException

getColumnTypes

public List<TypeInfo> getColumnTypes(org.apache.hadoop.conf.Configuration conf)
                              throws HiveJdbcDatabaseAccessException

Description copied from interface: DatabaseAccessor

Returns a list of types for the columns in the specified configuration. The type must represent as close as possible the respective type of the column stored in the database. Since it does not exist an exact mapping between database types and Hive types the result is approximate. When it is not possible to derive a type for a given column the TypeInfoFactory.unknownTypeInfo is used. There is a one-to-one correspondence between the types returned in this method and the column names obtained with DatabaseAccessor.getColumnNames(Configuration). Implementors of the method can derive the types by querying the database, exploit the state of the accessor, or use the configuration.

Specified by:

getColumnTypes in interface DatabaseAccessor

Throws:

HiveJdbcDatabaseAccessException - if some error occurs while accessing the database

getMetaDataQuery

protected String getMetaDataQuery(String sql)

getTotalNumberOfRecords

public int getTotalNumberOfRecords(org.apache.hadoop.conf.Configuration conf)
                            throws HiveJdbcDatabaseAccessException

Specified by:

getTotalNumberOfRecords in interface DatabaseAccessor

Throws:

HiveJdbcDatabaseAccessException

getRecordIterator

public JdbcRecordIterator getRecordIterator(org.apache.hadoop.conf.Configuration conf,
                                            String partitionColumn,
                                            String lowerBound,
                                            String upperBound,
                                            int limit,
                                            int offset)
                                     throws HiveJdbcDatabaseAccessException

Specified by:

getRecordIterator in interface DatabaseAccessor

Throws:

HiveJdbcDatabaseAccessException

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                         throws IOException

Specified by:

getRecordWriter in interface DatabaseAccessor

Throws:

IOException

constructQuery

protected String constructQuery(String table,
                                String[] columnNames)

Constructs the query used as the prepared statement to insert data.

Parameters:

table - the table to insert into

columnNames - the columns to insert into

addLimitAndOffsetToQuery

protected String addLimitAndOffsetToQuery(String sql,
                                          int limit,
                                          int offset)

Uses generic JDBC escape functions to add a limit and offset clause to a query string

Parameters:

sql -

limit -

offset -

Returns:

addLimitToQuery

protected String addLimitToQuery(String sql,
                                 int limit)

addBoundaryToQuery

protected String addBoundaryToQuery(String tableName,
                                    String sql,
                                    String partitionColumn,
                                    String lowerBound,
                                    String upperBound)

cleanupResources

protected void cleanupResources(Connection conn,
                                PreparedStatement ps,
                                ResultSet rs)

initializeDatabaseConnection

protected void initializeDatabaseConnection(org.apache.hadoop.conf.Configuration conf)
                                     throws Exception

Throws:

Exception

getConnectionPoolProperties

protected Properties getConnectionPoolProperties(org.apache.hadoop.conf.Configuration conf)
                                          throws Exception

Throws:

Exception

getDefaultDBCPProperties

protected Properties getDefaultDBCPProperties()

getFetchSize

protected int getFetchSize(org.apache.hadoop.conf.Configuration conf)

getBounds

public org.apache.commons.lang3.tuple.Pair<String,String> getBounds(org.apache.hadoop.conf.Configuration conf,
                                                                    String partitionColumn,
                                                                    boolean retrieveMin,
                                                                    boolean retrieveMax)
                                                             throws HiveJdbcDatabaseAccessException

Specified by:

getBounds in interface DatabaseAccessor

Throws:

HiveJdbcDatabaseAccessException

needColumnQuote

public boolean needColumnQuote()

Specified by:

needColumnQuote in interface DatabaseAccessor