org.apache.flink.connector.hbase.source

Class HBaseRowInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>

org.apache.flink.connector.hbase.source.HBaseRowInputFormat

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<org.apache.flink.types.Row,org.apache.flink.connector.hbase.source.TableInputSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.types.Row>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.connector.hbase.source.TableInputSplit>

@Internal
public class HBaseRowInputFormat
extends org.apache.flink.api.common.io.RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.types.Row>

InputFormat subclass that wraps the access for HTables. Returns the result as Row

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected byte[]	currentRow
protected boolean	endReached
protected org.apache.hadoop.hbase.client.ResultScanner	resultScanner HBase iterator wrapper.
protected org.apache.hadoop.hbase.client.Scan	scan
protected long	scannedRows
protected byte[]	serializedConfig
protected org.apache.hadoop.hbase.client.HTable	table

Constructor Summary

Constructors

Constructor and Description
HBaseRowInputFormat(org.apache.hadoop.conf.Configuration conf, String tableName, HBaseTableSchema schema)

Method Summary

Modifier and Type	Method and Description
void	close()
void	closeInputFormat()
void	configure(org.apache.flink.configuration.Configuration parameters) Creates a Scan object and opens the HTable connection.
org.apache.flink.connector.hbase.source.TableInputSplit[]	createInputSplits(int minNumSplits)
protected org.apache.hadoop.conf.Configuration	getHadoopConfiguration()
org.apache.flink.core.io.InputSplitAssigner	getInputSplitAssigner(org.apache.flink.connector.hbase.source.TableInputSplit[] inputSplits)
org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row>	getProducedType()
protected org.apache.hadoop.hbase.client.Scan	getScanner() Returns an instance of Scan that retrieves the required subset of records from the HBase table.
org.apache.flink.api.common.io.statistics.BaseStatistics	getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)
String	getTableName() What table is to be read.
protected boolean	includeRegionInScan(byte[] startKey, byte[] endKey) Test if the given region is to be included in the scan while splitting the regions of a table.
protected org.apache.flink.types.Row	mapResultToOutType(org.apache.hadoop.hbase.client.Result res) HBase returns an instance of Result.
T	nextRecord(T reuse)
void	open(org.apache.flink.connector.hbase.source.TableInputSplit split)
boolean	reachedEnd()

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

endReached

protected boolean endReached

table

protected transient org.apache.hadoop.hbase.client.HTable table

scan

protected transient org.apache.hadoop.hbase.client.Scan scan

resultScanner

protected org.apache.hadoop.hbase.client.ResultScanner resultScanner

HBase iterator wrapper.

currentRow

protected byte[] currentRow

scannedRows

protected long scannedRows

serializedConfig

protected byte[] serializedConfig

Constructor Detail

HBaseRowInputFormat

public HBaseRowInputFormat(org.apache.hadoop.conf.Configuration conf,
                           String tableName,
                           HBaseTableSchema schema)

Method Detail

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

Creates a Scan object and opens the HTable connection.

These are opened here because they are needed in the createInputSplits which is called before the openInputFormat method.

The connection is opened in this method and closed in closeInputFormat().

Specified by:

configure in interface org.apache.flink.api.common.io.InputFormat<org.apache.flink.types.Row,org.apache.flink.connector.hbase.source.TableInputSplit>

Parameters:

parameters - The configuration that is to be used

See Also:

Configuration

getScanner

protected org.apache.hadoop.hbase.client.Scan getScanner()

Returns an instance of Scan that retrieves the required subset of records from the HBase table.

Returns:

The appropriate instance of Scan for this use case.

getTableName

public String getTableName()

What table is to be read.

Per instance of a TableInputFormat derivative only a single table name is possible.

Returns:

The name of the table

mapResultToOutType

protected org.apache.flink.types.Row mapResultToOutType(org.apache.hadoop.hbase.client.Result res)

HBase returns an instance of Result.

This method maps the returned Result instance into the output type T.

Parameters:

res - The Result instance from HBase that needs to be converted

Returns:

The appropriate instance of T that contains the data of Result.

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.types.Row>

getHadoopConfiguration

protected org.apache.hadoop.conf.Configuration getHadoopConfiguration()

open

public void open(org.apache.flink.connector.hbase.source.TableInputSplit split)
          throws IOException

Throws:

IOException

nextRecord

public T nextRecord(T reuse)
             throws IOException

Throws:

IOException

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

closeInputFormat

public void closeInputFormat()
                      throws IOException

Overrides:

closeInputFormat in class org.apache.flink.api.common.io.RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>

Throws:

IOException

createInputSplits

public org.apache.flink.connector.hbase.source.TableInputSplit[] createInputSplits(int minNumSplits)
                                                                            throws IOException

Throws:

IOException

includeRegionInScan

protected boolean includeRegionInScan(byte[] startKey,
                                      byte[] endKey)

Test if the given region is to be included in the scan while splitting the regions of a table.

Parameters:

startKey - Start key of the region

endKey - End key of the region

Returns:

true, if this region needs to be included as part of the input (default).

getInputSplitAssigner

public org.apache.flink.core.io.InputSplitAssigner getInputSplitAssigner(org.apache.flink.connector.hbase.source.TableInputSplit[] inputSplits)

getStatistics

public org.apache.flink.api.common.io.statistics.BaseStatistics getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)