org.apache.flink.hadoopcompatibility.mapred.record

Class HadoopDataSource<K,V>

java.lang.Object

org.apache.flink.api.common.operators.Operator<OUT>

org.apache.flink.api.common.operators.base.GenericDataSourceBase<org.apache.flink.types.Record,T>

org.apache.flink.api.java.record.operators.GenericDataSource<HadoopRecordInputFormat<K,V>>

org.apache.flink.hadoopcompatibility.mapred.record.HadoopDataSource<K,V>

All Implemented Interfaces:

org.apache.flink.util.Visitable<org.apache.flink.api.common.operators.Operator<?>>

public class HadoopDataSource<K,V>
extends GenericDataSource<HadoopRecordInputFormat<K,V>>

The HadoopDataSource is a generic wrapper for all Hadoop InputFormats. Example usage:

                HadoopDataSource source = new HadoopDataSource(new org.apache.hadoop.mapred.TextInputFormat(), new JobConf(), "Input Lines");
                org.apache.hadoop.mapred.TextInputFormat.addInputPath(source.getJobConf(), new Path(dataInput));
 

Note that it is possible to provide custom data type converter. The HadoopDataSource provides two different standard converters: * WritableWrapperConverter: Converts Hadoop Types to a record that contains a WritableComparableWrapper (key) and a WritableWrapper * DefaultHadoopTypeConverter: Converts the standard hadoop types (longWritable, Text) to Flinks's Value types.

Field Summary

Fields inherited from class org.apache.flink.api.common.operators.base.GenericDataSourceBase

formatWrapper, statisticsKey

Fields inherited from class org.apache.flink.api.common.operators.Operator

compilerHints, name, operatorInfo, parameters

Constructor Summary

Constructors

Constructor and Description
HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat)
HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf)
HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name)
HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, HadoopTypeConverter<K,V> conv)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapred.JobConf	getJobConf()

Methods inherited from class org.apache.flink.api.common.operators.base.GenericDataSourceBase

accept, getFormatWrapper, getStatisticsKey, getUserCodeWrapper, setStatisticsKey, toString

Methods inherited from class org.apache.flink.api.common.operators.Operator

createUnionCascade, createUnionCascade, createUnionCascade, getCompilerHints, getDegreeOfParallelism, getName, getOperatorInfo, getParameters, setDegreeOfParallelism, setName, setParameter, setParameter, setParameter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

HadoopDataSource

public HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat,
                org.apache.hadoop.mapred.JobConf jobConf,
                String name,
                HadoopTypeConverter<K,V> conv)

Parameters:

hadoopFormat - Implementation of a Hadoop input format

jobConf - JobConf object (Hadoop)

name - Name of the DataSource

conv - Definition of a custom type converter DefaultHadoopTypeConverter.

HadoopDataSource

public HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat,
                org.apache.hadoop.mapred.JobConf jobConf,
                String name)

HadoopDataSource

public HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat,
                org.apache.hadoop.mapred.JobConf jobConf)

HadoopDataSource

public HadoopDataSource(org.apache.hadoop.mapred.InputFormat<K,V> hadoopFormat)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()