org.apache.flink.hadoopcompatibility.mapred.record

Class HadoopDataSink<K,V>

java.lang.Object

org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Nothing>

org.apache.flink.api.common.operators.base.GenericDataSinkBase<org.apache.flink.types.Record>

org.apache.flink.api.java.record.operators.GenericDataSink

org.apache.flink.hadoopcompatibility.mapred.record.HadoopDataSink<K,V>

All Implemented Interfaces:

org.apache.flink.compiler.contextcheck.Validatable, org.apache.flink.util.Visitable<org.apache.flink.api.common.operators.Operator<?>>

public class HadoopDataSink<K,V>
extends GenericDataSink
implements org.apache.flink.compiler.contextcheck.Validatable

The HadoopDataSink is a generic wrapper for all Hadoop OutputFormats. Example usage:

                HadoopDataSink out = new HadoopDataSink(new org.apache.hadoop.mapred.TextOutputFormat(), new JobConf(), "Hadoop TextOutputFormat",reducer, Text.class,IntWritable.class);
                org.apache.hadoop.mapred.TextOutputFormat.setOutputPath(out.getJobConf(), new Path(output));
 

Note that it is possible to provide custom data type converter. The HadoopDataSink provides a default converter: DefaultFlinkTypeConverter

Field Summary

Fields inherited from class org.apache.flink.api.common.operators.base.GenericDataSinkBase

formatWrapper, input

Fields inherited from class org.apache.flink.api.common.operators.Operator

compilerHints, name, operatorInfo, parameters

Constructor Summary

Constructors

Constructor and Description
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input, FlinkTypeConverter<K,V> conv, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input, FlinkTypeConverter<K,V> conv, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input, Class<K> keyClass, Class<V> valueClass)

Method Summary

Methods

Modifier and Type	Method and Description
void	check()
org.apache.hadoop.mapred.JobConf	getJobConf()

Methods inherited from class org.apache.flink.api.java.record.operators.GenericDataSink

addInput, addInputs, setInputs, setInputs

Methods inherited from class org.apache.flink.api.common.operators.base.GenericDataSinkBase

accept, getDataDistribution, getFormatWrapper, getInput, getLocalOrder, getPartitionOrdering, getUserCodeWrapper, setGlobalOrder, setGlobalOrder, setInput, setLocalOrder, setRangePartitioned, setRangePartitioned, toString

Methods inherited from class org.apache.flink.api.common.operators.Operator

createUnionCascade, createUnionCascade, createUnionCascade, getCompilerHints, getDegreeOfParallelism, getName, getOperatorInfo, getParameters, setDegreeOfParallelism, setName, setParameter, setParameter, setParameter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              String name,
              org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input,
              FlinkTypeConverter<K,V> conv,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              String name,
              org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record> input,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              String name,
              List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input,
              FlinkTypeConverter<K,V> conv,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              String name,
              List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              org.apache.hadoop.mapred.JobConf jobConf,
              List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input,
              Class<K> keyClass,
              Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
              List<org.apache.flink.api.common.operators.Operator<org.apache.flink.types.Record>> input,
              Class<K> keyClass,
              Class<V> valueClass)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

check

public void check()

Specified by:

check in interface org.apache.flink.compiler.contextcheck.Validatable