org.apache.flink.hadoopcompatibility.mapred.record

Class HadoopRecordOutputFormat<K,V>

java.lang.Object

org.apache.flink.hadoopcompatibility.mapred.record.HadoopRecordOutputFormat<K,V>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

public class HadoopRecordOutputFormat<K,V>
extends Object
implements org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
FlinkTypeConverter<K,V>	converter
HadoopFileOutputCommitter	fileOutputCommitterWrapper
org.apache.hadoop.mapred.OutputFormat<K,V>	hadoopOutputFormat
org.apache.hadoop.mapred.JobConf	jobConf
org.apache.hadoop.mapred.RecordWriter<K,V>	recordWriter

Constructor Summary

Constructors

Constructor and Description
HadoopRecordOutputFormat(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf job, FlinkTypeConverter<K,V> conv)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() commit the task by moving the output file out from the temporary directory.
void	configure(org.apache.flink.configuration.Configuration parameters)
org.apache.hadoop.mapred.OutputFormat<K,V>	getHadoopOutputFormat()
org.apache.hadoop.mapred.JobConf	getJobConf()
void	open(int taskNumber, int numTasks) create the temporary output file for hadoop RecordWriter.
void	setHadoopOutputFormat(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopOutputFormat)
void	setJobConf(org.apache.hadoop.mapred.JobConf job)
void	writeRecord(org.apache.flink.types.Record record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobConf

public org.apache.hadoop.mapred.JobConf jobConf

hadoopOutputFormat

public org.apache.hadoop.mapred.OutputFormat<K,V> hadoopOutputFormat

recordWriter

public org.apache.hadoop.mapred.RecordWriter<K,V> recordWriter

converter

public FlinkTypeConverter<K,V> converter

fileOutputCommitterWrapper

public HadoopFileOutputCommitter fileOutputCommitterWrapper

Constructor Detail

HadoopRecordOutputFormat

public HadoopRecordOutputFormat(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                        org.apache.hadoop.mapred.JobConf job,
                        FlinkTypeConverter<K,V> conv)

Method Detail

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

Specified by:

configure in interface org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

open

public void open(int taskNumber,
        int numTasks)
          throws IOException

create the temporary output file for hadoop RecordWriter.

Specified by:

open in interface org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

Parameters:

taskNumber - The number of the parallel instance.

numTasks - The number of parallel tasks.

Throws:

IOException

writeRecord

public void writeRecord(org.apache.flink.types.Record record)
                 throws IOException

Specified by:

writeRecord in interface org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

Throws:

IOException

close

public void close()
           throws IOException

commit the task by moving the output file out from the temporary directory.

Specified by:

close in interface org.apache.flink.api.common.io.OutputFormat<org.apache.flink.types.Record>

Throws:

IOException

setJobConf

public void setJobConf(org.apache.hadoop.mapred.JobConf job)

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

getHadoopOutputFormat

public org.apache.hadoop.mapred.OutputFormat<K,V> getHadoopOutputFormat()

setHadoopOutputFormat

public void setHadoopOutputFormat(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopOutputFormat)