org.apache.flink.api.java.hadoop.mapred

Class HadoopOutputFormatBase<K,V,T>

java.lang.Object

org.apache.flink.api.common.io.RichOutputFormat<T>

org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase<T>

org.apache.flink.api.java.hadoop.mapred.HadoopOutputFormatBase<K,V,T>

Type Parameters:

K - Type of Key

V - Type of Value

T - Record type.

All Implemented Interfaces:

Serializable, FinalizeOnMaster, OutputFormat<T>

Direct Known Subclasses:

HadoopOutputFormat

@Internal
public abstract class HadoopOutputFormatBase<K,V,T>
extends HadoopOutputFormatCommonBase<T>
implements FinalizeOnMaster

Common base for the mapred HadoopOutputFormat wrappers. There are implementations for Java and Scala.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static Object	CLOSE_MUTEX
protected static Object	CONFIGURE_MUTEX
protected org.apache.hadoop.mapred.TaskAttemptContext	context
protected org.apache.hadoop.mapred.JobConf	jobConf
protected org.apache.hadoop.mapred.OutputFormat<K,V>	mapredOutputFormat
protected static Object	OPEN_MUTEX
protected org.apache.hadoop.mapred.OutputCommitter	outputCommitter
protected org.apache.hadoop.mapred.RecordWriter<K,V>	recordWriter

Fields inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

credentials

Constructor Summary

Constructors

Constructor and Description
HadoopOutputFormatBase(org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat, org.apache.hadoop.mapred.JobConf job)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() commit the task by moving the output file out from the temporary directory.
void	configure(Configuration parameters)
void	finalizeGlobal(int parallelism)
org.apache.hadoop.mapred.JobConf	getJobConf()
void	open(int taskNumber, int numTasks) create the temporary output file for hadoop RecordWriter.

Methods inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

read, write

Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

writeRecord

Field Detail

OPEN_MUTEX

protected static final Object OPEN_MUTEX

CONFIGURE_MUTEX

protected static final Object CONFIGURE_MUTEX

CLOSE_MUTEX

protected static final Object CLOSE_MUTEX

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

mapredOutputFormat

protected org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat

recordWriter

protected transient org.apache.hadoop.mapred.RecordWriter<K,V> recordWriter

outputCommitter

protected transient org.apache.hadoop.mapred.OutputCommitter outputCommitter

context

protected transient org.apache.hadoop.mapred.TaskAttemptContext context

Constructor Detail

HadoopOutputFormatBase

public HadoopOutputFormatBase(org.apache.hadoop.mapred.OutputFormat<K,V> mapredOutputFormat,
                      org.apache.hadoop.mapred.JobConf job)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

configure

public void configure(Configuration parameters)

Specified by:

configure in interface OutputFormat<T>

open

public void open(int taskNumber,
        int numTasks)
          throws IOException

create the temporary output file for hadoop RecordWriter.

Specified by:

open in interface OutputFormat<T>

Parameters:

taskNumber - The number of the parallel instance.

numTasks - The number of parallel tasks.

Throws:

IOException

close

public void close()
           throws IOException

commit the task by moving the output file out from the temporary directory.

Specified by:

close in interface OutputFormat<T>

Throws:

IOException

finalizeGlobal

public void finalizeGlobal(int parallelism)
                    throws IOException

Specified by:

finalizeGlobal in interface FinalizeOnMaster

Throws:

IOException