org.apache.flink.api.java.hadoop.mapreduce

Class HadoopOutputFormatBase<K,V,T>

java.lang.Object

org.apache.flink.api.common.io.RichOutputFormat<T>

org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase<T>

org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormatBase<K,V,T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.FinalizeOnMaster, org.apache.flink.api.common.io.OutputFormat<T>

Direct Known Subclasses:

HadoopOutputFormat

@Internal
public abstract class HadoopOutputFormatBase<K,V,T>
extends HadoopOutputFormatCommonBase<T>
implements org.apache.flink.api.common.io.FinalizeOnMaster

Base class shared between the Java and Scala API of Flink

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static Object	CLOSE_MUTEX
protected org.apache.hadoop.conf.Configuration	configuration
protected static Object	CONFIGURE_MUTEX
protected org.apache.hadoop.mapreduce.TaskAttemptContext	context
protected org.apache.hadoop.mapreduce.OutputFormat<K,V>	mapreduceOutputFormat
protected static Object	OPEN_MUTEX
protected org.apache.hadoop.mapreduce.OutputCommitter	outputCommitter
protected org.apache.hadoop.mapreduce.RecordWriter<K,V>	recordWriter
protected int	taskNumber

Fields inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

credentials

Constructor Summary

Constructors

Constructor and Description
HadoopOutputFormatBase(org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat, org.apache.hadoop.mapreduce.Job job)

Method Summary

Modifier and Type	Method and Description
void	close() commit the task by moving the output file out from the temporary directory.
void	configure(org.apache.flink.configuration.Configuration parameters)
void	finalizeGlobal(int parallelism)
org.apache.hadoop.conf.Configuration	getConfiguration()
void	open(int taskNumber, int numTasks) create the temporary output file for hadoop RecordWriter.

Methods inherited from class org.apache.flink.api.java.hadoop.common.HadoopOutputFormatCommonBase

read, write

Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

writeRecord

Field Detail

OPEN_MUTEX

protected static final Object OPEN_MUTEX

CONFIGURE_MUTEX

protected static final Object CONFIGURE_MUTEX

CLOSE_MUTEX

protected static final Object CLOSE_MUTEX

configuration

protected org.apache.hadoop.conf.Configuration configuration

mapreduceOutputFormat

protected org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat

recordWriter

protected transient org.apache.hadoop.mapreduce.RecordWriter<K,V> recordWriter

outputCommitter

protected transient org.apache.hadoop.mapreduce.OutputCommitter outputCommitter

context

protected transient org.apache.hadoop.mapreduce.TaskAttemptContext context

taskNumber

protected transient int taskNumber

Constructor Detail

HadoopOutputFormatBase

public HadoopOutputFormatBase(org.apache.hadoop.mapreduce.OutputFormat<K,V> mapreduceOutputFormat,
                              org.apache.hadoop.mapreduce.Job job)

Method Detail

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

configure

public void configure(org.apache.flink.configuration.Configuration parameters)

Specified by:

configure in interface org.apache.flink.api.common.io.OutputFormat<T>

open

public void open(int taskNumber,
                 int numTasks)
          throws IOException

create the temporary output file for hadoop RecordWriter.

Specified by:

open in interface org.apache.flink.api.common.io.OutputFormat<T>

Parameters:

taskNumber - The number of the parallel instance.

numTasks - The number of parallel tasks.

Throws:

IOException

close

public void close()
           throws IOException

commit the task by moving the output file out from the temporary directory.

Specified by:

close in interface org.apache.flink.api.common.io.OutputFormat<T>

Throws:

IOException

finalizeGlobal

public void finalizeGlobal(int parallelism)
                    throws IOException

Specified by:

finalizeGlobal in interface org.apache.flink.api.common.io.FinalizeOnMaster

Throws:

IOException