org.apache.flink.streaming.connectors.fs

Class SequenceFileWriter<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.flink.streaming.connectors.fs.SequenceFileWriter<K,V>

Type Parameters:

K - The type of the first tuple field.

V - The type of the second tuple field.

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.InputTypeConfigurable, Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

public class SequenceFileWriter<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>
extends Object
implements Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>, org.apache.flink.api.java.typeutils.InputTypeConfigurable

A Writer that writes the bucket files as Hadoop SequenceFiles. The input to the RollingSink must be a Tuple2 of two Hadopo Writables.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SequenceFileWriter() Creates a new SequenceFileWriter that writes sequence files without compression.
SequenceFileWriter(String compressionCodecName, org.apache.hadoop.io.SequenceFile.CompressionType compressionType) Creates a new SequenceFileWriter that writes sequence with the given compression codec and compression type.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the Writer.
Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>	duplicate() Duplicates the Writer.
void	flush() Flushes out any internally held data.
void	open(org.apache.hadoop.fs.FSDataOutputStream outStream) Initializes the Writer for a newly opened bucket file.
void	setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)
void	write(org.apache.flink.api.java.tuple.Tuple2<K,V> element) Writes one element to the bucket file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SequenceFileWriter

public SequenceFileWriter()

Creates a new SequenceFileWriter that writes sequence files without compression.

SequenceFileWriter

public SequenceFileWriter(String compressionCodecName,
                          org.apache.hadoop.io.SequenceFile.CompressionType compressionType)

Creates a new SequenceFileWriter that writes sequence with the given compression codec and compression type.

Parameters:

compressionCodecName - Name of a Hadoop Compression Codec.

compressionType - The compression type to use.

Method Detail

open

public void open(org.apache.hadoop.fs.FSDataOutputStream outStream)
          throws IOException

Description copied from interface: Writer

Initializes the Writer for a newly opened bucket file. Any internal per-bucket initialization should be performed here.

Specified by:

open in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Parameters:

outStream - The FSDataOutputStream for the newly opened file.

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from interface: Writer

Flushes out any internally held data.

Specified by:

flush in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Closes the Writer. This must not close the FSDataOutputStream that was handed in in the Writer.open(org.apache.hadoop.fs.FSDataOutputStream) method. Only internally held state should be closed.

Specified by:

close in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Throws:

IOException

write

public void write(org.apache.flink.api.java.tuple.Tuple2<K,V> element)
           throws IOException

Description copied from interface: Writer

Writes one element to the bucket file.

Specified by:

write in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Throws:

IOException

setInputType

public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)

Specified by:

setInputType in interface org.apache.flink.api.java.typeutils.InputTypeConfigurable

duplicate

public Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>> duplicate()

Description copied from interface: Writer

Duplicates the Writer. This is used to get one Writer for each parallel instance of the sink.

Specified by:

duplicate in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>