org.apache.flink.streaming.connectors.fs

Class SequenceFileWriter<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.flink.streaming.connectors.fs.StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

org.apache.flink.streaming.connectors.fs.SequenceFileWriter<K,V>

Type Parameters:

K - The type of the first tuple field.

V - The type of the second tuple field.

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.InputTypeConfigurable, Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

public class SequenceFileWriter<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>
extends StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>
implements org.apache.flink.api.java.typeutils.InputTypeConfigurable

A Writer that writes the bucket files as Hadoop SequenceFiles. The input to the RollingSink must be a Tuple2 of two Hadopo Writables.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SequenceFileWriter() Creates a new SequenceFileWriter that writes sequence files without compression.
SequenceFileWriter(String compressionCodecName, org.apache.hadoop.io.SequenceFile.CompressionType compressionType) Creates a new SequenceFileWriter that writes sequence with the given compression codec and compression type.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the Writer.
Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>	duplicate() Duplicates the Writer.
void	open(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Initializes the Writer for a newly opened bucket file.
void	setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)
void	write(org.apache.flink.api.java.tuple.Tuple2<K,V> element) Writes one element to the bucket file.

Methods inherited from class org.apache.flink.streaming.connectors.fs.StreamWriterBase

flush, getPos, getStream, hflushOrSync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SequenceFileWriter

public SequenceFileWriter()

Creates a new SequenceFileWriter that writes sequence files without compression.

SequenceFileWriter

public SequenceFileWriter(String compressionCodecName,
                          org.apache.hadoop.io.SequenceFile.CompressionType compressionType)

Creates a new SequenceFileWriter that writes sequence with the given compression codec and compression type.

Parameters:

compressionCodecName - Name of a Hadoop Compression Codec.

compressionType - The compression type to use.

Method Detail

open

public void open(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path)
          throws IOException

Description copied from interface: Writer

Initializes the Writer for a newly opened bucket file. Any internal per-bucket initialization should be performed here.

Specified by:

open in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Overrides:

open in class StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Parameters:

fs - The FileSystem containing the newly opened file.

path - The Path of the newly opened file.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Closes the Writer. If the writer is already closed, no action will be taken. The call should close all state related to the current output file, including the output stream opened in open.

Specified by:

close in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Overrides:

close in class StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Throws:

IOException

write

public void write(org.apache.flink.api.java.tuple.Tuple2<K,V> element)
           throws IOException

Description copied from interface: Writer

Writes one element to the bucket file.

Specified by:

write in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>

Throws:

IOException

setInputType

public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)

Specified by:

setInputType in interface org.apache.flink.api.java.typeutils.InputTypeConfigurable

duplicate

public Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>> duplicate()

Description copied from interface: Writer

Duplicates the Writer. This is used to get one Writer for each parallel instance of the sink.

Specified by:

duplicate in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>