org.apache.flink.streaming.connectors.fs

Class AvroKeyValueSinkWriter<K,V>

java.lang.Object

org.apache.flink.streaming.connectors.fs.StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter<K,V>

All Implemented Interfaces:

Serializable, org.apache.flink.api.java.typeutils.InputTypeConfigurable, Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

public class AvroKeyValueSinkWriter<K,V>
extends StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>
implements Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>, org.apache.flink.api.java.typeutils.InputTypeConfigurable

Implementation of AvroKeyValue writer that can be used in Sink. Each entry would be wrapped in GenericRecord with key/value fields(same as in m/r lib)

Usage:

		RollingSink<Tuple2<Long , Long>> sink = new RollingSink<Tuple2<Long , Long>>("/tmp/path");
		sink.setBucketer(new DateTimeBucketer("yyyy-MM-dd/HH/mm/"));
		sink.setPendingSuffix(".avro");
		Map<String,String> properties = new HashMap<>();
		Schema longSchema = Schema.create(Type.LONG);
		String keySchema = longSchema.toString();
		String valueSchema = longSchema.toString();
		properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_KEY_SCHEMA, keySchema);
		properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_VALUE_SCHEMA, valueSchema);
		properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS, Boolean.toString(true));
		properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS_CODEC, DataFileConstants.SNAPPY_CODEC);
		
		sink.setWriter(new AvroSinkWriter<Long , Long>(properties));
		sink.setBatchSize(1024 * 1024 * 64); // this is 64 MB,

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroKeyValueSinkWriter.AvroKeyValue<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONF_COMPRESS
static String	CONF_COMPRESS_CODEC
static String	CONF_DEFLATE_LEVEL
static String	CONF_OUTPUT_KEY_SCHEMA
static String	CONF_OUTPUT_VALUE_SCHEMA
static String	CONF_XZ_LEVEL

Constructor Summary

Constructors

Constructor and Description
AvroKeyValueSinkWriter(Map<String,String> properties) C'tor for the writer

Method Summary

Modifier and Type	Method and Description
void	close() Closes the Writer.
Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>	duplicate() Duplicates the Writer.
long	flush() Flushes out any internally held data, and returns the offset that the file must be truncated to at recovery.
void	open(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Initializes the Writer for a newly opened bucket file.
void	setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)
void	write(org.apache.flink.api.java.tuple.Tuple2<K,V> element) Writes one element to the bucket file.

Methods inherited from class org.apache.flink.streaming.connectors.fs.StreamWriterBase

getPos, getStream, hflushOrSync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.connectors.fs.Writer

getPos

Field Detail

CONF_OUTPUT_KEY_SCHEMA

public static final String CONF_OUTPUT_KEY_SCHEMA

See Also:

Constant Field Values

CONF_OUTPUT_VALUE_SCHEMA

public static final String CONF_OUTPUT_VALUE_SCHEMA

See Also:

Constant Field Values

CONF_COMPRESS

public static final String CONF_COMPRESS

See Also:

Constant Field Values

CONF_COMPRESS_CODEC

public static final String CONF_COMPRESS_CODEC

See Also:

Constant Field Values

CONF_DEFLATE_LEVEL

public static final String CONF_DEFLATE_LEVEL

See Also:

Constant Field Values

CONF_XZ_LEVEL

public static final String CONF_XZ_LEVEL

See Also:

Constant Field Values

Constructor Detail

AvroKeyValueSinkWriter

public AvroKeyValueSinkWriter(Map<String,String> properties)

C'tor for the writer

You can provide different properties that will be used to configure avro key-value writer as simple properties map(see example above)

Parameters:

properties -

Method Detail

open

public void open(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path)
          throws IOException

Description copied from interface: Writer

Initializes the Writer for a newly opened bucket file. Any internal per-bucket initialization should be performed here.

Specified by:

open in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Overrides:

open in class StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Parameters:

fs - The FileSystem containing the newly opened file.

path - The Path of the newly opened file.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Closes the Writer. If the writer is already closed, no action will be taken. The call should close all state related to the current output file, including the output stream opened in open.

Specified by:

close in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Overrides:

close in class StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Throws:

IOException

flush

public long flush()
           throws IOException

Description copied from interface: Writer

Flushes out any internally held data, and returns the offset that the file must be truncated to at recovery.

Specified by:

flush in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Overrides:

flush in class StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Throws:

IOException

write

public void write(org.apache.flink.api.java.tuple.Tuple2<K,V> element)
           throws IOException

Description copied from interface: Writer

Writes one element to the bucket file.

Specified by:

write in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

Throws:

IOException

setInputType

public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)

Specified by:

setInputType in interface org.apache.flink.api.java.typeutils.InputTypeConfigurable

duplicate

public Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>> duplicate()

Description copied from interface: Writer

Duplicates the Writer. This is used to get one Writer for each parallel instance of the sink.

Specified by:

duplicate in interface Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>