org.apache.flink.streaming.connectors.fs

类 AvroKeyValueSinkWriter<K,V>

java.lang.Object

org.apache.flink.streaming.connectors.fs.StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

org.apache.flink.streaming.connectors.fs.AvroKeyValueSinkWriter<K,V>

所有已实现的接口:

Serializable, org.apache.flink.api.java.typeutils.InputTypeConfigurable, Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

public class AvroKeyValueSinkWriter<K,V>
extends StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>
implements Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>, org.apache.flink.api.java.typeutils.InputTypeConfigurable

Implementation of AvroKeyValue writer that can be used in Sink. Each entry would be wrapped in GenericRecord with key/value fields(same as in m/r lib)

Usage:

		BucketingSink<Tuple2<Long, Long>> sink = new BucketingSink<Tuple2<Long, Long>>("/tmp/path");
		sink.setBucketer(new DateTimeBucketer<Tuple2<Long, Long>>("yyyy-MM-dd/HH/mm/"));
		sink.setPendingSuffix(".avro");
		Map<String, String> properties = new HashMap<>();
		Schema longSchema = Schema.create(Type.LONG);
		String keySchema = longSchema.toString();
		String valueSchema = longSchema.toString();
		properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_KEY_SCHEMA, keySchema);
		properties.put(AvroKeyValueSinkWriter.CONF_OUTPUT_VALUE_SCHEMA, valueSchema);
		properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS, Boolean.toString(true));
		properties.put(AvroKeyValueSinkWriter.CONF_COMPRESS_CODEC, DataFileConstants.SNAPPY_CODEC);

		sink.setWriter(new AvroSinkWriter<Long, Long>(properties));
		sink.setBatchSize(1024 * 1024 * 64); // this is 64 MB,

另请参阅:

序列化表格

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	AvroKeyValueSinkWriter.AvroKeyValue<K,V> A reusable Avro generic record for writing key/value pairs to the file.

字段概要

字段

限定符和类型	字段和说明
static String	CONF_COMPRESS
static String	CONF_COMPRESS_CODEC
static String	CONF_DEFLATE_LEVEL
static String	CONF_OUTPUT_KEY_SCHEMA
static String	CONF_OUTPUT_VALUE_SCHEMA
static String	CONF_XZ_LEVEL

构造器概要

构造器

限定符	构造器和说明
protected	AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)
	AvroKeyValueSinkWriter(Map<String,String> properties) C'tor for the writer.

方法概要

限定符和类型	方法和说明
void	close() Closes the Writer.
AvroKeyValueSinkWriter<K,V>	duplicate() Duplicates the Writer.
long	flush() Flushes out any internally held data, and returns the offset that the file must be truncated to at recovery.
void	open(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Initializes the Writer for a newly opened bucket file.
void	setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)
void	write(org.apache.flink.api.java.tuple.Tuple2<K,V> element) Writes one element to the bucket file.

从类继承的方法 org.apache.flink.streaming.connectors.fs.StreamWriterBase

getPos, getStream, isSyncOnFlush, setSyncOnFlush

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.connectors.fs.Writer

getPos

字段详细资料

CONF_OUTPUT_KEY_SCHEMA

public static final String CONF_OUTPUT_KEY_SCHEMA

另请参阅:

常量字段值

CONF_OUTPUT_VALUE_SCHEMA

public static final String CONF_OUTPUT_VALUE_SCHEMA

另请参阅:

常量字段值

CONF_COMPRESS

public static final String CONF_COMPRESS

另请参阅:

常量字段值

CONF_COMPRESS_CODEC

public static final String CONF_COMPRESS_CODEC

另请参阅:

常量字段值

CONF_DEFLATE_LEVEL

public static final String CONF_DEFLATE_LEVEL

另请参阅:

常量字段值

CONF_XZ_LEVEL

public static final String CONF_XZ_LEVEL

另请参阅:

常量字段值

构造器详细资料

AvroKeyValueSinkWriter

public AvroKeyValueSinkWriter(Map<String,String> properties)

C'tor for the writer.

You can provide different properties that will be used to configure avro key-value writer as simple properties map(see example above)

参数:

properties -

AvroKeyValueSinkWriter

protected AvroKeyValueSinkWriter(AvroKeyValueSinkWriter<K,V> other)

方法详细资料

open

public void open(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path)
          throws IOException

从接口复制的说明: Writer

Initializes the Writer for a newly opened bucket file. Any internal per-bucket initialization should be performed here.

指定者:

open 在接口中 Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

覆盖:

open 在类中 StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

参数:

fs - The FileSystem containing the newly opened file.

path - The Path of the newly opened file.

抛出:

IOException

close

public void close()
           throws IOException

从接口复制的说明: Writer

Closes the Writer. If the writer is already closed, no action will be taken. The call should close all state related to the current output file, including the output stream opened in open.

指定者:

close 在接口中 Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

覆盖:

close 在类中 StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

抛出:

IOException

flush

public long flush()
           throws IOException

从接口复制的说明: Writer

Flushes out any internally held data, and returns the offset that the file must be truncated to at recovery.

指定者:

flush 在接口中 Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

覆盖:

flush 在类中 StreamWriterBase<org.apache.flink.api.java.tuple.Tuple2<K,V>>

抛出:

IOException

write

public void write(org.apache.flink.api.java.tuple.Tuple2<K,V> element)
           throws IOException

从接口复制的说明: Writer

Writes one element to the bucket file.

指定者:

write 在接口中 Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>

抛出:

IOException

setInputType

public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)

指定者:

setInputType 在接口中 org.apache.flink.api.java.typeutils.InputTypeConfigurable

duplicate

public AvroKeyValueSinkWriter<K,V> duplicate()

从接口复制的说明: Writer

Duplicates the Writer. This is used to get one Writer for each parallel instance of the sink.

指定者:

duplicate 在接口中 Writer<org.apache.flink.api.java.tuple.Tuple2<K,V>>