Package org.apache.pulsar.io.hdfs2.sink.seq

Class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

java.lang.Object

org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink<K,V>

org.apache.pulsar.io.hdfs2.sink.seq.HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

Type Parameters:

K - - The incoming Key type

V - - The incoming Value type

HdfsK - - The HDFS Key type

HdfsV - - The HDFS Value type

All Implemented Interfaces:

AutoCloseable, org.apache.pulsar.io.core.Sink<V>

Direct Known Subclasses:

HdfsSequentialTextSink, HdfsTextSink

public abstract class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>
extends HdfsAbstractSink<K,V>
implements org.apache.pulsar.io.core.Sink<V>

HDFS Sink that writes it contents to HDFS as Sequence Files.

Field Summary

Fields

Modifier and Type	Field	Description
protected AtomicLong	counter
protected org.apache.hadoop.fs.FSDataOutputStream	hdfsStream
protected org.apache.hadoop.io.SequenceFile.Writer	writer

Fields inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

hdfsSinkConfig, syncThread, unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor	Description
HdfsAbstractSequenceFileSink()

Method Summary

Modifier and Type	Method	Description
void	close()
abstract org.apache.pulsar.io.core.KeyValue<HdfsK,HdfsV>	convert(org.apache.pulsar.io.core.KeyValue<K,V> kv)
protected void	createWriter()
protected List<org.apache.hadoop.io.SequenceFile.Writer.Option>	getOptions()
protected org.apache.hadoop.io.SequenceFile.Writer	getWriter()
void	write(org.apache.pulsar.functions.api.Record<V> record)

Methods inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

connectToHdfs, extractKeyValue, getHdfsStream, getPath, launchSyncThread, open

Methods inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

open

Field Details

counter

protected AtomicLong counter

hdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream hdfsStream

writer

protected org.apache.hadoop.io.SequenceFile.Writer writer

Constructor Details

HdfsAbstractSequenceFileSink

public HdfsAbstractSequenceFileSink()

Method Details

convert

public abstract org.apache.pulsar.io.core.KeyValue<HdfsK,HdfsV> convert(org.apache.pulsar.io.core.KeyValue<K,V> kv)

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Overrides:

close in class HdfsAbstractSink<K,V>

Throws:

Exception

createWriter

protected void createWriter()
                     throws IOException

Specified by:

createWriter in class HdfsAbstractSink<K,V>

Throws:

IOException

write

public void write(org.apache.pulsar.functions.api.Record<V> record)

Specified by:

write in interface org.apache.pulsar.io.core.Sink<K>

getWriter

protected org.apache.hadoop.io.SequenceFile.Writer getWriter()
                                                      throws IOException

Throws:

IOException

getOptions

protected List<org.apache.hadoop.io.SequenceFile.Writer.Option> getOptions()
                                                                    throws IllegalArgumentException,
IOException

Throws:

IllegalArgumentException

IOException