程序包 org.apache.pulsar.io.hdfs3.sink

类 HdfsAbstractSink<K,V>

java.lang.Object

org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink<K,V>

所有已实现的接口:

AutoCloseable, org.apache.pulsar.io.core.Sink<V>

直接已知子类:

HdfsAbstractSequenceFileSink, HdfsAbstractTextFileSink

public abstract class HdfsAbstractSink<K,V>
extends AbstractHdfsConnector
implements org.apache.pulsar.io.core.Sink<V>

A Simple abstract class for HDFS sink. Users need to implement extractKeyValue function to use this sink.

字段概要

字段

修饰符和类型	字段	说明
protected HdfsSinkConfig	hdfsSinkConfig
protected HdfsSyncThread<V>	syncThread
protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>>	unackedRecords

从类继承的字段 org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

构造器概要

构造器

构造器	说明
HdfsAbstractSink()

方法概要

修饰符和类型	方法	说明
void	close()
protected final void	connectToHdfs()
protected abstract void	createWriter()
abstract org.apache.pulsar.io.core.KeyValue<K,V>	extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)
protected org.apache.hadoop.fs.FSDataOutputStream	getHdfsStream()
protected final org.apache.hadoop.fs.FSDataOutputStreamBuilder	getOutputStreamBuilder()
protected final org.apache.hadoop.fs.Path	getPath()
protected final void	launchSyncThread()
void	open(Map<String,Object> config, org.apache.pulsar.io.core.SinkContext sinkContext)

从类继承的方法 org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.pulsar.io.core.Sink

write

字段详细资料

hdfsSinkConfig

protected HdfsSinkConfig hdfsSinkConfig

unackedRecords

protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>> unackedRecords

syncThread

protected HdfsSyncThread<V> syncThread

构造器详细资料

HdfsAbstractSink

public HdfsAbstractSink()

方法详细资料

extractKeyValue

public abstract org.apache.pulsar.io.core.KeyValue<K,V> extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)

createWriter

protected abstract void createWriter()
                              throws IOException

抛出:

IOException

open

public void open(Map<String,Object> config,
 org.apache.pulsar.io.core.SinkContext sinkContext)
          throws Exception

指定者:

open 在接口中 org.apache.pulsar.io.core.Sink<K>

抛出:

Exception

close

public void close()
           throws Exception

指定者:

close 在接口中 AutoCloseable

抛出:

Exception

connectToHdfs

protected final void connectToHdfs()
                            throws IOException

抛出:

IOException

getOutputStreamBuilder

protected final org.apache.hadoop.fs.FSDataOutputStreamBuilder getOutputStreamBuilder()
                                                                               throws IOException

抛出:

IOException

getHdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream getHdfsStream()
                                                         throws IllegalArgumentException,
IOException

抛出:

IllegalArgumentException

IOException

getPath

protected final org.apache.hadoop.fs.Path getPath()

launchSyncThread

protected final void launchSyncThread()
                               throws IOException

抛出:

IOException