org.apache.flink.runtime.fs.hdfs

类 HadoopDataInputStream

java.lang.Object

java.io.InputStream

org.apache.flink.core.fs.FSDataInputStream

org.apache.flink.runtime.fs.hdfs.HadoopDataInputStream

所有已实现的接口:

Closeable, AutoCloseable

public final class HadoopDataInputStream
extends org.apache.flink.core.fs.FSDataInputStream

Concrete implementation of the FSDataInputStream for Hadoop's input streams. This supports all file systems supported by Hadoop, such as HDFS and S3 (S3a/S3n).

字段概要

字段

限定符和类型	字段和说明
static int	MIN_SKIP_BYTES Minimum amount of bytes to skip forward before we issue a seek instead of discarding read.

构造器概要

构造器

构造器和说明
HadoopDataInputStream(org.apache.hadoop.fs.FSDataInputStream fsDataInputStream) Creates a new data input stream from the given Hadoop input stream.

方法概要

限定符和类型	方法和说明
int	available()
void	close()
void	forceSeek(long seekPos) Positions the stream to the given location.
org.apache.hadoop.fs.FSDataInputStream	getHadoopInputStream() Gets the wrapped Hadoop input stream.
long	getPos()
int	read()
int	read(byte[] buffer, int offset, int length)
void	seek(long seekPos)
long	skip(long n)
void	skipFully(long bytes) Skips over a given amount of bytes in the stream.

从类继承的方法 java.io.InputStream

mark, markSupported, read, reset

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

MIN_SKIP_BYTES

public static final int MIN_SKIP_BYTES

Minimum amount of bytes to skip forward before we issue a seek instead of discarding read.

The current value is just a magic number. In the long run, this value could become configurable, but for now it is a conservative, relatively small value that should bring safe improvements for small skips (e.g. in reading meta data), that would hurt the most with frequent seeks.

The optimal value depends on the DFS implementation and configuration plus the underlying filesystem. For now, this number is chosen "big enough" to provide improvements for smaller seeks, and "small enough" to avoid disadvantages over real seeks. While the minimum should be the page size, a true optimum per system would be the amounts of bytes the can be consumed sequentially within the seektime. Unfortunately, seektime is not constant and devices, OS, and DFS potentially also use read buffers and read-ahead.

另请参阅:

常量字段值

构造器详细资料

HadoopDataInputStream

public HadoopDataInputStream(org.apache.hadoop.fs.FSDataInputStream fsDataInputStream)

Creates a new data input stream from the given Hadoop input stream.

参数:

fsDataInputStream - The Hadoop input stream

方法详细资料

seek

public void seek(long seekPos)
          throws IOException

指定者:

seek 在类中 org.apache.flink.core.fs.FSDataInputStream

抛出:

IOException

getPos

public long getPos()
            throws IOException

指定者:

getPos 在类中 org.apache.flink.core.fs.FSDataInputStream

抛出:

IOException

read

public int read()
         throws IOException

指定者:

read 在类中 InputStream

抛出:

IOException

close

public void close()
           throws IOException

指定者:

close 在接口中 Closeable

指定者:

close 在接口中 AutoCloseable

覆盖:

close 在类中 InputStream

抛出:

IOException

read

public int read(@Nonnull
                byte[] buffer,
                int offset,
                int length)
         throws IOException

覆盖:

read 在类中 InputStream

抛出:

IOException

available

public int available()
              throws IOException

覆盖:

available 在类中 InputStream

抛出:

IOException

skip

public long skip(long n)
          throws IOException

覆盖:

skip 在类中 InputStream

抛出:

IOException

getHadoopInputStream

public org.apache.hadoop.fs.FSDataInputStream getHadoopInputStream()

Gets the wrapped Hadoop input stream.

返回:

The wrapped Hadoop input stream.

forceSeek

public void forceSeek(long seekPos)
               throws IOException

Positions the stream to the given location. In contrast to seek(long), this method will always issue a "seek" command to the dfs and may not replace it by skip(long) for small seeks.

Notice that the underlying DFS implementation can still decide to do skip instead of seek.

参数:

seekPos - the position to seek to.

抛出:

IOException

skipFully

public void skipFully(long bytes)
               throws IOException

Skips over a given amount of bytes in the stream.

参数:

bytes - the number of bytes to skip.

抛出:

IOException