org.apache.flink.runtime.fs.hdfs

Class HadoopFileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

All Implemented Interfaces:

org.apache.flink.core.fs.HadoopFileSystemWrapper

public final class HadoopFileSystem
extends org.apache.flink.core.fs.FileSystem
implements org.apache.flink.core.fs.HadoopFileSystemWrapper

Concrete implementation of the FileSystem base class for the Hadoop File System. The class is a wrapper class which encapsulated the original Hadoop HDFS API. If no file system class is specified, the wrapper will automatically load the Hadoop distributed file system (HDFS).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

org.apache.flink.core.fs.FileSystem.FSKey, org.apache.flink.core.fs.FileSystem.WriteMode

Constructor Summary

Constructors

Constructor and Description
HadoopFileSystem(Class<? extends org.apache.hadoop.fs.FileSystem> fsClass) Creates a new DistributedFileSystem object to access HDFS

Method Summary

Modifier and Type	Method and Description
HadoopDataOutputStream	create(org.apache.flink.core.fs.Path f, boolean overwrite)
HadoopDataOutputStream	create(org.apache.flink.core.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
boolean	delete(org.apache.flink.core.fs.Path f, boolean recursive)
long	getDefaultBlockSize()
org.apache.flink.core.fs.BlockLocation[]	getFileBlockLocations(org.apache.flink.core.fs.FileStatus file, long start, long len)
org.apache.flink.core.fs.FileStatus	getFileStatus(org.apache.flink.core.fs.Path f)
static org.apache.hadoop.conf.Configuration	getHadoopConfiguration() Returns a new Hadoop Configuration object using the path to the hadoop conf configured in the main configuration (flink-conf.yaml).
org.apache.hadoop.fs.FileSystem	getHadoopFileSystem() Gets the underlying Hadoop FileSystem.
Class<?>	getHadoopWrapperClassNameForFileSystem(String scheme)
org.apache.flink.core.fs.Path	getHomeDirectory()
URI	getUri()
org.apache.flink.core.fs.Path	getWorkingDirectory()
void	initialize(URI path)
boolean	isDistributedFS()
org.apache.flink.core.fs.FileStatus[]	listStatus(org.apache.flink.core.fs.Path f)
boolean	mkdirs(org.apache.flink.core.fs.Path f)
HadoopDataInputStream	open(org.apache.flink.core.fs.Path f)
HadoopDataInputStream	open(org.apache.flink.core.fs.Path f, int bufferSize)
boolean	rename(org.apache.flink.core.fs.Path src, org.apache.flink.core.fs.Path dst)

Methods inherited from class org.apache.flink.core.fs.FileSystem

exists, get, getLocalFileSystem, initOutPathDistFS, initOutPathLocalFS, setDefaultScheme

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileSystem

public HadoopFileSystem(Class<? extends org.apache.hadoop.fs.FileSystem> fsClass)
                 throws IOException

Creates a new DistributedFileSystem object to access HDFS

Throws:

IOException - throw if the required HDFS classes cannot be instantiated

Method Detail

getHadoopConfiguration

public static org.apache.hadoop.conf.Configuration getHadoopConfiguration()

Returns a new Hadoop Configuration object using the path to the hadoop conf configured in the main configuration (flink-conf.yaml). This method is public because its being used in the HadoopDataSource.

getWorkingDirectory

public org.apache.flink.core.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.flink.core.fs.FileSystem

getHomeDirectory

public org.apache.flink.core.fs.Path getHomeDirectory()

Specified by:

getHomeDirectory in class org.apache.flink.core.fs.FileSystem

getUri

public URI getUri()

Specified by:

getUri in class org.apache.flink.core.fs.FileSystem

getHadoopFileSystem

public org.apache.hadoop.fs.FileSystem getHadoopFileSystem()

Gets the underlying Hadoop FileSystem.

Returns:

The underlying Hadoop FileSystem.

initialize

public void initialize(URI path)
                throws IOException

Specified by:

initialize in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.flink.core.fs.FileStatus getFileStatus(org.apache.flink.core.fs.Path f)
                                                  throws IOException

Specified by:

getFileStatus in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.flink.core.fs.BlockLocation[] getFileBlockLocations(org.apache.flink.core.fs.FileStatus file,
                                                                      long start,
                                                                      long len)
                                                               throws IOException

Specified by:

getFileBlockLocations in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(org.apache.flink.core.fs.Path f,
                                  int bufferSize)
                           throws IOException

Specified by:

open in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(org.apache.flink.core.fs.Path f)
                           throws IOException

Specified by:

open in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f,
                                     boolean overwrite,
                                     int bufferSize,
                                     short replication,
                                     long blockSize)
                              throws IOException

Specified by:

create in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(org.apache.flink.core.fs.Path f,
                                     boolean overwrite)
                              throws IOException

Specified by:

create in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.flink.core.fs.Path f,
                      boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.flink.core.fs.FileStatus[] listStatus(org.apache.flink.core.fs.Path f)
                                                 throws IOException

Specified by:

listStatus in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.flink.core.fs.Path f)
               throws IOException

Specified by:

mkdirs in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.flink.core.fs.Path src,
                      org.apache.flink.core.fs.Path dst)
               throws IOException

Specified by:

rename in class org.apache.flink.core.fs.FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.flink.core.fs.FileSystem

isDistributedFS

public boolean isDistributedFS()

Specified by:

isDistributedFS in class org.apache.flink.core.fs.FileSystem

getHadoopWrapperClassNameForFileSystem

public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme)

Specified by:

getHadoopWrapperClassNameForFileSystem in interface org.apache.flink.core.fs.HadoopFileSystemWrapper