org.apache.flink.core.fs

Class FileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

Direct Known Subclasses:

LocalFileSystem

@Public
public abstract class FileSystem
extends Object

An abstract base class for a fairly generic file system. It may be implemented as a distributed file system, or as a local one that reflects the locally-connected disk.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileSystem.FSKey An auxiliary class to identify a file system by its scheme and its authority.
static class	FileSystem.WriteMode Enumeration for write modes.

Constructor Summary

Constructors

Constructor and Description
FileSystem()

Method Summary

Modifier and Type	Method and Description
abstract FSDataOutputStream	create(Path f, boolean overwrite) Opens an FSDataOutputStream at the indicated Path.
abstract FSDataOutputStream	create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) Opens an FSDataOutputStream at the indicated Path.
abstract boolean	delete(Path f, boolean recursive) Delete a file.
boolean	exists(Path f) Check if exists.
static FileSystem	get(URI uri) Returns a reference to the FileSystem instance for accessing the file system identified by the given URI.
long	getDefaultBlockSize() Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
abstract BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
abstract FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
abstract Path	getHomeDirectory() Returns the path of the user's home directory in this file system.
static FileSystem	getLocalFileSystem() Returns a reference to the FileSystem instance for accessing the local file system.
abstract URI	getUri() Returns a URI whose scheme and authority identify this file system.
abstract Path	getWorkingDirectory() Returns the path of the file system's current working directory.
abstract void	initialize(URI name) Called after a new FileSystem instance is constructed.
boolean	initOutPathDistFS(Path outPath, FileSystem.WriteMode writeMode, boolean createDirectory) Initializes output directories on distributed file systems according to the given write mode.
boolean	initOutPathLocalFS(Path outPath, FileSystem.WriteMode writeMode, boolean createDirectory) Initializes output directories on local file systems according to the given write mode.
abstract boolean	isDistributedFS() Returns true if this is a distributed file system, false otherwise.
static boolean	isFlinkSupportedScheme(String scheme) Returns a boolean indicating whether a scheme has built-in Flink support.
abstract FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
abstract boolean	mkdirs(Path f) Make the given file and all non-existent parents into directories.
abstract FSDataInputStream	open(Path f) Opens an FSDataInputStream at the indicated Path.
abstract FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
abstract boolean	rename(Path src, Path dst) Renames the file/directory src to dst.
static void	setDefaultScheme(Configuration config) Sets the default filesystem scheme based on the user-specified configuration parameter fs.default-scheme.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileSystem

public FileSystem()

Method Detail

getLocalFileSystem

public static FileSystem getLocalFileSystem()

Returns a reference to the FileSystem instance for accessing the local file system.

Returns:

a reference to the FileSystem instance for accessing the local file system.

setDefaultScheme

public static void setDefaultScheme(Configuration config)
                             throws IOException

Sets the default filesystem scheme based on the user-specified configuration parameter fs.default-scheme. By default this is set to file:/// (see ConfigConstants.FILESYSTEM_SCHEME and ConfigConstants.DEFAULT_FILESYSTEM_SCHEME), and the local filesystem is used.

As an example, if set to hdfs://localhost:9000/, then an HDFS deployment with the namenode being on the local node and listening to port 9000 is going to be used. In this case, a file path specified as /user/USERNAME/in.txt is going to be transformed into hdfs://localhost:9000/user/USERNAME/in.txt. By default this is set to file:/// which points to the local filesystem.

Parameters:

config - the configuration from where to fetch the parameter.

Throws:

IOException

get

public static FileSystem get(URI uri)
                      throws IOException

Returns a reference to the FileSystem instance for accessing the file system identified by the given URI.

Parameters:

uri - the URI identifying the file system

Returns:

a reference to the FileSystem instance for accessing the file system identified by the given URI.

Throws:

IOException - thrown if a reference to the file system instance could not be obtained

isFlinkSupportedScheme

public static boolean isFlinkSupportedScheme(String scheme)

Returns a boolean indicating whether a scheme has built-in Flink support.

Parameters:

scheme - a file system scheme

Returns:

a boolean indicating whether the provided scheme has built-in Flink support

getWorkingDirectory

public abstract Path getWorkingDirectory()

Returns the path of the file system's current working directory.

Returns:

the path of the file system's current working directory

getHomeDirectory

public abstract Path getHomeDirectory()

Returns the path of the user's home directory in this file system.

Returns:

the path of the user's home directory in this file system.

getUri

public abstract URI getUri()

Returns a URI whose scheme and authority identify this file system.

Returns:

a URI whose scheme and authority identify this file system

initialize

public abstract void initialize(URI name)
                         throws IOException

Called after a new FileSystem instance is constructed.

Parameters:

name - a URI whose authority section names the host, port, etc. for this file system

Throws:

IOException

getFileStatus

public abstract FileStatus getFileStatus(Path f)
                                  throws IOException

Return a file status object that represents the path.

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getFileBlockLocations

public abstract BlockLocation[] getFileBlockLocations(FileStatus file,
                                                      long start,
                                                      long len)
                                               throws IOException

Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

Throws:

IOException

open

public abstract FSDataInputStream open(Path f,
                                       int bufferSize)
                                throws IOException

Opens an FSDataInputStream at the indicated Path.

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException

open

public abstract FSDataInputStream open(Path f)
                                throws IOException

Opens an FSDataInputStream at the indicated Path.

Parameters:

f - the file to open

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Return the number of bytes that large input files should be optimally be split into to minimize I/O time.

Returns:

the number of bytes that large input files should be optimally be split into to minimize I/O time

listStatus

public abstract FileStatus[] listStatus(Path f)
                                 throws IOException

List the statuses of the files/directories in the given path if the path is a directory.

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

IOException

exists

public boolean exists(Path f)
               throws IOException

Check if exists.

Parameters:

f - source file

Throws:

IOException

delete

public abstract boolean delete(Path f,
                               boolean recursive)
                        throws IOException

Delete a file.

Parameters:

f - the path to delete

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false

Returns:

true if delete is successful, false otherwise

Throws:

IOException

mkdirs

public abstract boolean mkdirs(Path f)
                        throws IOException

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Parameters:

f - the directory/directories to be created

Returns:

true if at least one new directory has been created, false otherwise

Throws:

IOException - thrown if an I/O error occurs while creating the directory

create

public abstract FSDataOutputStream create(Path f,
                                          boolean overwrite,
                                          int bufferSize,
                                          short replication,
                                          long blockSize)
                                   throws IOException

Opens an FSDataOutputStream at the indicated Path.

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - the size of the file blocks

Throws:

IOException

create

public abstract FSDataOutputStream create(Path f,
                                          boolean overwrite)
                                   throws IOException

Opens an FSDataOutputStream at the indicated Path.

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

Throws:

IOException

rename

public abstract boolean rename(Path src,
                               Path dst)
                        throws IOException

Renames the file/directory src to dst.

Parameters:

src - the file/directory to rename

dst - the new name of the file/directory

Returns:

true if the renaming was successful, false otherwise

Throws:

IOException

initOutPathLocalFS

public boolean initOutPathLocalFS(Path outPath,
                                  FileSystem.WriteMode writeMode,
                                  boolean createDirectory)
                           throws IOException

Initializes output directories on local file systems according to the given write mode. WriteMode.NO_OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing directory is reused, files contained in the directory are NOT deleted. - An existing file raises an exception. WriteMode.NO_OVERWRITE & NONE parallel output: - An existing file or directory raises an exception. WriteMode.OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing directory is reused, files contained in the directory are NOT deleted. - An existing file is deleted and replaced by a new directory. WriteMode.OVERWRITE & NONE parallel output: - An existing file or directory (and all its content) is deleted Files contained in an existing directory are not deleted, because multiple instances of a DataSinkTask might call this function at the same time and hence might perform concurrent delete operations on the file system (possibly deleting output files of concurrently running tasks). Since concurrent DataSinkTasks are not aware of each other, coordination of delete and create operations would be difficult.

Parameters:

outPath - Output path that should be prepared.

writeMode - Write mode to consider.

createDirectory - True, to initialize a directory at the given path, false otherwise.

Returns:

True, if the path was successfully prepared, false otherwise.

Throws:

IOException

initOutPathDistFS

public boolean initOutPathDistFS(Path outPath,
                                 FileSystem.WriteMode writeMode,
                                 boolean createDirectory)
                          throws IOException

Initializes output directories on distributed file systems according to the given write mode. WriteMode.NO_OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing file or directory raises an exception. WriteMode.NO_OVERWRITE & NONE parallel output: - An existing file or directory raises an exception. WriteMode.OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing directory and its content is deleted and a new directory is created. - An existing file is deleted and replaced by a new directory. WriteMode.OVERWRITE & NONE parallel output: - An existing file or directory is deleted and replaced by a new directory.

Parameters:

outPath - Output path that should be prepared.

writeMode - Write mode to consider.

createDirectory - True, to initialize a directory at the given path, false otherwise.

Returns:

True, if the path was successfully prepared, false otherwise.

Throws:

IOException

isDistributedFS

public abstract boolean isDistributedFS()

Returns true if this is a distributed file system, false otherwise.

Returns:

True if this is a distributed file system, false otherwise.