org.apache.flink.contrib.streaming.state

Class RocksDBStateBackend

java.lang.Object

org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.contrib.streaming.state.RocksDBStateBackend

All Implemented Interfaces:

Serializable

public class RocksDBStateBackend
extends org.apache.flink.runtime.state.AbstractStateBackend

A StateBackend that stores its state in RocksDB. This state backend can store very large state that exceeds memory and spills to disk.

All key/value state (including windows) is stored in the key/value index of RocksDB. For persistence against loss of machines, checkpoints take a snapshot of the RocksDB database, and persist that snapshot in a file system (by default) or another configurable state backend.

The behavior of the RocksDB instances can be parametrized by setting RocksDB Options using the methods setPredefinedOptions(PredefinedOptions) and setOptions(OptionsFactory).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
RocksDBStateBackend(org.apache.flink.runtime.state.AbstractStateBackend checkpointStreamBackend) Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams.
RocksDBStateBackend(String checkpointDataUri) Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
RocksDBStateBackend(URI checkpointDataUri) Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

Method Summary

Methods

Modifier and Type	Method and Description
<K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>	createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env, org.apache.flink.api.common.JobID jobID, String operatorIdentifier, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, int numberOfKeyGroups, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry)
org.apache.flink.runtime.state.CheckpointStreamFactory	createStreamFactory(org.apache.flink.api.common.JobID jobId, String operatorIdentifier)
org.rocksdb.ColumnFamilyOptions	getColumnOptions() Gets the RocksDB ColumnFamilyOptions to be used for all RocksDB instances.
org.rocksdb.DBOptions	getDbOptions() Gets the RocksDB DBOptions to be used for all RocksDB instances.
String[]	getDbStoragePaths()
OptionsFactory	getOptions() Gets the options factory that lazily creates the RocksDB options.
PredefinedOptions	getPredefinedOptions() Gets the currently set predefined options for RocksDB.
void	setDbStoragePath(String path) Sets the path where the RocksDB local database files should be stored on the local file system.
void	setDbStoragePaths(String... paths) Sets the paths across which the local RocksDB database files are distributed on the local file system.
void	setOptions(OptionsFactory optionsFactory) Sets Options for the RocksDB instances.
void	setPredefinedOptions(PredefinedOptions options) Sets the predefined options for RocksDB.
String	toString()

Methods inherited from class org.apache.flink.runtime.state.AbstractStateBackend

createOperatorStateBackend

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RocksDBStateBackend

public RocksDBStateBackend(String checkpointDataUri)
                    throws IOException

Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

RocksDBStateBackend

public RocksDBStateBackend(URI checkpointDataUri)
                    throws IOException

Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

RocksDBStateBackend

public RocksDBStateBackend(org.apache.flink.runtime.state.AbstractStateBackend checkpointStreamBackend)

Creates a new RocksDBStateBackend that uses the given state backend to store its checkpoint data streams. Typically, one would supply a filesystem or database state backend here where the snapshots from RocksDB would be stored.

The snapshots of the RocksDB state will be stored using the given backend's checkpoint stream.

Parameters:

checkpointStreamBackend - The backend to store the

Method Detail

createStreamFactory

public org.apache.flink.runtime.state.CheckpointStreamFactory createStreamFactory(org.apache.flink.api.common.JobID jobId,
                                                                         String operatorIdentifier)
                                                                           throws IOException

Specified by:

createStreamFactory in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

IOException

createKeyedStateBackend

public <K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K> createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env,
                                                                                      org.apache.flink.api.common.JobID jobID,
                                                                                      String operatorIdentifier,
                                                                                      org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                                                                                      int numberOfKeyGroups,
                                                                                      org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                                                                      org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry)
                                                                                    throws Exception

Specified by:

createKeyedStateBackend in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

setDbStoragePath

public void setDbStoragePath(String path)

Sets the path where the RocksDB local database files should be stored on the local file system. Setting this path overrides the default behavior, where the files are stored across the configured temp directories.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

path - The path where the local RocksDB database files are stored.

setDbStoragePaths

public void setDbStoragePaths(String... paths)

Sets the paths across which the local RocksDB database files are distributed on the local file system. Setting these paths overrides the default behavior, where the files are stored across the configured temp directories.

Each distinct state will be stored in one path, but when the state backend creates multiple states, they will store their files on different paths.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

paths - The paths across which the local RocksDB database files will be spread.

getDbStoragePaths

public String[] getDbStoragePaths()

Returns:

The configured DB storage paths, or null, if none were configured.

setPredefinedOptions

public void setPredefinedOptions(PredefinedOptions options)

Sets the predefined options for RocksDB.

If a user-defined options factory is set (via setOptions(OptionsFactory)), then the options from the factory are applied on top of the here specified predefined options.

Parameters:

options - The options to set (must not be null).

getPredefinedOptions

public PredefinedOptions getPredefinedOptions()

Gets the currently set predefined options for RocksDB. The default options (if nothing was set via setPredefinedOptions(PredefinedOptions)) are PredefinedOptions.DEFAULT.

If a user-defined options factory is set (via setOptions(OptionsFactory)), then the options from the factory are applied on top of the predefined options.

Returns:

The currently set predefined options for RocksDB.

setOptions

public void setOptions(OptionsFactory optionsFactory)

Sets Options for the RocksDB instances. Because the options are not serializable and hold native code references, they must be specified through a factory.

The options created by the factory here are applied on top of the pre-defined options profile selected via setPredefinedOptions(PredefinedOptions). If the pre-defined options profile is the default (PredefinedOptions.DEFAULT), then the factory fully controls the RocksDB options.

Parameters:

optionsFactory - The options factory that lazily creates the RocksDB options.

getOptions

public OptionsFactory getOptions()

Gets the options factory that lazily creates the RocksDB options.

Returns:

The options factory.

getDbOptions

public org.rocksdb.DBOptions getDbOptions()

Gets the RocksDB DBOptions to be used for all RocksDB instances.

getColumnOptions

public org.rocksdb.ColumnFamilyOptions getColumnOptions()

Gets the RocksDB ColumnFamilyOptions to be used for all RocksDB instances.

toString

public String toString()

Overrides:

toString in class Object