org.apache.flink.contrib.streaming.state

Class RocksDBStateBackend

java.lang.Object

org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.contrib.streaming.state.RocksDBStateBackend

All Implemented Interfaces:

Serializable

public class RocksDBStateBackend
extends org.apache.flink.runtime.state.AbstractStateBackend

A StateBackend that stores its state in RocksDB. This state backend can store very large state that exceeds memory and spills to disk.

All key/value state (including windows) is stored in the key/value index of RocksDB. For persistence against loss of machines, checkpoints take a snapshot of the RocksDB database, and persist that snapshot in a file system (by default) or another configurable state backend.

The behavior of the RocksDB instances can be parametrized by setting RocksDB Options using the methods setPredefinedOptions(PredefinedOptions) and setOptions(OptionsFactory).

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputStream, org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView

Field Summary

Fields

Modifier and Type	Field and Description
protected org.rocksdb.RocksDB	db Our RocksDB data base, this is used by the actual subclasses of AbstractRocksDBState to store state.

Fields inherited from class org.apache.flink.runtime.state.AbstractStateBackend

currentKey, keySerializer, keyValueStatesByName, userCodeClassLoader

Constructor Summary

Constructors

Constructor and Description
RocksDBStateBackend(String checkpointDataUri) Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
RocksDBStateBackend(String checkpointDataUri, org.apache.flink.runtime.state.AbstractStateBackend nonPartitionedStateBackend)
RocksDBStateBackend(URI checkpointDataUri) Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.
RocksDBStateBackend(URI checkpointDataUri, org.apache.flink.runtime.state.AbstractStateBackend nonPartitionedStateBackend)

Method Summary

Modifier and Type	Method and Description
<S extends Serializable> org.apache.flink.runtime.state.StateHandle<S>	checkpointStateSerializable(S state, long checkpointID, long timestamp)
void	close()
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputStream	createCheckpointStateOutputStream(long checkpointID, long timestamp)
protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC>	createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ListState<T>	createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ReducingState<T>	createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ValueState<T>	createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ValueStateDescriptor<T> stateDesc)
Object	currentKey() Used by k/v states to access the current key.
void	disableFullyAsyncSnapshots() Disables fully asynchronous snapshotting of the partitioned state held in RocksDB.
void	dispose()
void	disposeAllStateForCurrentJob()
void	enableFullyAsyncSnapshots() Enables fully asynchronous snapshotting of the partitioned state held in RocksDB.
protected org.rocksdb.ColumnFamilyHandle	getColumnFamily(org.apache.flink.api.common.state.StateDescriptor descriptor) Creates a column family handle for use with a k/v state.
org.rocksdb.ColumnFamilyOptions	getColumnOptions() Gets the RocksDB ColumnFamilyOptions to be used for all RocksDB instances.
org.rocksdb.DBOptions	getDbOptions() Gets the RocksDB DBOptions to be used for all RocksDB instances.
String[]	getDbStoragePaths()
OptionsFactory	getOptions() Gets the options factory that lazily creates the RocksDB options.
PredefinedOptions	getPredefinedOptions() Gets the currently set predefined options for RocksDB.
File[]	getStoragePaths() Visible for tests.
void	initializeForJob(org.apache.flink.runtime.execution.Environment env, String operatorIdentifier, org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)
void	injectKeyValueStateSnapshots(HashMap<String,org.apache.flink.runtime.state.KvStateSnapshot> keyValueStateSnapshots)
org.apache.flink.api.common.typeutils.TypeSerializer	keySerializer() Used by k/v states to access the key serializer.
void	setDbStoragePath(String path) Sets the path where the RocksDB local database files should be stored on the local file system.
void	setDbStoragePaths(String... paths) Sets the paths across which the local RocksDB database files are distributed on the local file system.
void	setOptions(OptionsFactory optionsFactory) Sets Options for the RocksDB instances.
void	setPredefinedOptions(PredefinedOptions options) Sets the predefined options for RocksDB.
HashMap<String,org.apache.flink.runtime.state.KvStateSnapshot<?,?,?,?,?>>	snapshotPartitionedState(long checkpointId, long timestamp)

Methods inherited from class org.apache.flink.runtime.state.AbstractStateBackend

createCheckpointStateOutputView, getCurrentKey, getPartitionedState, mergePartitionedStates, notifyOfCompletedCheckpoint, setCurrentKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

db

protected transient volatile org.rocksdb.RocksDB db

Our RocksDB data base, this is used by the actual subclasses of AbstractRocksDBState to store state. The different k/v states that we have don't each have their own RocksDB instance. They all write to this instance but to their own column family.

Constructor Detail

RocksDBStateBackend

public RocksDBStateBackend(String checkpointDataUri)
                    throws IOException

Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

RocksDBStateBackend

public RocksDBStateBackend(URI checkpointDataUri)
                    throws IOException

Creates a new RocksDBStateBackend that stores its checkpoint data in the file system and location defined by the given URI.

A state backend that stores checkpoints in HDFS or S3 must specify the file system host and port in the URI, or have the Hadoop configuration that describes the file system (host / high-availability group / possibly credentials) either referenced from the Flink config, or included in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem and path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

RocksDBStateBackend

public RocksDBStateBackend(String checkpointDataUri,
                           org.apache.flink.runtime.state.AbstractStateBackend nonPartitionedStateBackend)
                    throws IOException

Throws:

IOException

RocksDBStateBackend

public RocksDBStateBackend(URI checkpointDataUri,
                           org.apache.flink.runtime.state.AbstractStateBackend nonPartitionedStateBackend)
                    throws IOException

Throws:

IOException

Method Detail

initializeForJob

public void initializeForJob(org.apache.flink.runtime.execution.Environment env,
                             String operatorIdentifier,
                             org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)
                      throws Exception

Overrides:

initializeForJob in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

disposeAllStateForCurrentJob

public void disposeAllStateForCurrentJob()
                                  throws Exception

Specified by:

disposeAllStateForCurrentJob in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

dispose

public void dispose()

Overrides:

dispose in class org.apache.flink.runtime.state.AbstractStateBackend

close

public void close()
           throws Exception

Specified by:

close in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

getStoragePaths

public File[] getStoragePaths()

Visible for tests.

snapshotPartitionedState

public HashMap<String,org.apache.flink.runtime.state.KvStateSnapshot<?,?,?,?,?>> snapshotPartitionedState(long checkpointId,
                                                                                                          long timestamp)
                                                                                                   throws Exception

Overrides:

snapshotPartitionedState in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

injectKeyValueStateSnapshots

public final void injectKeyValueStateSnapshots(HashMap<String,org.apache.flink.runtime.state.KvStateSnapshot> keyValueStateSnapshots)
                                        throws Exception

Overrides:

injectKeyValueStateSnapshots in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

getColumnFamily

protected org.rocksdb.ColumnFamilyHandle getColumnFamily(org.apache.flink.api.common.state.StateDescriptor descriptor)

Creates a column family handle for use with a k/v state. When restoring from a snapshot we don't restore the individual k/v states, just the global RocksDB data base and the list of column families. When a k/v state is first requested we check here whether we already have a column family for that and return it or create a new one if it doesn't exist.

This also checks whether the StateDescriptor for a state matches the one that we checkpointed, i.e. is already in the map of column families.

currentKey

public Object currentKey()

Used by k/v states to access the current key.

keySerializer

public org.apache.flink.api.common.typeutils.TypeSerializer keySerializer()

Used by k/v states to access the key serializer.

createValueState

protected <N,T> org.apache.flink.api.common.state.ValueState<T> createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                 org.apache.flink.api.common.state.ValueStateDescriptor<T> stateDesc)
                                                                          throws Exception

Specified by:

createValueState in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

createListState

protected <N,T> org.apache.flink.api.common.state.ListState<T> createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                               org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc)
                                                                        throws Exception

Specified by:

createListState in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

createReducingState

protected <N,T> org.apache.flink.api.common.state.ReducingState<T> createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                       org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc)
                                                                                throws Exception

Specified by:

createReducingState in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

createFoldingState

protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC> createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                             org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc)
                                                                                      throws Exception

Specified by:

createFoldingState in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

createCheckpointStateOutputStream

public org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputStream createCheckpointStateOutputStream(long checkpointID,
                                                                                                                         long timestamp)
                                                                                                                  throws Exception

Specified by:

createCheckpointStateOutputStream in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

checkpointStateSerializable

public <S extends Serializable> org.apache.flink.runtime.state.StateHandle<S> checkpointStateSerializable(S state,
                                                                                                          long checkpointID,
                                                                                                          long timestamp)
                                                                                                   throws Exception

Specified by:

checkpointStateSerializable in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

enableFullyAsyncSnapshots

public void enableFullyAsyncSnapshots()

Enables fully asynchronous snapshotting of the partitioned state held in RocksDB.

By default, this is disabled. This means that RocksDB state is copied in a synchronous step, during which normal processing of elements pauses, followed by an asynchronous step of copying the RocksDB backup to the final checkpoint location. Fully asynchronous snapshots take longer (linear time requirement with respect to number of unique keys) but normal processing of elements is not paused.

disableFullyAsyncSnapshots

public void disableFullyAsyncSnapshots()

Disables fully asynchronous snapshotting of the partitioned state held in RocksDB.

By default, this is disabled.

setDbStoragePath

public void setDbStoragePath(String path)

Sets the path where the RocksDB local database files should be stored on the local file system. Setting this path overrides the default behavior, where the files are stored across the configured temp directories.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

path - The path where the local RocksDB database files are stored.

setDbStoragePaths

public void setDbStoragePaths(String... paths)

Sets the paths across which the local RocksDB database files are distributed on the local file system. Setting these paths overrides the default behavior, where the files are stored across the configured temp directories.

Each distinct state will be stored in one path, but when the state backend creates multiple states, they will store their files on different paths.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

paths - The paths across which the local RocksDB database files will be spread.

getDbStoragePaths

public String[] getDbStoragePaths()

Returns:

The configured DB storage paths, or null, if none were configured.

setPredefinedOptions

public void setPredefinedOptions(PredefinedOptions options)

Sets the predefined options for RocksDB.

If a user-defined options factory is set (via setOptions(OptionsFactory)), then the options from the factory are applied on top of the here specified predefined options.

Parameters:

options - The options to set (must not be null).

getPredefinedOptions

public PredefinedOptions getPredefinedOptions()

Gets the currently set predefined options for RocksDB. The default options (if nothing was set via setPredefinedOptions(PredefinedOptions)) are PredefinedOptions.DEFAULT.

If a user-defined options factory is set (via setOptions(OptionsFactory)), then the options from the factory are applied on top of the predefined options.

Returns:

The currently set predefined options for RocksDB.

setOptions

public void setOptions(OptionsFactory optionsFactory)

Sets Options for the RocksDB instances. Because the options are not serializable and hold native code references, they must be specified through a factory.

The options created by the factory here are applied on top of the pre-defined options profile selected via setPredefinedOptions(PredefinedOptions). If the pre-defined options profile is the default (PredefinedOptions.DEFAULT), then the factory fully controls the RocksDB options.

Parameters:

optionsFactory - The options factory that lazily creates the RocksDB options.

getOptions

public OptionsFactory getOptions()

Gets the options factory that lazily creates the RocksDB options.

Returns:

The options factory.

getDbOptions

public org.rocksdb.DBOptions getDbOptions()

Gets the RocksDB DBOptions to be used for all RocksDB instances.

getColumnOptions

public org.rocksdb.ColumnFamilyOptions getColumnOptions()

Gets the RocksDB ColumnFamilyOptions to be used for all RocksDB instances.