org.apache.flink.contrib.streaming.state

Class EmbeddedRocksDBStateBackend

java.lang.Object

org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend

org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend

All Implemented Interfaces:

Serializable, org.apache.flink.runtime.state.ConfigurableStateBackend, org.apache.flink.runtime.state.StateBackend

@PublicEvolving
public class EmbeddedRocksDBStateBackend
extends org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend
implements org.apache.flink.runtime.state.ConfigurableStateBackend

A StateBackend that stores its state in an embedded RocksDB instance. This state backend can store very large state that exceeds memory and spills to local disk. All key/value state (including windows) is stored in the key/value index of RocksDB. For persistence against loss of machines, please configure a CheckpointStorage instance for the Job.

The behavior of the RocksDB instances can be parametrized by setting RocksDB Options using the methods setPredefinedOptions(PredefinedOptions) and setRocksDBOptions(RocksDBOptionsFactory).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EmbeddedRocksDBStateBackend.PriorityQueueStateType The options to chose for the type of priority queue state.

Field Summary

Fields inherited from class org.apache.flink.runtime.state.AbstractStateBackend

latencyTrackingConfigBuilder

Constructor Summary

Constructors

Constructor and Description
EmbeddedRocksDBStateBackend() Creates a new EmbeddedRocksDBStateBackend for storing local state.
EmbeddedRocksDBStateBackend(boolean enableIncrementalCheckpointing) Creates a new EmbeddedRocksDBStateBackend for storing local state.
EmbeddedRocksDBStateBackend(org.apache.flink.util.TernaryBoolean enableIncrementalCheckpointing) Creates a new EmbeddedRocksDBStateBackend for storing local state.

Method Summary

Modifier and Type	Method and Description
EmbeddedRocksDBStateBackend	configure(org.apache.flink.configuration.ReadableConfig config, ClassLoader classLoader) Creates a copy of this state backend that uses the values defined in the configuration for fields where that were not yet specified in this state backend.
<K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>	createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env, org.apache.flink.api.common.JobID jobID, String operatorIdentifier, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, int numberOfKeyGroups, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry, org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider, org.apache.flink.metrics.MetricGroup metricGroup, Collection<org.apache.flink.runtime.state.KeyedStateHandle> stateHandles, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry)
<K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>	createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env, org.apache.flink.api.common.JobID jobID, String operatorIdentifier, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, int numberOfKeyGroups, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry, org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider, org.apache.flink.metrics.MetricGroup metricGroup, Collection<org.apache.flink.runtime.state.KeyedStateHandle> stateHandles, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, double managedMemoryFraction)
org.apache.flink.runtime.state.OperatorStateBackend	createOperatorStateBackend(org.apache.flink.runtime.execution.Environment env, String operatorIdentifier, Collection<org.apache.flink.runtime.state.OperatorStateHandle> stateHandles, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry)
String[]	getDbStoragePaths() Gets the configured local DB storage paths, or null, if none were configured.
RocksDBMemoryConfiguration	getMemoryConfiguration() Gets the memory configuration object, which offers settings to control RocksDB's memory usage.
int	getNumberOfTransferThreads() Gets the number of threads used to transfer files while snapshotting/restoring.
PredefinedOptions	getPredefinedOptions() Gets the currently set predefined options for RocksDB.
EmbeddedRocksDBStateBackend.PriorityQueueStateType	getPriorityQueueStateType() Gets the type of the priority queue state.
RocksDBOptionsFactory	getRocksDBOptions() Gets Options for the RocksDB instances.
long	getWriteBatchSize() Gets the max batch size will be used in RocksDBWriteBatchWrapper.
boolean	isIncrementalCheckpointsEnabled() Gets whether incremental checkpoints are enabled for this state backend.
void	setDbStoragePath(String path) Sets the path where the RocksDB local database files should be stored on the local file system.
void	setDbStoragePaths(String... paths) Sets the directories in which the local RocksDB database puts its files (like SST and metadata files).
void	setNumberOfTransferThreads(int numberOfTransferThreads) Sets the number of threads used to transfer files while snapshotting/restoring.
void	setPredefinedOptions(PredefinedOptions options) Sets the predefined options for RocksDB.
void	setPriorityQueueStateType(EmbeddedRocksDBStateBackend.PriorityQueueStateType priorityQueueStateType) Sets the type of the priority queue state.
void	setRocksDBOptions(RocksDBOptionsFactory optionsFactory) Sets Options for the RocksDB instances.
void	setWriteBatchSize(long writeBatchSize) Sets the max batch size will be used in RocksDBWriteBatchWrapper, no positive value will disable memory size controller, just use item count controller.
String	toString()

Methods inherited from class org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend

useManagedMemory

Methods inherited from class org.apache.flink.runtime.state.AbstractStateBackend

getCompressionDecorator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.state.StateBackend

useManagedMemory

Constructor Detail

EmbeddedRocksDBStateBackend

public EmbeddedRocksDBStateBackend()

Creates a new EmbeddedRocksDBStateBackend for storing local state.

EmbeddedRocksDBStateBackend

public EmbeddedRocksDBStateBackend(boolean enableIncrementalCheckpointing)

Creates a new EmbeddedRocksDBStateBackend for storing local state.

Parameters:

enableIncrementalCheckpointing - True if incremental checkpointing is enabled.

EmbeddedRocksDBStateBackend

public EmbeddedRocksDBStateBackend(org.apache.flink.util.TernaryBoolean enableIncrementalCheckpointing)

Creates a new EmbeddedRocksDBStateBackend for storing local state.

Parameters:

enableIncrementalCheckpointing - True if incremental checkpointing is enabled.

Method Detail

configure

public EmbeddedRocksDBStateBackend configure(org.apache.flink.configuration.ReadableConfig config,
                                             ClassLoader classLoader)

Creates a copy of this state backend that uses the values defined in the configuration for fields where that were not yet specified in this state backend.

Specified by:

configure in interface org.apache.flink.runtime.state.ConfigurableStateBackend

Parameters:

config - The configuration.

classLoader - The class loader.

Returns:

The re-configured variant of the state backend

createKeyedStateBackend

public <K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K> createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env,
                                                                                               org.apache.flink.api.common.JobID jobID,
                                                                                               String operatorIdentifier,
                                                                                               org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                                                                                               int numberOfKeyGroups,
                                                                                               org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                                                                               org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry,
                                                                                               org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider,
                                                                                               org.apache.flink.metrics.MetricGroup metricGroup,
                                                                                               @Nonnull
                                                                                               Collection<org.apache.flink.runtime.state.KeyedStateHandle> stateHandles,
                                                                                               org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry)
                                                                                        throws IOException

Specified by:

createKeyedStateBackend in interface org.apache.flink.runtime.state.StateBackend

Specified by:

createKeyedStateBackend in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

IOException

createKeyedStateBackend

public <K> org.apache.flink.runtime.state.AbstractKeyedStateBackend<K> createKeyedStateBackend(org.apache.flink.runtime.execution.Environment env,
                                                                                               org.apache.flink.api.common.JobID jobID,
                                                                                               String operatorIdentifier,
                                                                                               org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                                                                                               int numberOfKeyGroups,
                                                                                               org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                                                                               org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry,
                                                                                               org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider,
                                                                                               org.apache.flink.metrics.MetricGroup metricGroup,
                                                                                               @Nonnull
                                                                                               Collection<org.apache.flink.runtime.state.KeyedStateHandle> stateHandles,
                                                                                               org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry,
                                                                                               double managedMemoryFraction)
                                                                                        throws IOException

Specified by:

createKeyedStateBackend in interface org.apache.flink.runtime.state.StateBackend

Specified by:

createKeyedStateBackend in class org.apache.flink.runtime.state.AbstractManagedMemoryStateBackend

Throws:

IOException

createOperatorStateBackend

public org.apache.flink.runtime.state.OperatorStateBackend createOperatorStateBackend(org.apache.flink.runtime.execution.Environment env,
                                                                                      String operatorIdentifier,
                                                                                      @Nonnull
                                                                                      Collection<org.apache.flink.runtime.state.OperatorStateHandle> stateHandles,
                                                                                      org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry)
                                                                               throws Exception

Specified by:

createOperatorStateBackend in interface org.apache.flink.runtime.state.StateBackend

Specified by:

createOperatorStateBackend in class org.apache.flink.runtime.state.AbstractStateBackend

Throws:

Exception

getMemoryConfiguration

public RocksDBMemoryConfiguration getMemoryConfiguration()

Gets the memory configuration object, which offers settings to control RocksDB's memory usage.

setDbStoragePath

public void setDbStoragePath(String path)

Sets the path where the RocksDB local database files should be stored on the local file system. Setting this path overrides the default behavior, where the files are stored across the configured temp directories.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

path - The path where the local RocksDB database files are stored.

setDbStoragePaths

public void setDbStoragePaths(String... paths)

Sets the directories in which the local RocksDB database puts its files (like SST and metadata files). These directories do not need to be persistent, they can be ephemeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints.

If nothing is configured, these directories default to the TaskManager's local temporary file directories.

Each distinct state will be stored in one path, but when the state backend creates multiple states, they will store their files on different paths.

Passing null to this function restores the default behavior, where the configured temp directories will be used.

Parameters:

paths - The paths across which the local RocksDB database files will be spread.

getDbStoragePaths

public String[] getDbStoragePaths()

Gets the configured local DB storage paths, or null, if none were configured.

Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints.

If nothing is configured, these directories default to the TaskManager's local temporary file directories.

isIncrementalCheckpointsEnabled

public boolean isIncrementalCheckpointsEnabled()

Gets whether incremental checkpoints are enabled for this state backend.

getPriorityQueueStateType

public EmbeddedRocksDBStateBackend.PriorityQueueStateType getPriorityQueueStateType()

Gets the type of the priority queue state. It will fallback to the default value, if it is not explicitly set.

Returns:

The type of the priority queue state.

setPriorityQueueStateType

public void setPriorityQueueStateType(EmbeddedRocksDBStateBackend.PriorityQueueStateType priorityQueueStateType)

Sets the type of the priority queue state. It will fallback to the default value, if it is not explicitly set.

setPredefinedOptions

public void setPredefinedOptions(@Nonnull
                                 PredefinedOptions options)

Sets the predefined options for RocksDB.

If user-configured options within RocksDBConfigurableOptions is set (through flink-conf.yaml) or a user-defined options factory is set (via setRocksDBOptions(RocksDBOptionsFactory)), then the options from the factory are applied on top of the here specified predefined options and customized options.

Parameters:

options - The options to set (must not be null).

getPredefinedOptions

@VisibleForTesting
public PredefinedOptions getPredefinedOptions()

Gets the currently set predefined options for RocksDB. The default options (if nothing was set via setPredefinedOptions(PredefinedOptions)) are PredefinedOptions.DEFAULT.

If user-configured options within RocksDBConfigurableOptions is set (through flink-conf.yaml) of a user-defined options factory is set (via setRocksDBOptions(RocksDBOptionsFactory)), then the options from the factory are applied on top of the predefined and customized options.

Returns:

The currently set predefined options for RocksDB.

setRocksDBOptions

public void setRocksDBOptions(RocksDBOptionsFactory optionsFactory)

Sets Options for the RocksDB instances. Because the options are not serializable and hold native code references, they must be specified through a factory.

The options created by the factory here are applied on top of the pre-defined options profile selected via setPredefinedOptions(PredefinedOptions). If the pre-defined options profile is the default (PredefinedOptions.DEFAULT), then the factory fully controls the RocksDB options.

Parameters:

optionsFactory - The options factory that lazily creates the RocksDB options.

getRocksDBOptions

@Nullable
public RocksDBOptionsFactory getRocksDBOptions()

Gets Options for the RocksDB instances.

The options created by the factory here are applied on top of the pre-defined options profile selected via setPredefinedOptions(PredefinedOptions). If the pre-defined options profile is the default (PredefinedOptions.DEFAULT), then the factory fully controls the RocksDB options.

getNumberOfTransferThreads

public int getNumberOfTransferThreads()

Gets the number of threads used to transfer files while snapshotting/restoring.

setNumberOfTransferThreads

public void setNumberOfTransferThreads(int numberOfTransferThreads)

Sets the number of threads used to transfer files while snapshotting/restoring.

Parameters:

numberOfTransferThreads - The number of threads used to transfer files while snapshotting/restoring.

getWriteBatchSize

public long getWriteBatchSize()

Gets the max batch size will be used in RocksDBWriteBatchWrapper.

setWriteBatchSize

public void setWriteBatchSize(long writeBatchSize)

Sets the max batch size will be used in RocksDBWriteBatchWrapper, no positive value will disable memory size controller, just use item count controller.

Parameters:

writeBatchSize - The size will used to be used in RocksDBWriteBatchWrapper.

toString

public String toString()

Overrides:

toString in class Object