org.apache.flink.contrib.streaming.state

Class RocksDBKeyedStateBackend<K>

java.lang.Object

org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend<K>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.flink.runtime.state.KeyedStateBackend<K>, org.apache.flink.runtime.state.Snapshotable<org.apache.flink.runtime.state.KeyGroupsStateHandle>

public class RocksDBKeyedStateBackend<K>
extends org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

A AbstractKeyedStateBackend that stores its state in RocksDB and will serialize state to streams provided by a CheckpointStreamFactory upon checkpointing. This state backend can store very large state that exceeds memory and spills to disk. Except for the snapshotting, this class should be accessed as if it is not threadsafe.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.rocksdb.RocksDB	db Our RocksDB data base, this is used by the actual subclasses of AbstractRocksDBState to store state.

Fields inherited from class org.apache.flink.runtime.state.AbstractKeyedStateBackend

cancelStreamRegistry, currentKey, keyGroupRange, keySerializer, keyValueStatesByName, kvStateRegistry, numberOfKeyGroups, userCodeClassLoader

Constructor Summary

Constructors

Constructor and Description

RocksDBKeyedStateBackend(org.apache.flink.api.common.JobID jobId, String operatorIdentifier, ClassLoader userCodeClassLoader, File instanceBasePath, org.rocksdb.DBOptions dbOptions, org.rocksdb.ColumnFamilyOptions columnFamilyOptions, org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, int numberOfKeyGroups, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange)

Method Summary

Methods

Modifier and Type	Method and Description
protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC>	createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ListState<T>	createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ReducingState<T>	createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc)
protected <N,T> org.apache.flink.api.common.state.ValueState<T>	createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ValueStateDescriptor<T> stateDesc)
void	dispose() Should only be called by one thread, and only after all accesses to the DB happened.
protected <N,S> org.rocksdb.ColumnFamilyHandle	getColumnFamily(org.apache.flink.api.common.state.StateDescriptor<?,S> descriptor, org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer) Creates a column family handle for use with a k/v state.
File	getInstanceBasePath() Only visible for testing, DO NOT USE.
int	getKeyGroupPrefixBytes()
void	restore(Collection<org.apache.flink.runtime.state.KeyGroupsStateHandle> restoreState)
RunnableFuture<org.apache.flink.runtime.state.KeyGroupsStateHandle>	snapshot(long checkpointId, long timestamp, org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory) Triggers an asynchronous snapshot of the keyed state backend from RocksDB.

Methods inherited from class org.apache.flink.runtime.state.AbstractKeyedStateBackend

close, getCurrentKey, getCurrentKeyGroupIndex, getKeyGroupRange, getKeySerializer, getNumberOfKeyGroups, getPartitionedState, mergePartitionedStates, setCurrentKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

db

protected org.rocksdb.RocksDB db

Our RocksDB data base, this is used by the actual subclasses of AbstractRocksDBState to store state. The different k/v states that we have don't each have their own RocksDB instance. They all write to this instance but to their own column family.

Constructor Detail

RocksDBKeyedStateBackend

public RocksDBKeyedStateBackend(org.apache.flink.api.common.JobID jobId,
                        String operatorIdentifier,
                        ClassLoader userCodeClassLoader,
                        File instanceBasePath,
                        org.rocksdb.DBOptions dbOptions,
                        org.rocksdb.ColumnFamilyOptions columnFamilyOptions,
                        org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry,
                        org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                        int numberOfKeyGroups,
                        org.apache.flink.runtime.state.KeyGroupRange keyGroupRange)
                         throws IOException

Throws:

IOException

Method Detail

dispose

public void dispose()

Should only be called by one thread, and only after all accesses to the DB happened.

Specified by:

dispose in interface org.apache.flink.runtime.state.KeyedStateBackend<K>

Overrides:

dispose in class org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

getKeyGroupPrefixBytes

public int getKeyGroupPrefixBytes()

snapshot

public RunnableFuture<org.apache.flink.runtime.state.KeyGroupsStateHandle> snapshot(long checkpointId,
                                                                           long timestamp,
                                                                           org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory)
                                                                             throws Exception

Triggers an asynchronous snapshot of the keyed state backend from RocksDB. This snapshot can be canceled and is also stopped when the backend is closed through dispose(). For each backend, this method must always be called by the same thread.

Parameters:

checkpointId - The Id of the checkpoint.

timestamp - The timestamp of the checkpoint.

streamFactory - The factory that we can use for writing our state to streams.

Returns:

Future to the state handle of the snapshot data.

Throws:

Exception

restore

public void restore(Collection<org.apache.flink.runtime.state.KeyGroupsStateHandle> restoreState)
             throws Exception

Throws:

Exception

getColumnFamily

protected <N,S> org.rocksdb.ColumnFamilyHandle getColumnFamily(org.apache.flink.api.common.state.StateDescriptor<?,S> descriptor,
                                                   org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer)
                                                  throws IOException

Creates a column family handle for use with a k/v state. When restoring from a snapshot we don't restore the individual k/v states, just the global RocksDB data base and the list of column families. When a k/v state is first requested we check here whether we already have a column family for that and return it or create a new one if it doesn't exist.

This also checks whether the StateDescriptor for a state matches the one that we checkpointed, i.e. is already in the map of column families.

Throws:

IOException

createValueState

protected <N,T> org.apache.flink.api.common.state.ValueState<T> createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                     org.apache.flink.api.common.state.ValueStateDescriptor<T> stateDesc)
                                                                    throws Exception

Specified by:

createValueState in class org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

Throws:

Exception

createListState

protected <N,T> org.apache.flink.api.common.state.ListState<T> createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                   org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc)
                                                                  throws Exception

Specified by:

createListState in class org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

Throws:

Exception

createReducingState

protected <N,T> org.apache.flink.api.common.state.ReducingState<T> createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                           org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc)
                                                                          throws Exception

Specified by:

createReducingState in class org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

Throws:

Exception

createFoldingState

protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC> createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                 org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc)
                                                                            throws Exception

Specified by:

createFoldingState in class org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

Throws:

Exception

getInstanceBasePath

public File getInstanceBasePath()

Only visible for testing, DO NOT USE.