org.apache.flink.contrib.streaming.state

类 RocksDBKeyedStateBackend<K>

java.lang.Object

org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend<K>

所有已实现的接口:

Closeable, AutoCloseable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.runtime.state.heap.InternalKeyContext<K>, org.apache.flink.runtime.state.KeyedStateBackend<K>, org.apache.flink.runtime.state.KeyedStateFactory, org.apache.flink.runtime.state.PriorityQueueSetFactory, org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.KeyedStateHandle>>, org.apache.flink.util.Disposable

public class RocksDBKeyedStateBackend<K>
extends org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

An AbstractKeyedStateBackend that stores its state in RocksDB and serializes state to streams provided by a CheckpointStreamFactory upon checkpointing. This state backend can store very large state that exceeds memory and spills to disk. Except for the snapshotting, this class should be accessed as if it is not threadsafe.

This class follows the rules for closing/releasing native RocksDB resources as described in + this document.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	RocksDBKeyedStateBackend.RocksDbKvStateInfo Rocks DB specific information about the k/v states.

从接口继承的嵌套类/接口 org.apache.flink.runtime.state.KeyedStateBackend

org.apache.flink.runtime.state.KeyedStateBackend.KeySelectionListener<K>

字段概要

字段

限定符和类型	字段和说明
protected org.rocksdb.RocksDB	db Our RocksDB database, this is used by the actual subclasses of AbstractRocksDBState to store state.
static String	MERGE_OPERATOR_NAME The name of the merge operator in RocksDB.

从类继承的字段 org.apache.flink.runtime.state.AbstractKeyedStateBackend

cancelStreamRegistry, keyGroupCompressionDecorator, keyGroupRange, keySerializerProvider, kvStateRegistry, numberOfKeyGroups, ttlTimeProvider, userCodeClassLoader

构造器概要

构造器

构造器和说明

RocksDBKeyedStateBackend(ClassLoader userCodeClassLoader, File instanceBasePath, org.rocksdb.DBOptions dbOptions, java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory, org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry, org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider, int numberOfKeyGroups, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, org.apache.flink.api.common.ExecutionConfig executionConfig, org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider, org.rocksdb.RocksDB db, LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation, int keyGroupPrefixBytes, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, org.apache.flink.runtime.state.StreamCompressionDecorator keyGroupCompressionDecorator, org.apache.flink.util.ResourceGuard rocksDBResourceGuard, RocksDBSnapshotStrategyBase<K> checkpointSnapshotStrategy, RocksDBSnapshotStrategyBase<K> savepointSnapshotStrategy, RocksDBWriteBatchWrapper writeBatchWrapper, org.rocksdb.ColumnFamilyHandle defaultColumnFamilyHandle, RocksDBNativeMetricMonitor nativeMetricMonitor, org.apache.flink.contrib.streaming.state.RocksDBSerializedCompositeKeyBuilder<K> sharedRocksKeyBuilder, org.apache.flink.runtime.state.PriorityQueueSetFactory priorityQueueFactory, RocksDbTtlCompactFiltersManager ttlCompactFiltersManager)

方法概要

限定符和类型	方法和说明
void	compactState(org.apache.flink.api.common.state.StateDescriptor<?,?> stateDesc)
<T extends org.apache.flink.runtime.state.heap.HeapPriorityQueueElement & org.apache.flink.runtime.state.PriorityComparable & org.apache.flink.runtime.state.Keyed> org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue<T>	create(String stateName, org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer)
<N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS	createInternalState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc, org.apache.flink.runtime.state.StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory)
void	dispose() Should only be called by one thread, and only after all accesses to the DB happened.
int	getKeyGroupPrefixBytes()
<N> java.util.stream.Stream<K>	getKeys(String state, N namespace)
org.rocksdb.WriteOptions	getWriteOptions()
void	notifyCheckpointComplete(long completedCheckpointId)
int	numKeyValueStateEntries()
boolean	requiresLegacySynchronousTimerSnapshots()
void	setCurrentKey(K newKey)
RunnableFuture<org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.KeyedStateHandle>>	snapshot(long checkpointId, long timestamp, org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory, org.apache.flink.runtime.checkpoint.CheckpointOptions checkpointOptions) Triggers an asynchronous snapshot of the keyed state backend from RocksDB.
boolean	supportsAsynchronousSnapshots()

从类继承的方法 org.apache.flink.runtime.state.AbstractKeyedStateBackend

applyToAllKeys, close, deregisterKeySelectionListener, getCurrentKey, getCurrentKeyGroupIndex, getKeyGroupCompressionDecorator, getKeyGroupRange, getKeySerializer, getNumberOfKeyGroups, getOrCreateKeyedState, getPartitionedState, registerKeySelectionListener

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.state.KeyedStateFactory

createInternalState

字段详细资料

MERGE_OPERATOR_NAME

public static final String MERGE_OPERATOR_NAME

The name of the merge operator in RocksDB. Do not change except you know exactly what you do.

另请参阅:

常量字段值

db

protected final org.rocksdb.RocksDB db

Our RocksDB database, this is used by the actual subclasses of AbstractRocksDBState to store state. The different k/v states that we have don't each have their own RocksDB instance. They all write to this instance but to their own column family.

构造器详细资料

RocksDBKeyedStateBackend

public RocksDBKeyedStateBackend(ClassLoader userCodeClassLoader,
                                File instanceBasePath,
                                org.rocksdb.DBOptions dbOptions,
                                java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
                                org.apache.flink.runtime.query.TaskKvStateRegistry kvStateRegistry,
                                org.apache.flink.runtime.state.StateSerializerProvider<K> keySerializerProvider,
                                int numberOfKeyGroups,
                                org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                org.apache.flink.api.common.ExecutionConfig executionConfig,
                                org.apache.flink.runtime.state.ttl.TtlTimeProvider ttlTimeProvider,
                                org.rocksdb.RocksDB db,
                                LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
                                int keyGroupPrefixBytes,
                                org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry,
                                org.apache.flink.runtime.state.StreamCompressionDecorator keyGroupCompressionDecorator,
                                org.apache.flink.util.ResourceGuard rocksDBResourceGuard,
                                RocksDBSnapshotStrategyBase<K> checkpointSnapshotStrategy,
                                RocksDBSnapshotStrategyBase<K> savepointSnapshotStrategy,
                                RocksDBWriteBatchWrapper writeBatchWrapper,
                                org.rocksdb.ColumnFamilyHandle defaultColumnFamilyHandle,
                                RocksDBNativeMetricMonitor nativeMetricMonitor,
                                org.apache.flink.contrib.streaming.state.RocksDBSerializedCompositeKeyBuilder<K> sharedRocksKeyBuilder,
                                org.apache.flink.runtime.state.PriorityQueueSetFactory priorityQueueFactory,
                                RocksDbTtlCompactFiltersManager ttlCompactFiltersManager)

方法详细资料

getKeys

public <N> java.util.stream.Stream<K> getKeys(String state,
                                              N namespace)

setCurrentKey

public void setCurrentKey(K newKey)

指定者:

setCurrentKey 在接口中 org.apache.flink.runtime.state.KeyedStateBackend<K>

覆盖:

setCurrentKey 在类中 org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

dispose

public void dispose()

Should only be called by one thread, and only after all accesses to the DB happened.

指定者:

dispose 在接口中 org.apache.flink.runtime.state.KeyedStateBackend<K>

指定者:

dispose 在接口中 org.apache.flink.util.Disposable

覆盖:

dispose 在类中 org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

create

@Nonnull
public <T extends org.apache.flink.runtime.state.heap.HeapPriorityQueueElement & org.apache.flink.runtime.state.PriorityComparable & org.apache.flink.runtime.state.Keyed> org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue<T> create(@Nonnull
                                                                                                                                                                                                                                                             String stateName,
                                                                                                                                                                                                                                                             @Nonnull
                                                                                                                                                                                                                                                             org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer)

getKeyGroupPrefixBytes

public int getKeyGroupPrefixBytes()

getWriteOptions

public org.rocksdb.WriteOptions getWriteOptions()

snapshot

@Nonnull
public RunnableFuture<org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.KeyedStateHandle>> snapshot(long checkpointId,
                                                                                                                                        long timestamp,
                                                                                                                                        @Nonnull
                                                                                                                                        org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory,
                                                                                                                                        @Nonnull
                                                                                                                                        org.apache.flink.runtime.checkpoint.CheckpointOptions checkpointOptions)
                                                                                                                                 throws Exception

Triggers an asynchronous snapshot of the keyed state backend from RocksDB. This snapshot can be canceled and is also stopped when the backend is closed through dispose(). For each backend, this method must always be called by the same thread.

参数:

checkpointId - The Id of the checkpoint.

timestamp - The timestamp of the checkpoint.

streamFactory - The factory that we can use for writing our state to streams.

checkpointOptions - Options for how to perform this checkpoint.

返回:

Future to the state handle of the snapshot data.

抛出:

Exception - indicating a problem in the synchronous part of the checkpoint.

notifyCheckpointComplete

public void notifyCheckpointComplete(long completedCheckpointId)
                              throws Exception

抛出:

Exception

createInternalState

@Nonnull
public <N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS createInternalState(@Nonnull
                                                                                                                 org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                                                 @Nonnull
                                                                                                                 org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc,
                                                                                                                 @Nonnull
                                                                                                                 org.apache.flink.runtime.state.StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory)
                                                                                                          throws Exception

抛出:

Exception

supportsAsynchronousSnapshots

public boolean supportsAsynchronousSnapshots()

覆盖:

supportsAsynchronousSnapshots 在类中 org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

numKeyValueStateEntries

@VisibleForTesting
public int numKeyValueStateEntries()

指定者:

numKeyValueStateEntries 在类中 org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

requiresLegacySynchronousTimerSnapshots

public boolean requiresLegacySynchronousTimerSnapshots()

覆盖:

requiresLegacySynchronousTimerSnapshots 在类中 org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

compactState

@VisibleForTesting
public void compactState(org.apache.flink.api.common.state.StateDescriptor<?,?> stateDesc)
                                     throws org.rocksdb.RocksDBException

抛出:

org.rocksdb.RocksDBException