org.apache.flink.contrib.streaming.state.snapshot

Class RocksDBSnapshotStrategyBase<K,R extends org.apache.flink.runtime.state.SnapshotResources>

java.lang.Object

org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase<K,R>

Type Parameters:

K - type of the backend keys.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>

Direct Known Subclasses:

RocksIncrementalSnapshotStrategy, RocksNativeFullSnapshotStrategy

public abstract class RocksDBSnapshotStrategyBase<K,R extends org.apache.flink.runtime.state.SnapshotResources>
extends Object
implements org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>, AutoCloseable

Abstract base class for SnapshotStrategy implementations for RocksDB state backend.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources A SnapshotResources for native rocksdb snapshot.
protected static class	RocksDBSnapshotStrategyBase.PreviousSnapshot Previous snapshot with uploaded sst files.
protected class	RocksDBSnapshotStrategyBase.RocksDBSnapshotOperation Common operation in native rocksdb snapshot result supplier.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

org.apache.flink.runtime.state.SnapshotStrategy.SnapshotResultSupplier<S extends org.apache.flink.runtime.state.StateObject>

Field Summary

Fields

Modifier and Type	Field and Description
protected UUID	backendUID The state handle ids of all sst files materialized in snapshots for previous checkpoints.
protected org.rocksdb.RocksDB	db RocksDB instance from the backend.
protected static RocksDBSnapshotStrategyBase.PreviousSnapshot	EMPTY_PREVIOUS_SNAPSHOT
protected File	instanceBasePath Base path of the RocksDB instance.
protected int	keyGroupPrefixBytes Number of bytes in the key-group prefix.
protected org.apache.flink.runtime.state.KeyGroupRange	keyGroupRange The key-group range for the task.
protected org.apache.flink.api.common.typeutils.TypeSerializer<K>	keySerializer The key serializer of the backend.
protected LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo>	kvStateInformation Key/Value state meta info from the backend.
protected String	localDirectoryName The local directory name of the current snapshot strategy.
protected org.apache.flink.runtime.state.LocalRecoveryConfig	localRecoveryConfig The configuration for local recovery.
protected org.apache.flink.util.ResourceGuard	rocksDBResourceGuard Resource guard for the RocksDB instance.

Constructor Summary

Constructors

Constructor and Description

RocksDBSnapshotStrategyBase(String description, org.rocksdb.RocksDB db, org.apache.flink.util.ResourceGuard rocksDBResourceGuard, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation, org.apache.flink.runtime.state.KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig, File instanceBasePath, UUID backendUID)

Method Summary

Modifier and Type	Method and Description
protected void	cleanupIncompleteSnapshot(org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry, org.apache.flink.runtime.state.SnapshotDirectory localBackupDirectory)
abstract void	close()
String	getDescription()
protected org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.StreamStateHandle>	materializeMetaData(org.apache.flink.core.fs.CloseableRegistry snapshotCloseableRegistry, org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry, List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots, long checkpointId, org.apache.flink.runtime.state.CheckpointStreamFactory checkpointStreamFactory)
protected org.apache.flink.runtime.state.SnapshotDirectory	prepareLocalSnapshotDirectory(long checkpointId)
protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot	snapshotMetaData(long checkpointId, List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots)
RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources	syncPrepareResources(long checkpointId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

asyncSnapshot

Field Detail

db

@Nonnull
protected org.rocksdb.RocksDB db

RocksDB instance from the backend.

rocksDBResourceGuard

@Nonnull
protected final org.apache.flink.util.ResourceGuard rocksDBResourceGuard

Resource guard for the RocksDB instance.

keySerializer

@Nonnull
protected final org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer

The key serializer of the backend.

kvStateInformation

@Nonnull
protected final LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation

Key/Value state meta info from the backend.

keyGroupRange

@Nonnull
protected final org.apache.flink.runtime.state.KeyGroupRange keyGroupRange

The key-group range for the task.

keyGroupPrefixBytes

@Nonnegative
protected final int keyGroupPrefixBytes

Number of bytes in the key-group prefix.

localRecoveryConfig

@Nonnull
protected final org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig

The configuration for local recovery.

instanceBasePath

@Nonnull
protected final File instanceBasePath

Base path of the RocksDB instance.

localDirectoryName

protected final String localDirectoryName

The local directory name of the current snapshot strategy.

backendUID

@Nonnull
protected final UUID backendUID

The state handle ids of all sst files materialized in snapshots for previous checkpoints.

EMPTY_PREVIOUS_SNAPSHOT

protected static final RocksDBSnapshotStrategyBase.PreviousSnapshot EMPTY_PREVIOUS_SNAPSHOT

Constructor Detail

RocksDBSnapshotStrategyBase

public RocksDBSnapshotStrategyBase(@Nonnull
                                   String description,
                                   @Nonnull
                                   org.rocksdb.RocksDB db,
                                   @Nonnull
                                   org.apache.flink.util.ResourceGuard rocksDBResourceGuard,
                                   @Nonnull
                                   org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                                   @Nonnull
                                   LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
                                   @Nonnull
                                   org.apache.flink.runtime.state.KeyGroupRange keyGroupRange,
                                   @Nonnegative
                                   int keyGroupPrefixBytes,
                                   @Nonnull
                                   org.apache.flink.runtime.state.LocalRecoveryConfig localRecoveryConfig,
                                   @Nonnull
                                   File instanceBasePath,
                                   @Nonnull
                                   UUID backendUID)

Method Detail

getDescription

@Nonnull
public String getDescription()

syncPrepareResources

public RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources syncPrepareResources(long checkpointId)
                                                                                throws Exception

Specified by:

syncPrepareResources in interface org.apache.flink.runtime.state.SnapshotStrategy<org.apache.flink.runtime.state.KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>

Throws:

Exception

snapshotMetaData

protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot snapshotMetaData(long checkpointId,
                                                                                 @Nonnull
                                                                                 List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots)

prepareLocalSnapshotDirectory

@Nonnull
protected org.apache.flink.runtime.state.SnapshotDirectory prepareLocalSnapshotDirectory(long checkpointId)
                                                                                           throws IOException

Throws:

IOException

cleanupIncompleteSnapshot

protected void cleanupIncompleteSnapshot(@Nonnull
                                         org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry,
                                         @Nonnull
                                         org.apache.flink.runtime.state.SnapshotDirectory localBackupDirectory)

materializeMetaData

@Nonnull
protected org.apache.flink.runtime.state.SnapshotResult<org.apache.flink.runtime.state.StreamStateHandle> materializeMetaData(@Nonnull
                                                                                                                                       org.apache.flink.core.fs.CloseableRegistry snapshotCloseableRegistry,
                                                                                                                                       @Nonnull
                                                                                                                                       org.apache.flink.core.fs.CloseableRegistry tmpResourcesRegistry,
                                                                                                                                       @Nonnull
                                                                                                                                       List<org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot> stateMetaInfoSnapshots,
                                                                                                                                       long checkpointId,
                                                                                                                                       @Nonnull
                                                                                                                                       org.apache.flink.runtime.state.CheckpointStreamFactory checkpointStreamFactory)
                                                                                                                                throws Exception

Throws:

Exception

close

public abstract void close()
                    throws IOException

Specified by:

close in interface AutoCloseable

Throws:

IOException