DuplicatingStateChangeFsUploader (Flink : DSTL : DFS 1.16.1 API)

java.lang.Object
- org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
- - org.apache.flink.changelog.fs.DuplicatingStateChangeFsUploader

All Implemented Interfaces:

AutoCloseable, StateChangeUploader
```
public class DuplicatingStateChangeFsUploader
extends AbstractStateChangeFsUploader
```
A StateChangeFsUploader implementation that writes the changes to remote and local.
Local dstl files are only managed by TM side, LocalChangelogRegistry, TaskChangelogRegistry and ChangelogTaskLocalStateStore are responsible for managing them.
Remote dstl files are managed by TM side and JM side, TaskChangelogRegistry is responsible for TM side, and SharedStateRegistry is responsible for JM side.
The total discard logic of local dstl files is:
1. Register files to TaskChangelogRegistry.startTracking(org.apache.flink.runtime.state.StreamStateHandle, java.util.Set<java.util.UUID>) on AbstractStateChangeFsUploader.upload(java.util.Collection<org.apache.flink.changelog.fs.StateChangeUploadScheduler.UploadTask>).
2. Store the meta of files into ChangelogTaskLocalStateStore by AsyncCheckpointRunnable#reportCompletedSnapshotStates().
3. Pass control of the file to LocalChangelogRegistry#register when ChangelogKeyedStateBackend#notifyCheckpointComplete() , files of the previous checkpoint will be deleted by LocalChangelogRegistry#discardUpToCheckpoint at the same time.
4. When ChangelogTruncateHelper#materialized() or ChangelogTruncateHelper#checkpointSubsumed() is called, TaskChangelogRegistry.notUsed(org.apache.flink.runtime.state.StreamStateHandle, java.util.UUID) is responsible for deleting local files.
5. When one checkpoint is aborted, the dstl files of this checkpoint will be deleted by LocalChangelogRegistry#prune in FsStateChangelogWriter.reset(org.apache.flink.runtime.state.changelog.SequenceNumber, org.apache.flink.runtime.state.changelog.SequenceNumber, long).

Nested Class Summary
- Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader
  StateChangeUploader.UploadTasksResult

Field Summary
- Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
  bufferSize, compression, metrics

Constructor Summary

Constructors
Constructor and Description
`DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID, org.apache.flink.core.fs.Path basePath, org.apache.flink.core.fs.FileSystem fileSystem, boolean compression, int bufferSize, ChangelogStorageMetricGroup metrics, TaskChangelogRegistry changelogRegistry, org.apache.flink.runtime.state.LocalRecoveryDirectoryProvider localRecoveryDirectoryProvider)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type Method and Description

void close()

org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
- Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
  generateFileName, upload
- Methods inherited from class java.lang.Object
  clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`org.apache.flink.changelog.fs.OutputStreamWithPos`	`prepareStream()`

Constructor Detail

DuplicatingStateChangeFsUploader

public DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID,
                                        org.apache.flink.core.fs.Path basePath,
                                        org.apache.flink.core.fs.FileSystem fileSystem,
                                        boolean compression,
                                        int bufferSize,
                                        ChangelogStorageMetricGroup metrics,
                                        TaskChangelogRegistry changelogRegistry,
                                        org.apache.flink.runtime.state.LocalRecoveryDirectoryProvider localRecoveryDirectoryProvider)

Method Detail

prepareStream

public org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
                                                                throws IOException

Throws:: IOException

close

public void close()
           throws Exception

Throws:: Exception

Class DuplicatingStateChangeFsUploader

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader

Field Summary

Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

Constructor Summary

Method Summary

Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

Methods inherited from class java.lang.Object

Constructor Detail

DuplicatingStateChangeFsUploader

Method Detail

prepareStream

close