org.apache.flink.fs.osshadoop.writer

Class OSSRecoverableWriter

java.lang.Object

org.apache.flink.fs.osshadoop.writer.OSSRecoverableWriter

All Implemented Interfaces:

org.apache.flink.core.fs.RecoverableWriter

public class OSSRecoverableWriter
extends Object
implements org.apache.flink.core.fs.RecoverableWriter

An implementation of the RecoverableWriter against OSS.

This implementation makes heavy use of MultiPart Uploads in OSS to persist intermediate data as soon as possible.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.core.fs.RecoverableWriter

org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable

Constructor Summary

Constructors

Constructor and Description
OSSRecoverableWriter(OSSAccessor ossAccessor, long ossUploadPartSize, int streamConcurrentUploads, Executor executor, org.apache.flink.util.function.FunctionWithException<File,org.apache.flink.core.fs.RefCountedFileWithStream,IOException> cachedFileCreator)

Method Summary

Modifier and Type	Method and Description
boolean	cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable>	getCommitRecoverableSerializer()
org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable>	getResumeRecoverableSerializer()
org.apache.flink.core.fs.RecoverableFsDataOutputStream	open(org.apache.flink.core.fs.Path path)
OSSRecoverableFsDataOutputStream	recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer	recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable resumable)
boolean	requiresCleanupOfRecoverableState()
boolean	supportsResume()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OSSRecoverableWriter

public OSSRecoverableWriter(OSSAccessor ossAccessor,
                            long ossUploadPartSize,
                            int streamConcurrentUploads,
                            Executor executor,
                            org.apache.flink.util.function.FunctionWithException<File,org.apache.flink.core.fs.RefCountedFileWithStream,IOException> cachedFileCreator)

Method Detail

open

public org.apache.flink.core.fs.RecoverableFsDataOutputStream open(org.apache.flink.core.fs.Path path)
                                                            throws IOException

Specified by:

open in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

recover

public OSSRecoverableFsDataOutputStream recover(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
                                         throws IOException

Specified by:

recover in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

requiresCleanupOfRecoverableState

public boolean requiresCleanupOfRecoverableState()

Specified by:

requiresCleanupOfRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

cleanupRecoverableState

public boolean cleanupRecoverableState(org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable)
                                throws IOException

Specified by:

cleanupRecoverableState in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

recoverForCommit

public org.apache.flink.core.fs.RecoverableFsDataOutputStream.Committer recoverForCommit(org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable resumable)
                                                                                  throws IOException

Specified by:

recoverForCommit in interface org.apache.flink.core.fs.RecoverableWriter

Throws:

IOException

getCommitRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.CommitRecoverable> getCommitRecoverableSerializer()

Specified by:

getCommitRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

getResumeRecoverableSerializer

public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable> getResumeRecoverableSerializer()

Specified by:

getResumeRecoverableSerializer in interface org.apache.flink.core.fs.RecoverableWriter

supportsResume

public boolean supportsResume()

Specified by:

supportsResume in interface org.apache.flink.core.fs.RecoverableWriter