RollingSink (flink-connector-filesystem 1.6.4 API)

java.lang.Object
- org.apache.flink.api.common.functions.AbstractRichFunction
- - org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
  - - org.apache.flink.streaming.connectors.fs.RollingSink<T>

类型参数:

T - Type of the elements emitted by this sink

所有已实现的接口:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.InputTypeConfigurable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

已过时。
use BucketingSink instead.
```
@Deprecated
public class RollingSink<T>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
implements org.apache.flink.api.java.typeutils.InputTypeConfigurable, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.runtime.state.CheckpointListener
```
Sink that emits its input elements to rolling FileSystem files. This is integrated with the checkpointing mechanism to provide exactly once semantics.
When creating the sink a basePath must be specified. The base directory contains one directory for every bucket. The bucket directories themselves contain several part files. These contain the actual written data.
The sink uses a Bucketer to determine the name of bucket directories inside the base directory. Whenever the Bucketer returns a different directory name than it returned before the sink will close the current part files inside that bucket and start the new bucket directory. The default bucketer is a DateTimeBucketer with date format string ""yyyy-MM-dd--HH". You can specify a custom Bucketer using setBucketer(Bucketer). For example, use NonRollingBucketer if you don't want to have buckets but still write part files in a fault-tolerant way.
The filenames of the part files contain the part prefix, the parallel subtask index of the sink and a rolling counter, for example "part-1-17". Per default the part prefix is "part" but this can be configured using setPartPrefix(String). When a part file becomes bigger than the batch size the current part file is closed, the part counter is increased and a new part file is created. The batch size defaults to 384MB, this can be configured using setBatchSize(long).
Part files can be in one of three states: in-progress, pending or finished. The reason for this is how the sink works together with the checkpointing mechanism to provide exactly-once semantics and fault-tolerance. The part file that is currently being written to is in-progress. Once a part file is closed for writing it becomes pending. When a checkpoint is successful the currently pending files will be moved to finished. If a failure occurs the pending files will be deleted to reset state to the last checkpoint. The data in in-progress files will also have to be rolled back. If the FileSystem supports the truncate call this will be used to reset the file back to a previous state. If not, a special file with the same name as the part file and the suffix ".valid-length" will be written that contains the length up to which the file contains valid data. When reading the file it must be ensured that it is only read up to that point. The prefixes and suffixes for the different file states and valid-length files can be configured, for example with setPendingSuffix(String).
Note: If checkpointing is not enabled the pending files will never be moved to the finished state. In that case, the pending suffix/prefix can be set to "" to make the sink work in a non-fault-tolerant way but still provide output without prefixes and suffixes.
The part files are written using an instance of Writer. By default StringWriter is used, which writes the result of toString() for every element. Separated by newlines. You can configure the writer using setWriter(Writer). For example, SequenceFileWriter can be used to write Hadoop SequenceFiles.
Example:
```
     new RollingSink<Tuple2<IntWritable, Text>>(outPath)
         .setWriter(new SequenceFileWriter<IntWritable, Text>())
         .setBucketer(new DateTimeBucketer("yyyy-MM-dd--HHmm")
 
```
This will create a sink that writes to SequenceFiles and rolls every minute.
另请参阅:

DateTimeBucketer, StringWriter, SequenceFileWriter, 序列化表格

嵌套类概要

嵌套类
限定符和类型	类和说明
`static class`	`RollingSink.BucketState` 已过时。 This is used for keeping track of the current in-progress files and files that we mark for moving from pending to final location after we get a checkpoint-complete notification.

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.sink.SinkFunction
org.apache.flink.streaming.api.functions.sink.SinkFunction.Context<T>

构造器概要

构造器
构造器和说明

RollingSink(String basePath)
已过时。

Creates a new RollingSink that writes files to the given base directory.

构造器
构造器和说明
`RollingSink(String basePath)` 已过时。 Creates a new `RollingSink` that writes files to the given base directory.

方法概要

所有方法实例方法具体方法已过时的方法
限定符和类型	方法和说明
`void`	`close()` 已过时。
`RollingSink<T>`	`disableCleanupOnOpen()` 已过时。 This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.
`void`	`initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)` 已过时。
`void`	`invoke(T value)` 已过时。
`void`	`notifyCheckpointComplete(long checkpointId)` 已过时。
`void`	`open(org.apache.flink.configuration.Configuration parameters)` 已过时。
`RollingSink<T>`	`setAsyncTimeout(long timeout)` 已过时。 Sets the default timeout for asynchronous operations such as recoverLease and truncate.
`RollingSink<T>`	`setBatchSize(long batchSize)` 已过时。 Sets the maximum bucket size in bytes.
`RollingSink<T>`	`setBucketer(Bucketer bucketer)` 已过时。 Sets the `Bucketer` to use for determining the bucket files to write to.
`RollingSink<T>`	`setFSConfig(org.apache.flink.configuration.Configuration config)` 已过时。 Specify a custom `Configuration` that will be used when creating the `FileSystem` for writing.
`RollingSink<T>`	`setFSConfig(org.apache.hadoop.conf.Configuration config)` 已过时。 Specify a custom `Configuration` that will be used when creating the `FileSystem` for writing.
`RollingSink<T>`	`setInProgressPrefix(String inProgressPrefix)` 已过时。 Sets the prefix of in-progress part files.
`RollingSink<T>`	`setInProgressSuffix(String inProgressSuffix)` 已过时。 Sets the suffix of in-progress part files.
`void`	`setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)` 已过时。
`RollingSink<T>`	`setPartPrefix(String partPrefix)` 已过时。 Sets the prefix of part files.
`RollingSink<T>`	`setPendingPrefix(String pendingPrefix)` 已过时。 Sets the prefix of pending part files.
`RollingSink<T>`	`setPendingSuffix(String pendingSuffix)` 已过时。 Sets the suffix of pending part files.
`RollingSink<T>`	`setValidLengthPrefix(String validLengthPrefix)` 已过时。 Sets the prefix of valid-length files.
`RollingSink<T>`	`setValidLengthSuffix(String validLengthSuffix)` 已过时。 Sets the suffix of valid-length files.
`RollingSink<T>`	`setWriter(Writer<T> writer)` 已过时。 Sets the `Writer` to be used for writing the incoming elements to bucket files.
`void`	`snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)` 已过时。

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction
getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

从类继承的方法 java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.functions.sink.SinkFunction
invoke

- 构造器详细资料
  - RollingSink
```
public RollingSink(String basePath)
```
    已过时。
    
    Creates a new RollingSink that writes files to the given base directory.
    This uses aDateTimeBucketer as bucketer and a StringWriter has writer. The maximum bucket size is set to 384 MB.
    
    参数:
    
    basePath - The directory to which to write the bucket files.
- 方法详细资料
  - setFSConfig
```
public RollingSink<T> setFSConfig(org.apache.flink.configuration.Configuration config)
```
    已过时。
    
    Specify a custom Configuration that will be used when creating the FileSystem for writing.
  - setFSConfig
```
public RollingSink<T> setFSConfig(org.apache.hadoop.conf.Configuration config)
```
    已过时。
    
    Specify a custom Configuration that will be used when creating the FileSystem for writing.
  - setInputType
```
public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)
```
    已过时。
    
    指定者:
    
    setInputType 在接口中 org.apache.flink.api.java.typeutils.InputTypeConfigurable
  - initializeState
```
public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception
```
    已过时。
    
    指定者:
    
    initializeState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
    
    抛出:
    
    Exception
  - open
```
public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception
```
    已过时。
    
    指定者:
    
    open 在接口中 org.apache.flink.api.common.functions.RichFunction
    
    覆盖:
    
    open 在类中 org.apache.flink.api.common.functions.AbstractRichFunction
    
    抛出:
    
    Exception
  - close
```
public void close()
           throws Exception
```
    已过时。
    
    指定者:
    
    close 在接口中 org.apache.flink.api.common.functions.RichFunction
    
    覆盖:
    
    close 在类中 org.apache.flink.api.common.functions.AbstractRichFunction
    
    抛出:
    
    Exception
  - invoke
```
public void invoke(T value)
            throws Exception
```
    已过时。
    
    指定者:
    
    invoke 在接口中 org.apache.flink.streaming.api.functions.sink.SinkFunction<T>
    
    抛出:
    
    Exception
  - notifyCheckpointComplete
```
public void notifyCheckpointComplete(long checkpointId)
                              throws Exception
```
    已过时。
    
    指定者:
    
    notifyCheckpointComplete 在接口中 org.apache.flink.runtime.state.CheckpointListener
    
    抛出:
    
    Exception
  - snapshotState
```
public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception
```
    已过时。
    
    指定者:
    
    snapshotState 在接口中 org.apache.flink.streaming.api.checkpoint.CheckpointedFunction
    
    抛出:
    
    Exception
  - setBatchSize
```
public RollingSink<T> setBatchSize(long batchSize)
```
    已过时。
    
    Sets the maximum bucket size in bytes.
    When a bucket part file becomes larger than this size a new bucket part file is started and the old one is closed. The name of the bucket files depends on the Bucketer.
    
    参数:
    
    batchSize - The bucket part file size in bytes.
  - setBucketer
```
public RollingSink<T> setBucketer(Bucketer bucketer)
```
    已过时。
    
    Sets the Bucketer to use for determining the bucket files to write to.
    
    参数:
    
    bucketer - The bucketer to use.
  - setWriter
```
public RollingSink<T> setWriter(Writer<T> writer)
```
    已过时。
    
    Sets the Writer to be used for writing the incoming elements to bucket files.
    
    参数:
    
    writer - The Writer to use.
  - setInProgressSuffix
```
public RollingSink<T> setInProgressSuffix(String inProgressSuffix)
```
    已过时。
    
    Sets the suffix of in-progress part files. The default is "in-progress".
  - setInProgressPrefix
```
public RollingSink<T> setInProgressPrefix(String inProgressPrefix)
```
    已过时。
    
    Sets the prefix of in-progress part files. The default is "_".
  - setPendingSuffix
```
public RollingSink<T> setPendingSuffix(String pendingSuffix)
```
    已过时。
    
    Sets the suffix of pending part files. The default is ".pending".
  - setPendingPrefix
```
public RollingSink<T> setPendingPrefix(String pendingPrefix)
```
    已过时。
    
    Sets the prefix of pending part files. The default is "_".
  - setValidLengthSuffix
```
public RollingSink<T> setValidLengthSuffix(String validLengthSuffix)
```
    已过时。
    
    Sets the suffix of valid-length files. The default is ".valid-length".
  - setValidLengthPrefix
```
public RollingSink<T> setValidLengthPrefix(String validLengthPrefix)
```
    已过时。
    
    Sets the prefix of valid-length files. The default is "_".
  - setPartPrefix
```
public RollingSink<T> setPartPrefix(String partPrefix)
```
    已过时。
    
    Sets the prefix of part files. The default is "part".
  - disableCleanupOnOpen
```
@Deprecated
public RollingSink<T> disableCleanupOnOpen()
```
    已过时。 This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.
    
    Disable cleanup of leftover in-progress/pending files when the sink is opened.
    This should only be disabled if using the sink without checkpoints, to not remove the files already in the directory.
  - setAsyncTimeout
```
public RollingSink<T> setAsyncTimeout(long timeout)
```
    已过时。
    
    Sets the default timeout for asynchronous operations such as recoverLease and truncate.
    
    参数:
    
    timeout - The timeout, in milliseconds.

类 RollingSink<T>

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.streaming.api.functions.sink.SinkFunction

构造器概要

方法概要

从类继承的方法 org.apache.flink.api.common.functions.AbstractRichFunction

从类继承的方法 java.lang.Object

从接口继承的方法 org.apache.flink.streaming.api.functions.sink.SinkFunction

构造器详细资料

RollingSink

方法详细资料

setFSConfig

setFSConfig

setInputType

initializeState

open

close

invoke

notifyCheckpointComplete

snapshotState

setBatchSize

setBucketer

setWriter

setInProgressSuffix

setInProgressPrefix

setPendingSuffix

setPendingPrefix

setValidLengthSuffix

setValidLengthPrefix

setPartPrefix

disableCleanupOnOpen

setAsyncTimeout