org.apache.flink.streaming.connectors.fs.bucketing

Class BucketingSink<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink<T>

Type Parameters:

T - Type of the elements emitted by this sink

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.InputTypeConfigurable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedRestoring<RollingSink.BucketState>, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>, org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

public class BucketingSink<T>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
implements org.apache.flink.api.java.typeutils.InputTypeConfigurable, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedRestoring<RollingSink.BucketState>, org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

Sink that emits its input elements to FileSystem files within buckets. This is integrated with the checkpointing mechanism to provide exactly once semantics.

When creating the sink a basePath must be specified. The base directory contains one directory for every bucket. The bucket directories themselves contain several part files, one for each parallel subtask of the sink. These part files contain the actual output data.

The sink uses a Bucketer to determine in which bucket directory each element should be written to inside the base directory. The Bucketer can, for example, use time or a property of the element to determine the bucket directory. The default Bucketer is a DateTimeBucketer which will create one new bucket every hour. You can specify a custom Bucketer using setBucketer(Bucketer). For example, use the BasePathBucketer if you don't want to have buckets but still want to write part-files in a fault-tolerant way.

The filenames of the part files contain the part prefix, the parallel subtask index of the sink and a rolling counter. For example the file "part-1-17" contains the data from subtask 1 of the sink and is the 17th bucket created by that subtask. Per default the part prefix is "part" but this can be configured using setPartPrefix(String). When a part file becomes bigger than the user-specified batch size the current part file is closed, the part counter is increased and a new part file is created. The batch size defaults to 384MB, this can be configured using setBatchSize(long).

In some scenarios, the open buckets are required to change based on time. In these cases, the sink needs to determine when a bucket has become inactive, in order to flush and close the part file. To support this there are two configurable settings:

1. the frequency to check for inactive buckets, configured by setInactiveBucketCheckInterval(long), and
2. the minimum amount of time a bucket has to not receive any data before it is considered inactive, configured by setInactiveBucketThreshold(long)

Both of these parameters default to 60,000 ms, or 1 min.

Part files can be in one of three states: in-progress, pending or finished. The reason for this is how the sink works together with the checkpointing mechanism to provide exactly-once semantics and fault-tolerance. The part file that is currently being written to is in-progress. Once a part file is closed for writing it becomes pending. When a checkpoint is successful the currently pending files will be moved to finished.

If case of a failure, and in order to guarantee exactly-once semantics, the sink should roll back to the state it had when that last successful checkpoint occurred. To this end, when restoring, the restored files in pending state are transferred into the finished state while any in-progress files are rolled back, so that they do not contain data that arrived after the checkpoint from which we restore. If the FileSystem supports the truncate() method this will be used to reset the file back to its previous state. If not, a special file with the same name as the part file and the suffix ".valid-length" will be created that contains the length up to which the file contains valid data. When reading the file, it must be ensured that it is only read up to that point. The prefixes and suffixes for the different file states and valid-length files can be configured using the adequate setter method, e.g. setPendingSuffix(String).

NOTE:

1. If checkpointing is not enabled the pending files will never be moved to the finished state. In that case, the pending suffix/prefix can be set to "" to make the sink work in a non-fault-tolerant way but still provide output without prefixes and suffixes.
2. The part files are written using an instance of Writer. By default, a StringWriter is used, which writes the result of toString() for every element, separated by newlines. You can configure the writer using the setWriter(Writer). For example, SequenceFileWriter can be used to write Hadoop SequenceFiles.

Example:

     new BucketingSink<Tuple2<IntWritable, Text>>(outPath)
         .setWriter(new SequenceFileWriter<IntWritable, Text>())
         .setBucketer(new DateTimeBucketer("yyyy-MM-dd--HHmm")
 

This will create a sink that writes to SequenceFiles and rolls every minute.

See Also:

DateTimeBucketer, StringWriter, SequenceFileWriter, Serialized Form

Constructor Summary

Constructors

Constructor and Description
BucketingSink(String basePath) Creates a new BucketingSink that writes files to the given base directory.

Method Summary

Modifier and Type	Method and Description
void	close()
BucketingSink<T>	disableCleanupOnOpen() Deprecated. This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.
org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.State<T>	getState()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	invoke(T value)
void	notifyCheckpointComplete(long checkpointId)
void	onProcessingTime(long timestamp)
void	open(org.apache.flink.configuration.Configuration parameters)
void	restoreState(RollingSink.BucketState state)
BucketingSink<T>	setAsyncTimeout(long timeout) Sets the default timeout for asynchronous operations such as recoverLease and truncate.
BucketingSink<T>	setBatchSize(long batchSize) Sets the maximum bucket size in bytes.
BucketingSink<T>	setBucketer(Bucketer<T> bucketer) Sets the Bucketer to use for determining the bucket files to write to.
BucketingSink<T>	setFSConfig(org.apache.flink.configuration.Configuration config) Specify a custom Configuration that will be used when creating the FileSystem for writing.
BucketingSink<T>	setFSConfig(org.apache.hadoop.conf.Configuration config) Specify a custom Configuration that will be used when creating the FileSystem for writing.
BucketingSink<T>	setInactiveBucketCheckInterval(long interval) Sets the default time between checks for inactive buckets.
BucketingSink<T>	setInactiveBucketThreshold(long threshold) Sets the default threshold for marking a bucket as inactive and closing its part files.
BucketingSink<T>	setInProgressPrefix(String inProgressPrefix) Sets the prefix of in-progress part files.
BucketingSink<T>	setInProgressSuffix(String inProgressSuffix) Sets the suffix of in-progress part files.
void	setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type, org.apache.flink.api.common.ExecutionConfig executionConfig)
BucketingSink<T>	setPartPrefix(String partPrefix) Sets the prefix of part files.
BucketingSink<T>	setPendingPrefix(String pendingPrefix) Sets the prefix of pending part files.
BucketingSink<T>	setPendingSuffix(String pendingSuffix) Sets the suffix of pending part files.
BucketingSink<T>	setValidLengthPrefix(String validLengthPrefix) Sets the prefix of valid-length files.
BucketingSink<T>	setValidLengthSuffix(String validLengthSuffix) Sets the suffix of valid-length files.
BucketingSink<T>	setWriter(Writer<T> writer) Sets the Writer to be used for writing the incoming elements to bucket files.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BucketingSink

public BucketingSink(String basePath)

Creates a new BucketingSink that writes files to the given base directory.

This uses aDateTimeBucketer as Bucketer and a StringWriter has writer. The maximum bucket size is set to 384 MB.

Parameters:

basePath - The directory to which to write the bucket files.

Method Detail

setFSConfig

public BucketingSink<T> setFSConfig(org.apache.flink.configuration.Configuration config)

Specify a custom Configuration that will be used when creating the FileSystem for writing.

setFSConfig

public BucketingSink<T> setFSConfig(org.apache.hadoop.conf.Configuration config)

Specify a custom Configuration that will be used when creating the FileSystem for writing.

setInputType

public void setInputType(org.apache.flink.api.common.typeinfo.TypeInformation<?> type,
                         org.apache.flink.api.common.ExecutionConfig executionConfig)

Specified by:

setInputType in interface org.apache.flink.api.java.typeutils.InputTypeConfigurable

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

invoke

public void invoke(T value)
            throws Exception

Specified by:

invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<T>

Specified by:

invoke in class org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>

Throws:

Exception

onProcessingTime

public void onProcessingTime(long timestamp)
                      throws Exception

Specified by:

onProcessingTime in interface org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

restoreState

public void restoreState(RollingSink.BucketState state)
                  throws Exception

Specified by:

restoreState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedRestoring<RollingSink.BucketState>

Throws:

Exception

setBatchSize

public BucketingSink<T> setBatchSize(long batchSize)

Sets the maximum bucket size in bytes.

When a bucket part file becomes larger than this size a new bucket part file is started and the old one is closed. The name of the bucket files depends on the Bucketer.

Parameters:

batchSize - The bucket part file size in bytes.

setInactiveBucketCheckInterval

public BucketingSink<T> setInactiveBucketCheckInterval(long interval)

Sets the default time between checks for inactive buckets.

Parameters:

interval - The timeout, in milliseconds.

setInactiveBucketThreshold

public BucketingSink<T> setInactiveBucketThreshold(long threshold)

Sets the default threshold for marking a bucket as inactive and closing its part files. Buckets which haven't been written to for at least this period of time become inactive.

Parameters:

threshold - The timeout, in milliseconds.

setBucketer

public BucketingSink<T> setBucketer(Bucketer<T> bucketer)

Sets the Bucketer to use for determining the bucket files to write to.

Parameters:

bucketer - The bucketer to use.

setWriter

public BucketingSink<T> setWriter(Writer<T> writer)

Sets the Writer to be used for writing the incoming elements to bucket files.

Parameters:

writer - The Writer to use.

setInProgressSuffix

public BucketingSink<T> setInProgressSuffix(String inProgressSuffix)

Sets the suffix of in-progress part files. The default is "in-progress".

setInProgressPrefix

public BucketingSink<T> setInProgressPrefix(String inProgressPrefix)

Sets the prefix of in-progress part files. The default is "_".

setPendingSuffix

public BucketingSink<T> setPendingSuffix(String pendingSuffix)

Sets the suffix of pending part files. The default is ".pending".

setPendingPrefix

public BucketingSink<T> setPendingPrefix(String pendingPrefix)

Sets the prefix of pending part files. The default is "_".

setValidLengthSuffix

public BucketingSink<T> setValidLengthSuffix(String validLengthSuffix)

Sets the suffix of valid-length files. The default is ".valid-length".

setValidLengthPrefix

public BucketingSink<T> setValidLengthPrefix(String validLengthPrefix)

Sets the prefix of valid-length files. The default is "_".

setPartPrefix

public BucketingSink<T> setPartPrefix(String partPrefix)

Sets the prefix of part files. The default is "part".

disableCleanupOnOpen

@Deprecated
public BucketingSink<T> disableCleanupOnOpen()

Deprecated. This option is deprecated and remains only for backwards compatibility. We do not clean up lingering files anymore.

Disable cleanup of leftover in-progress/pending files when the sink is opened.

This should only be disabled if using the sink without checkpoints, to not remove the files already in the directory.

setAsyncTimeout

public BucketingSink<T> setAsyncTimeout(long timeout)

Sets the default timeout for asynchronous operations such as recoverLease and truncate.

Parameters:

timeout - The timeout, in milliseconds.

getState

@VisibleForTesting
public org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink.State<T> getState()