org.apache.flink.streaming.connectors.fs

Class DateTimeBucketer

java.lang.Object

org.apache.flink.streaming.connectors.fs.DateTimeBucketer

All Implemented Interfaces:

Serializable, Bucketer

public class DateTimeBucketer
extends Object
implements Bucketer

A Bucketer that assigns to buckets based on current system time.

The DateTimeBucketer will create directories of the following form: /{basePath}/{dateTimePath}/. The basePath is the path that was specified as a base path when creating the RollingSink. The dateTimePath is determined based on the current system time and the user provided format string.

SimpleDateFormat is used to derive a date string from the current system time and the date format string. The default format string is "yyyy-MM-dd--HH" so the rolling files will have a granularity of hours.

Example:

     Bucketer buck = new DateTimeBucketer("yyyy-MM-dd--HH");
 

This will create for example the following bucket path: /base/1976-12-31-14/

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DateTimeBucketer() Creates a new DateTimeBucketer with format string "yyyy-MM-dd--HH".
DateTimeBucketer(String formatString) Creates a new DateTimeBucketer with the given date/time format string.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	getNextBucketPath(org.apache.hadoop.fs.Path basePath) Returns the Path of a new bucket file.
static void	setClock(Clock newClock) This sets the internal Clock implementation.
boolean	shouldStartNewBucket(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.Path currentBucketPath) Returns true when a new bucket should be started.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DateTimeBucketer

public DateTimeBucketer()

Creates a new DateTimeBucketer with format string "yyyy-MM-dd--HH".

DateTimeBucketer

public DateTimeBucketer(String formatString)

Creates a new DateTimeBucketer with the given date/time format string.

Parameters:

formatString - The format string that will be given to SimpleDateFormat to determine the bucket path.

Method Detail

shouldStartNewBucket

public boolean shouldStartNewBucket(org.apache.hadoop.fs.Path basePath,
                                    org.apache.hadoop.fs.Path currentBucketPath)

Description copied from interface: Bucketer

Returns true when a new bucket should be started.

Specified by:

shouldStartNewBucket in interface Bucketer

currentBucketPath - The bucket Path that is currently being used.

getNextBucketPath

public org.apache.hadoop.fs.Path getNextBucketPath(org.apache.hadoop.fs.Path basePath)

Description copied from interface: Bucketer

Returns the Path of a new bucket file.

Specified by:

getNextBucketPath in interface Bucketer

Parameters:

basePath - The base path containing all the buckets.

Returns:

The complete new Path of the new bucket. This should include the basePath and also the subtaskIndex tp avoid clashes with parallel sinks.

toString

public String toString()

Overrides:

toString in class Object

setClock

public static void setClock(Clock newClock)

This sets the internal Clock implementation. This method should only be used for testing

Parameters:

newClock - The new clock to set.