org.apache.flink.streaming.connectors.fs.bucketing

Class DateTimeBucketer<T>

java.lang.Object

org.apache.flink.streaming.connectors.fs.bucketing.DateTimeBucketer<T>

All Implemented Interfaces:

Serializable, Bucketer<T>

public class DateTimeBucketer<T>
extends Object
implements Bucketer<T>

A Bucketer that assigns to buckets based on current system time.

The DateTimeBucketer will create directories of the following form: /{basePath}/{dateTimePath}/. The basePath is the path that was specified as a base path when creating the BucketingSink. The dateTimePath is determined based on the current system time and the user provided format string.

SimpleDateFormat is used to derive a date string from the current system time and the date format string. The default format string is "yyyy-MM-dd--HH" so the rolling files will have a granularity of hours.

Example:

     Bucketer buck = new DateTimeBucketer("yyyy-MM-dd--HH");
 

This will create for example the following bucket path: /base/1976-12-31-14/

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DateTimeBucketer() Creates a new DateTimeBucketer with format string "yyyy-MM-dd--HH".
DateTimeBucketer(String formatString) Creates a new DateTimeBucketer with the given date/time format string.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	getBucketPath(Clock clock, org.apache.hadoop.fs.Path basePath, T element) Returns the Path of a bucket file.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DateTimeBucketer

public DateTimeBucketer()

Creates a new DateTimeBucketer with format string "yyyy-MM-dd--HH".

DateTimeBucketer

public DateTimeBucketer(String formatString)

Creates a new DateTimeBucketer with the given date/time format string.

Parameters:

formatString - The format string that will be given to SimpleDateFormat to determine the bucket path.

Method Detail

getBucketPath

public org.apache.hadoop.fs.Path getBucketPath(Clock clock,
                                               org.apache.hadoop.fs.Path basePath,
                                               T element)

Description copied from interface: Bucketer

Returns the Path of a bucket file.

Specified by:

getBucketPath in interface Bucketer<T>

basePath - The base path containing all the buckets.

element - The current element being processed.

Returns:

The complete Path of the bucket which the provided element should fall in. This should include the basePath and also the subtaskIndex to avoid clashes with parallel sinks.

toString

public String toString()

Overrides:

toString in class Object