org.apache.hudi.util

Class StreamerUtil

java.lang.Object

org.apache.hudi.util.StreamerUtil

public class StreamerUtil
extends Object

Utilities for Flink stream read and write.

Constructor Summary

Constructors

Constructor and Description
StreamerUtil()

Method Summary

Modifier and Type	Method and Description
static TypedProperties	appendKafkaProps(FlinkStreamerConfig config)
static TypedProperties	buildProperties(List<String> props)
static HoodieTableMetaClient	createMetaClient(org.apache.flink.configuration.Configuration conf) Creates the meta client.
static HoodieTableMetaClient	createMetaClient(String basePath, org.apache.hadoop.conf.Configuration hadoopConf) Creates the meta client.
static Option<Transformer>	createTransformer(List<String> classNames)
static boolean	fileExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static TypedProperties	flinkConf2TypedProperties(org.apache.flink.configuration.Configuration conf) Converts the give Configuration to TypedProperties.
static String	generateBucketKey(String partitionPath, String fileId) Generates the bucket ID using format {partition path}_{fileID}.
static HoodieIndexConfig	getIndexConfig(org.apache.flink.configuration.Configuration conf) Returns the index config with given configuration.
static String	getLastCompletedInstant(HoodieTableMetaClient metaClient)
static String	getLastPendingInstant(HoodieTableMetaClient metaClient)
static String	getLastPendingInstant(HoodieTableMetaClient metaClient, boolean reloadTimeline)
static org.apache.avro.Schema	getLatestTableSchema(String path, org.apache.hadoop.conf.Configuration hadoopConf)
static long	getMaxCompactionMemoryInBytes(org.apache.flink.configuration.Configuration conf) Returns the max compaction memory in bytes with given conf.
static HoodiePayloadConfig	getPayloadConfig(org.apache.flink.configuration.Configuration conf) Returns the payload config with given configuration.
static TypedProperties	getProps(FlinkStreamerConfig cfg)
static org.apache.avro.Schema	getSourceSchema(org.apache.flink.configuration.Configuration conf)
static org.apache.avro.Schema	getTableAvroSchema(HoodieTableMetaClient metaClient, boolean includeMetadataFields)
static Option<HoodieTableConfig>	getTableConfig(String basePath, org.apache.hadoop.conf.Configuration hadoopConf) Returns the table config or empty if the table does not exist.
static boolean	haveSuccessfulCommits(HoodieTableMetaClient metaClient) Returns whether there are successful commits on the timeline.
static HoodieTableMetaClient	initTableIfNotExists(org.apache.flink.configuration.Configuration conf) Initialize the table if it does not exist.
static HoodieTableMetaClient	initTableIfNotExists(org.apache.flink.configuration.Configuration conf, org.apache.hadoop.conf.Configuration hadoopConf) Initialize the table if it does not exist.
static long	instantTimeDiffSeconds(String newInstantTime, String oldInstantTime) Returns the time interval in seconds between the given instant time.
static boolean	isValidFile(org.apache.hadoop.fs.FileStatus fileStatus) Returns whether the give file is in valid hoodie format.
static boolean	isWriteCommit(HoodieTableType tableType, HoodieInstant instant, HoodieTimeline timeline) Returns whether the given instant is a data writing commit.
static Option<String>	medianInstantTime(String highVal, String lowVal) Returns the median instant time between the given two instant time.
static HoodieTableMetaClient	metaClientForReader(org.apache.flink.configuration.Configuration conf, org.apache.hadoop.conf.Configuration hadoopConf) Creates the meta client for reader.
static boolean	partitionExists(String tablePath, String partitionPath, org.apache.hadoop.conf.Configuration hadoopConf) Returns whether the hoodie partition exists under given table path tablePath and partition path partitionPath.
static DFSPropertiesConfiguration	readConfig(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.Path cfgPath, List<String> overriddenProps) Read config from properties file (`--props` option) and cmd line (`--hoodie-conf` option).
static boolean	tableExists(String basePath, org.apache.hadoop.conf.Configuration hadoopConf) Returns whether the hoodie table exists under given path basePath.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamerUtil

public StreamerUtil()

Method Detail

appendKafkaProps

public static TypedProperties appendKafkaProps(FlinkStreamerConfig config)

getProps

public static TypedProperties getProps(FlinkStreamerConfig cfg)

buildProperties

public static TypedProperties buildProperties(List<String> props)

getSourceSchema

public static org.apache.avro.Schema getSourceSchema(org.apache.flink.configuration.Configuration conf)

readConfig

public static DFSPropertiesConfiguration readConfig(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                    org.apache.hadoop.fs.Path cfgPath,
                                                    List<String> overriddenProps)

Read config from properties file (`--props` option) and cmd line (`--hoodie-conf` option).

getPayloadConfig

public static HoodiePayloadConfig getPayloadConfig(org.apache.flink.configuration.Configuration conf)

Returns the payload config with given configuration.

getIndexConfig

public static HoodieIndexConfig getIndexConfig(org.apache.flink.configuration.Configuration conf)

Returns the index config with given configuration.

flinkConf2TypedProperties

public static TypedProperties flinkConf2TypedProperties(org.apache.flink.configuration.Configuration conf)

Converts the give Configuration to TypedProperties. The default values are also set up.

Parameters:

conf - The flink configuration

Returns:

a TypedProperties instance

initTableIfNotExists

public static HoodieTableMetaClient initTableIfNotExists(org.apache.flink.configuration.Configuration conf)
                                                  throws IOException

Initialize the table if it does not exist.

Parameters:

conf - the configuration

Throws:

IOException - if errors happens when writing metadata

initTableIfNotExists

public static HoodieTableMetaClient initTableIfNotExists(org.apache.flink.configuration.Configuration conf,
                                                         org.apache.hadoop.conf.Configuration hadoopConf)
                                                  throws IOException

Initialize the table if it does not exist.

Parameters:

conf - the configuration

Throws:

IOException - if errors happens when writing metadata

tableExists

public static boolean tableExists(String basePath,
                                  org.apache.hadoop.conf.Configuration hadoopConf)

Returns whether the hoodie table exists under given path basePath.

partitionExists

public static boolean partitionExists(String tablePath,
                                      String partitionPath,
                                      org.apache.hadoop.conf.Configuration hadoopConf)

Returns whether the hoodie partition exists under given table path tablePath and partition path partitionPath.

Parameters:

tablePath - Base path of the table.

partitionPath - The path of the partition.

hadoopConf - The hadoop configuration.

generateBucketKey

public static String generateBucketKey(String partitionPath,
                                       String fileId)

Generates the bucket ID using format {partition path}_{fileID}.

metaClientForReader

public static HoodieTableMetaClient metaClientForReader(org.apache.flink.configuration.Configuration conf,
                                                        org.apache.hadoop.conf.Configuration hadoopConf)

Creates the meta client for reader.

The streaming pipeline process is long-running, so empty table path is allowed, the reader would then check and refresh the meta client.

See Also:

StreamReadMonitoringFunction

createMetaClient

public static HoodieTableMetaClient createMetaClient(String basePath,
                                                     org.apache.hadoop.conf.Configuration hadoopConf)

Creates the meta client.

createMetaClient

public static HoodieTableMetaClient createMetaClient(org.apache.flink.configuration.Configuration conf)

Creates the meta client.

getTableConfig

public static Option<HoodieTableConfig> getTableConfig(String basePath,
                                                       org.apache.hadoop.conf.Configuration hadoopConf)

Returns the table config or empty if the table does not exist.

medianInstantTime

public static Option<String> medianInstantTime(String highVal,
                                               String lowVal)

Returns the median instant time between the given two instant time.

instantTimeDiffSeconds

public static long instantTimeDiffSeconds(String newInstantTime,
                                          String oldInstantTime)

Returns the time interval in seconds between the given instant time.

createTransformer

public static Option<Transformer> createTransformer(List<String> classNames)
                                             throws IOException

Throws:

IOException

isValidFile

public static boolean isValidFile(org.apache.hadoop.fs.FileStatus fileStatus)

Returns whether the give file is in valid hoodie format. For example, filtering out the empty or corrupt files.

getLastPendingInstant

public static String getLastPendingInstant(HoodieTableMetaClient metaClient)

getLastPendingInstant

public static String getLastPendingInstant(HoodieTableMetaClient metaClient,
                                           boolean reloadTimeline)

getLastCompletedInstant

public static String getLastCompletedInstant(HoodieTableMetaClient metaClient)

haveSuccessfulCommits

public static boolean haveSuccessfulCommits(HoodieTableMetaClient metaClient)

Returns whether there are successful commits on the timeline.

Parameters:

metaClient - The meta client

Returns:

true if there is any successful commit

getMaxCompactionMemoryInBytes

public static long getMaxCompactionMemoryInBytes(org.apache.flink.configuration.Configuration conf)

Returns the max compaction memory in bytes with given conf.

getTableAvroSchema

public static org.apache.avro.Schema getTableAvroSchema(HoodieTableMetaClient metaClient,
                                                        boolean includeMetadataFields)
                                                 throws Exception

Throws:

Exception

getLatestTableSchema

public static org.apache.avro.Schema getLatestTableSchema(String path,
                                                          org.apache.hadoop.conf.Configuration hadoopConf)

fileExists

public static boolean fileExists(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path)

isWriteCommit

public static boolean isWriteCommit(HoodieTableType tableType,
                                    HoodieInstant instant,
                                    HoodieTimeline timeline)

Returns whether the given instant is a data writing commit.

Parameters:

tableType - The table type

instant - The instant

timeline - The timeline