org.apache.hudi.util

Class StreamerUtil

java.lang.Object

org.apache.hudi.util.StreamerUtil

public class StreamerUtil
extends Object

Utilities for Flink stream read and write.

Constructor Summary

Constructors

Constructor and Description
StreamerUtil()

Method Summary

Modifier and Type	Method and Description
static TypedProperties	appendKafkaProps(FlinkStreamerConfig config)
static void	checkRequiredProperties(TypedProperties props, List<String> checkPropNames)
static HoodieTableMetaClient	createMetaClient(org.apache.flink.configuration.Configuration conf) Creates the meta client.
static HoodieTableMetaClient	createMetaClient(String basePath, org.apache.hadoop.conf.Configuration hadoopConf) Creates the meta client.
static Option<Transformer>	createTransformer(List<String> classNames)
static HoodieFlinkWriteClient	createWriteClient(org.apache.flink.configuration.Configuration conf) Creates the Flink write client.
static HoodieFlinkWriteClient	createWriteClient(org.apache.flink.configuration.Configuration conf, org.apache.flink.api.common.functions.RuntimeContext runtimeContext) Creates the Flink write client.
static HoodieFlinkWriteClient	createWriteClient(org.apache.flink.configuration.Configuration conf, org.apache.flink.api.common.functions.RuntimeContext runtimeContext, boolean loadFsViewStorageConfig) Creates the Flink write client.
static boolean	fileExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static TypedProperties	flinkConf2TypedProperties(org.apache.flink.configuration.Configuration conf) Converts the give Configuration to TypedProperties.
static String	generateBucketKey(String partitionPath, String fileId) Generates the bucket ID using format {partition path}_{fileID}.
static String	getAuxiliaryPath(org.apache.flink.configuration.Configuration conf) Returns the auxiliary path.
static HoodieWriteConfig	getHoodieClientConfig(org.apache.flink.configuration.Configuration conf) Mainly used for tests.
static HoodieWriteConfig	getHoodieClientConfig(org.apache.flink.configuration.Configuration conf, boolean loadFsViewStorageConfig)
static HoodieWriteConfig	getHoodieClientConfig(org.apache.flink.configuration.Configuration conf, boolean enableEmbeddedTimelineService, boolean loadFsViewStorageConfig)
static String	getLastCompletedInstant(HoodieTableMetaClient metaClient)
static String	getLastPendingInstant(HoodieTableMetaClient metaClient)
static String	getLastPendingInstant(HoodieTableMetaClient metaClient, boolean reloadTimeline)
static org.apache.avro.Schema	getLatestTableSchema(String path, org.apache.hadoop.conf.Configuration hadoopConf)
static long	getMaxCompactionMemoryInBytes(org.apache.flink.configuration.Configuration conf) Returns the max compaction memory in bytes with given conf.
static HoodiePayloadConfig	getPayloadConfig(org.apache.flink.configuration.Configuration conf) Returns the payload config with given configuration.
static TypedProperties	getProps(FlinkStreamerConfig cfg)
static org.apache.avro.Schema	getSourceSchema(org.apache.flink.configuration.Configuration conf)
static org.apache.avro.Schema	getSourceSchema(FlinkStreamerConfig cfg)
static org.apache.avro.Schema	getTableAvroSchema(HoodieTableMetaClient metaClient, boolean includeMetadataFields)
static boolean	haveSuccessfulCommits(HoodieTableMetaClient metaClient) Returns whether there are successful commits on the timeline.
static HoodieTableMetaClient	initTableIfNotExists(org.apache.flink.configuration.Configuration conf) Initialize the table if it does not exist.
static HoodieTableMetaClient	initTableIfNotExists(org.apache.flink.configuration.Configuration conf, org.apache.hadoop.conf.Configuration hadoopConf) Initialize the table if it does not exist.
static long	instantTimeDiffSeconds(String newInstantTime, String oldInstantTime) Returns the time interval in seconds between the given instant time.
static boolean	isValidFile(org.apache.hadoop.fs.FileStatus fileStatus) Returns whether the give file is in valid hoodie format.
static Option<String>	medianInstantTime(String highVal, String lowVal) Returns the median instant time between the given two instant time.
static HoodieTableMetaClient	metaClientForReader(org.apache.flink.configuration.Configuration conf, org.apache.hadoop.conf.Configuration hadoopConf) Creates the meta client for reader.
static DFSPropertiesConfiguration	readConfig(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.Path cfgPath, List<String> overriddenProps) Read config from properties file (`--props` option) and cmd line (`--hoodie-conf` option).
static boolean	tableExists(String basePath, org.apache.hadoop.conf.Configuration hadoopConf) Returns whether the hoodie table exists under given path basePath.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamerUtil

public StreamerUtil()

Method Detail

appendKafkaProps

public static TypedProperties appendKafkaProps(FlinkStreamerConfig config)

getProps

public static TypedProperties getProps(FlinkStreamerConfig cfg)

getSourceSchema

public static org.apache.avro.Schema getSourceSchema(FlinkStreamerConfig cfg)

getSourceSchema

public static org.apache.avro.Schema getSourceSchema(org.apache.flink.configuration.Configuration conf)

readConfig

public static DFSPropertiesConfiguration readConfig(org.apache.hadoop.conf.Configuration hadoopConfig,
                                                    org.apache.hadoop.fs.Path cfgPath,
                                                    List<String> overriddenProps)

Read config from properties file (`--props` option) and cmd line (`--hoodie-conf` option).

getHoodieClientConfig

public static HoodieWriteConfig getHoodieClientConfig(org.apache.flink.configuration.Configuration conf)

Mainly used for tests.

getHoodieClientConfig

public static HoodieWriteConfig getHoodieClientConfig(org.apache.flink.configuration.Configuration conf,
                                                      boolean loadFsViewStorageConfig)

getHoodieClientConfig

public static HoodieWriteConfig getHoodieClientConfig(org.apache.flink.configuration.Configuration conf,
                                                      boolean enableEmbeddedTimelineService,
                                                      boolean loadFsViewStorageConfig)

getPayloadConfig

public static HoodiePayloadConfig getPayloadConfig(org.apache.flink.configuration.Configuration conf)

Returns the payload config with given configuration.

flinkConf2TypedProperties

public static TypedProperties flinkConf2TypedProperties(org.apache.flink.configuration.Configuration conf)

Converts the give Configuration to TypedProperties. The default values are also set up.

Parameters:

conf - The flink configuration

Returns:

a TypedProperties instance

checkRequiredProperties

public static void checkRequiredProperties(TypedProperties props,
                                           List<String> checkPropNames)

initTableIfNotExists

public static HoodieTableMetaClient initTableIfNotExists(org.apache.flink.configuration.Configuration conf)
                                                  throws IOException

Initialize the table if it does not exist.

Parameters:

conf - the configuration

Throws:

IOException - if errors happens when writing metadata

initTableIfNotExists

public static HoodieTableMetaClient initTableIfNotExists(org.apache.flink.configuration.Configuration conf,
                                                         org.apache.hadoop.conf.Configuration hadoopConf)
                                                  throws IOException

Initialize the table if it does not exist.

Parameters:

conf - the configuration

Throws:

IOException - if errors happens when writing metadata

tableExists

public static boolean tableExists(String basePath,
                                  org.apache.hadoop.conf.Configuration hadoopConf)

Returns whether the hoodie table exists under given path basePath.

generateBucketKey

public static String generateBucketKey(String partitionPath,
                                       String fileId)

Generates the bucket ID using format {partition path}_{fileID}.

metaClientForReader

public static HoodieTableMetaClient metaClientForReader(org.apache.flink.configuration.Configuration conf,
                                                        org.apache.hadoop.conf.Configuration hadoopConf)

Creates the meta client for reader.

The streaming pipeline process is long-running, so empty table path is allowed, the reader would then check and refresh the meta client.

See Also:

StreamReadMonitoringFunction

createMetaClient

public static HoodieTableMetaClient createMetaClient(String basePath,
                                                     org.apache.hadoop.conf.Configuration hadoopConf)

Creates the meta client.

createMetaClient

public static HoodieTableMetaClient createMetaClient(org.apache.flink.configuration.Configuration conf)

Creates the meta client.

createWriteClient

public static HoodieFlinkWriteClient createWriteClient(org.apache.flink.configuration.Configuration conf,
                                                       org.apache.flink.api.common.functions.RuntimeContext runtimeContext)

Creates the Flink write client.

This expects to be used by client, the driver should start an embedded timeline server.

createWriteClient

public static HoodieFlinkWriteClient createWriteClient(org.apache.flink.configuration.Configuration conf,
                                                       org.apache.flink.api.common.functions.RuntimeContext runtimeContext,
                                                       boolean loadFsViewStorageConfig)

Creates the Flink write client.

This expects to be used by client, set flag loadFsViewStorageConfig to use remote filesystem view storage config, or an in-memory filesystem view storage is used.

createWriteClient

public static HoodieFlinkWriteClient createWriteClient(org.apache.flink.configuration.Configuration conf)
                                                throws IOException

Creates the Flink write client.

This expects to be used by the driver, the client can then send requests for files view.

The task context supplier is a constant: the write token is always '0-1-0'.

Throws:

IOException

medianInstantTime

public static Option<String> medianInstantTime(String highVal,
                                               String lowVal)

Returns the median instant time between the given two instant time.

instantTimeDiffSeconds

public static long instantTimeDiffSeconds(String newInstantTime,
                                          String oldInstantTime)

Returns the time interval in seconds between the given instant time.

createTransformer

public static Option<Transformer> createTransformer(List<String> classNames)
                                             throws IOException

Throws:

IOException

isValidFile

public static boolean isValidFile(org.apache.hadoop.fs.FileStatus fileStatus)

Returns whether the give file is in valid hoodie format. For example, filtering out the empty or corrupt files.

getLastPendingInstant

public static String getLastPendingInstant(HoodieTableMetaClient metaClient)

getLastPendingInstant

public static String getLastPendingInstant(HoodieTableMetaClient metaClient,
                                           boolean reloadTimeline)

getLastCompletedInstant

public static String getLastCompletedInstant(HoodieTableMetaClient metaClient)

haveSuccessfulCommits

public static boolean haveSuccessfulCommits(HoodieTableMetaClient metaClient)

Returns whether there are successful commits on the timeline.

Parameters:

metaClient - The meta client

Returns:

true if there is any successful commit

getMaxCompactionMemoryInBytes

public static long getMaxCompactionMemoryInBytes(org.apache.flink.configuration.Configuration conf)

Returns the max compaction memory in bytes with given conf.

getTableAvroSchema

public static org.apache.avro.Schema getTableAvroSchema(HoodieTableMetaClient metaClient,
                                                        boolean includeMetadataFields)
                                                 throws Exception

Throws:

Exception

getLatestTableSchema

public static org.apache.avro.Schema getLatestTableSchema(String path,
                                                          org.apache.hadoop.conf.Configuration hadoopConf)

fileExists

public static boolean fileExists(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path)

getAuxiliaryPath

public static String getAuxiliaryPath(org.apache.flink.configuration.Configuration conf)

Returns the auxiliary path.