org.apache.flink.streaming.connectors.pulsar.table

Class PulsarTableOptions

java.lang.Object

org.apache.flink.streaming.connectors.pulsar.table.PulsarTableOptions

public class PulsarTableOptions
extends Object

Option utils for pulsar table source sink.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PulsarTableOptions.StartupOptions pulsar startup options.
static class	PulsarTableOptions.ValueFieldsStrategy Strategies to derive the data type of a value format by considering a key format.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.flink.configuration.ConfigOption<String>	ADMIN_URL
static org.apache.flink.configuration.ConfigOption<List<String>>	KEY_FIELDS
static org.apache.flink.configuration.ConfigOption<String>	KEY_FIELDS_PREFIX
static org.apache.flink.configuration.ConfigOption<String>	KEY_FORMAT
static org.apache.flink.configuration.ConfigOption<Long>	PARTITION_DISCOVERY_INTERVAL_MILLIS
static org.apache.flink.configuration.ConfigOption<Map<String,String>>	PROPERTIES
static String	PROPERTIES_PREFIX
static org.apache.flink.configuration.ConfigOption<String>	SCAN_STARTUP_MODE
static String	SCAN_STARTUP_MODE_VALUE_EARLIEST
static String	SCAN_STARTUP_MODE_VALUE_EXTERNAL_SUBSCRIPTION
static String	SCAN_STARTUP_MODE_VALUE_LATEST
static String	SCAN_STARTUP_MODE_VALUE_SPECIFIC_OFFSETS
static org.apache.flink.configuration.ConfigOption<String>	SCAN_STARTUP_SPECIFIC_OFFSETS
static org.apache.flink.configuration.ConfigOption<String>	SCAN_STARTUP_SUB_NAME
static org.apache.flink.configuration.ConfigOption<String>	SCAN_STARTUP_SUB_START_OFFSET
static org.apache.flink.configuration.ConfigOption<Long>	SCAN_STARTUP_TIMESTAMP_MILLIS
static org.apache.flink.configuration.ConfigOption<String>	SERVICE_URL
static org.apache.flink.configuration.ConfigOption<String>	SINK_MESSAGE_ROUTER
static String	SINK_MESSAGE_ROUTER_VALUE_KEY_HASH
static String	SINK_MESSAGE_ROUTER_VALUE_ROUND_ROBIN
static org.apache.flink.configuration.ConfigOption<String>	SINK_SEMANTIC
static String	SINK_SEMANTIC_VALUE_AT_LEAST_ONCE
static String	SINK_SEMANTIC_VALUE_EXACTLY_ONCE
static String	SINK_SEMANTIC_VALUE_NONE
static org.apache.flink.configuration.ConfigOption<List<String>>	TOPIC
static org.apache.flink.configuration.ConfigOption<String>	TOPIC_PATTERN
static org.apache.flink.configuration.ConfigOption<PulsarTableOptions.ValueFieldsStrategy>	VALUE_FIELDS_INCLUDE
static org.apache.flink.configuration.ConfigOption<String>	VALUE_FORMAT

Method Summary

Modifier and Type	Method and Description
static int[]	createKeyFormatProjection(org.apache.flink.configuration.ReadableConfig options, org.apache.flink.table.types.DataType physicalDataType) Creates an array of indices that determine which physical fields of the table schema to include in the key format and the order that those fields have in the key format.
static int[]	createValueFormatProjection(org.apache.flink.configuration.ReadableConfig options, org.apache.flink.table.types.DataType physicalDataType) Creates an array of indices that determine which physical fields of the table schema to include in the value format.
static Optional<org.apache.pulsar.client.api.MessageRouter>	getMessageRouter(org.apache.flink.configuration.ReadableConfig tableOptions, ClassLoader classLoader) The messageRouter can be either "key-hash", "round-robin" or a customized messageRouter subClass full class name.
static Properties	getPulsarProperties(Map<String,String> tableOptions)
static PulsarSinkSemantic	getSinkSemantic(org.apache.flink.configuration.ReadableConfig tableOptions)
static PulsarTableOptions.StartupOptions	getStartupOptions(org.apache.flink.configuration.ReadableConfig tableOptions)
static Map<Integer,String>	parseSpecificOffsets(String specificOffsetsStr, String optionKey) Parses SpecificOffsets String to Map.
static void	validateSinkMessageRouter(org.apache.flink.configuration.ReadableConfig tableOptions)
static void	validateSinkTopic(org.apache.flink.configuration.ReadableConfig tableOptions)
static void	validateSourceTopic(org.apache.flink.configuration.ReadableConfig tableOptions)
static void	validateTableSinkOptions(org.apache.flink.configuration.ReadableConfig tableOptions)
static void	validateTableSourceOptions(org.apache.flink.configuration.ReadableConfig tableOptions)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEY_FORMAT

public static final org.apache.flink.configuration.ConfigOption<String> KEY_FORMAT

VALUE_FORMAT

public static final org.apache.flink.configuration.ConfigOption<String> VALUE_FORMAT

KEY_FIELDS

public static final org.apache.flink.configuration.ConfigOption<List<String>> KEY_FIELDS

VALUE_FIELDS_INCLUDE

public static final org.apache.flink.configuration.ConfigOption<PulsarTableOptions.ValueFieldsStrategy> VALUE_FIELDS_INCLUDE

KEY_FIELDS_PREFIX

public static final org.apache.flink.configuration.ConfigOption<String> KEY_FIELDS_PREFIX

SERVICE_URL

public static final org.apache.flink.configuration.ConfigOption<String> SERVICE_URL

ADMIN_URL

public static final org.apache.flink.configuration.ConfigOption<String> ADMIN_URL

TOPIC

public static final org.apache.flink.configuration.ConfigOption<List<String>> TOPIC

TOPIC_PATTERN

public static final org.apache.flink.configuration.ConfigOption<String> TOPIC_PATTERN

SCAN_STARTUP_MODE

public static final org.apache.flink.configuration.ConfigOption<String> SCAN_STARTUP_MODE

SCAN_STARTUP_SPECIFIC_OFFSETS

public static final org.apache.flink.configuration.ConfigOption<String> SCAN_STARTUP_SPECIFIC_OFFSETS

SCAN_STARTUP_SUB_NAME

public static final org.apache.flink.configuration.ConfigOption<String> SCAN_STARTUP_SUB_NAME

SCAN_STARTUP_SUB_START_OFFSET

public static final org.apache.flink.configuration.ConfigOption<String> SCAN_STARTUP_SUB_START_OFFSET

SCAN_STARTUP_TIMESTAMP_MILLIS

public static final org.apache.flink.configuration.ConfigOption<Long> SCAN_STARTUP_TIMESTAMP_MILLIS

PARTITION_DISCOVERY_INTERVAL_MILLIS

public static final org.apache.flink.configuration.ConfigOption<Long> PARTITION_DISCOVERY_INTERVAL_MILLIS

SINK_MESSAGE_ROUTER

public static final org.apache.flink.configuration.ConfigOption<String> SINK_MESSAGE_ROUTER

SINK_SEMANTIC

public static final org.apache.flink.configuration.ConfigOption<String> SINK_SEMANTIC

PROPERTIES

public static final org.apache.flink.configuration.ConfigOption<Map<String,String>> PROPERTIES

SCAN_STARTUP_MODE_VALUE_EARLIEST

public static final String SCAN_STARTUP_MODE_VALUE_EARLIEST

See Also:

Constant Field Values

SCAN_STARTUP_MODE_VALUE_LATEST

public static final String SCAN_STARTUP_MODE_VALUE_LATEST

See Also:

Constant Field Values

SCAN_STARTUP_MODE_VALUE_EXTERNAL_SUBSCRIPTION

public static final String SCAN_STARTUP_MODE_VALUE_EXTERNAL_SUBSCRIPTION

See Also:

Constant Field Values

SCAN_STARTUP_MODE_VALUE_SPECIFIC_OFFSETS

public static final String SCAN_STARTUP_MODE_VALUE_SPECIFIC_OFFSETS

See Also:

Constant Field Values

SINK_MESSAGE_ROUTER_VALUE_KEY_HASH

public static final String SINK_MESSAGE_ROUTER_VALUE_KEY_HASH

See Also:

Constant Field Values

SINK_MESSAGE_ROUTER_VALUE_ROUND_ROBIN

public static final String SINK_MESSAGE_ROUTER_VALUE_ROUND_ROBIN

See Also:

Constant Field Values

SINK_SEMANTIC_VALUE_EXACTLY_ONCE

public static final String SINK_SEMANTIC_VALUE_EXACTLY_ONCE

See Also:

Constant Field Values

SINK_SEMANTIC_VALUE_AT_LEAST_ONCE

public static final String SINK_SEMANTIC_VALUE_AT_LEAST_ONCE

See Also:

Constant Field Values

SINK_SEMANTIC_VALUE_NONE

public static final String SINK_SEMANTIC_VALUE_NONE

See Also:

Constant Field Values

PROPERTIES_PREFIX

public static final String PROPERTIES_PREFIX

See Also:

Constant Field Values

Method Detail

validateTableSourceOptions

public static void validateTableSourceOptions(org.apache.flink.configuration.ReadableConfig tableOptions)

validateSourceTopic

public static void validateSourceTopic(org.apache.flink.configuration.ReadableConfig tableOptions)

validateTableSinkOptions

public static void validateTableSinkOptions(org.apache.flink.configuration.ReadableConfig tableOptions)

validateSinkMessageRouter

public static void validateSinkMessageRouter(org.apache.flink.configuration.ReadableConfig tableOptions)

validateSinkTopic

public static void validateSinkTopic(org.apache.flink.configuration.ReadableConfig tableOptions)

getSinkSemantic

public static PulsarSinkSemantic getSinkSemantic(org.apache.flink.configuration.ReadableConfig tableOptions)

getPulsarProperties

public static Properties getPulsarProperties(Map<String,String> tableOptions)

getMessageRouter

public static Optional<org.apache.pulsar.client.api.MessageRouter> getMessageRouter(org.apache.flink.configuration.ReadableConfig tableOptions,
                                                                                    ClassLoader classLoader)

The messageRouter can be either "key-hash", "round-robin" or a customized messageRouter subClass full class name.

getStartupOptions

public static PulsarTableOptions.StartupOptions getStartupOptions(org.apache.flink.configuration.ReadableConfig tableOptions)

parseSpecificOffsets

public static Map<Integer,String> parseSpecificOffsets(String specificOffsetsStr,
                                                       String optionKey)

Parses SpecificOffsets String to Map.

SpecificOffsets String format was given as following:

     scan.startup.specific-offsets = 42:1012:0;44:1011:1
 

Returns:

SpecificOffsets with Map format, key is partition, and value is offset

createKeyFormatProjection

public static int[] createKeyFormatProjection(org.apache.flink.configuration.ReadableConfig options,
                                              org.apache.flink.table.types.DataType physicalDataType)

Creates an array of indices that determine which physical fields of the table schema to include in the key format and the order that those fields have in the key format.

See KEY_FORMAT, KEY_FIELDS, and KEY_FIELDS_PREFIX for more information.

createValueFormatProjection

public static int[] createValueFormatProjection(org.apache.flink.configuration.ReadableConfig options,
                                                org.apache.flink.table.types.DataType physicalDataType)

Creates an array of indices that determine which physical fields of the table schema to include in the value format.

See VALUE_FORMAT, VALUE_FIELDS_INCLUDE, and KEY_FIELDS_PREFIX for more information.