org.apache.flink.connector.pulsar.table

Class PulsarDynamicTableSource

java.lang.Object

org.apache.flink.connector.pulsar.table.PulsarDynamicTableSource

All Implemented Interfaces:

SupportsReadingMetadata, org.apache.flink.table.connector.source.DynamicTableSource, org.apache.flink.table.connector.source.ScanTableSource

public class PulsarDynamicTableSource
extends Object
implements org.apache.flink.table.connector.source.ScanTableSource, SupportsReadingMetadata

pulsar dynamic table source.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PulsarDynamicTableSource.ReadableMetadata readableMetadata for new table api.

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.ScanTableSource

org.apache.flink.table.connector.source.ScanTableSource.ScanContext, org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

org.apache.flink.table.connector.source.DynamicTableSource.Context, org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter

Field Summary

Fields

Modifier and Type	Field and Description
protected String	adminUrl The Pulsar topic to consume.
protected org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>>	decodingFormat Scan format for decoding records from Pulsar.
protected List<String>	metadataKeys Metadata that is appended at the end of a physical source row.
protected org.apache.flink.table.types.DataType	physicalDataType Data type to configure the format.
protected org.apache.flink.table.types.DataType	producedDataType Data type that describes the final output of the source.
protected Properties	properties Properties for the Pulsar consumer.
protected String	serviceUrl The Pulsar topic to consume.
protected PulsarTableOptions.StartupOptions	startupOptions The startup mode for the contained consumer (default is StartupMode.LATEST).
protected String	topicPattern The Pulsar topic to consume.
protected List<String>	topics The Pulsar topic to consume.
protected boolean	useExtendField

Constructor Summary

Constructors

Constructor and Description

PulsarDynamicTableSource(org.apache.flink.table.types.DataType physicalDataType, org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> decodingFormat, List<String> topics, String topicPattern, String serviceUrl, String adminUrl, Properties properties, PulsarTableOptions.StartupOptions startupOptions)

Method Summary

Modifier and Type	Method and Description
void	applyReadableMetadata(List<String> metadataKeys, org.apache.flink.table.types.DataType producedDataType) Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.
String	asSummaryString()
org.apache.flink.table.connector.source.DynamicTableSource	copy()
boolean	equals(Object o)
org.apache.flink.table.connector.ChangelogMode	getChangelogMode()
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider	getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext runtimeProviderContext)
int	hashCode()
Map<String,org.apache.flink.table.types.DataType>	listReadableMetadata() Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

producedDataType

protected org.apache.flink.table.types.DataType producedDataType

Data type that describes the final output of the source.

physicalDataType

protected final org.apache.flink.table.types.DataType physicalDataType

Data type to configure the format.

metadataKeys

protected List<String> metadataKeys

Metadata that is appended at the end of a physical source row.

decodingFormat

protected final org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> decodingFormat

Scan format for decoding records from Pulsar.

topics

protected final List<String> topics

The Pulsar topic to consume.

topicPattern

protected final String topicPattern

The Pulsar topic to consume.

serviceUrl

protected final String serviceUrl

The Pulsar topic to consume.

useExtendField

protected final boolean useExtendField

adminUrl

protected final String adminUrl

The Pulsar topic to consume.

properties

protected final Properties properties

Properties for the Pulsar consumer.

startupOptions

protected final PulsarTableOptions.StartupOptions startupOptions

The startup mode for the contained consumer (default is StartupMode.LATEST).

Constructor Detail

PulsarDynamicTableSource

public PulsarDynamicTableSource(org.apache.flink.table.types.DataType physicalDataType,
                                org.apache.flink.table.connector.format.DecodingFormat<org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.table.data.RowData>> decodingFormat,
                                List<String> topics,
                                String topicPattern,
                                String serviceUrl,
                                String adminUrl,
                                Properties properties,
                                PulsarTableOptions.StartupOptions startupOptions)

Method Detail

getChangelogMode

public org.apache.flink.table.connector.ChangelogMode getChangelogMode()

Specified by:

getChangelogMode in interface org.apache.flink.table.connector.source.ScanTableSource

getScanRuntimeProvider

public org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext runtimeProviderContext)

Specified by:

getScanRuntimeProvider in interface org.apache.flink.table.connector.source.ScanTableSource

copy

public org.apache.flink.table.connector.source.DynamicTableSource copy()

Specified by:

copy in interface org.apache.flink.table.connector.source.DynamicTableSource

asSummaryString

public String asSummaryString()

Specified by:

asSummaryString in interface org.apache.flink.table.connector.source.DynamicTableSource

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

listReadableMetadata

public Map<String,org.apache.flink.table.types.DataType> listReadableMetadata()

Description copied from interface: SupportsReadingMetadata

Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

The returned map will be used by the planner for validation and insertion of explicit casts (see LogicalTypeCasts.supportsExplicitCast(LogicalType, LogicalType)) if necessary.

The iteration order of the returned map determines the order of metadata keys in the list passed in SupportsReadingMetadata.applyReadableMetadata(List, DataType). Therefore, it might be beneficial to return a LinkedHashMap if a strict metadata column order is required.

If a source forwards metadata from one or more formats, we recommend the following column order for consistency:

   KEY FORMAT METADATA COLUMNS + VALUE FORMAT METADATA COLUMNS + SOURCE METADATA COLUMNS
 

Metadata key names follow the same pattern as mentioned in Factory. In case of duplicate names in format and source keys, format keys shall have higher precedence.

Regardless of the returned DataTypes, a metadata column is always represented using internal data structures (see RowData).

Specified by:

listReadableMetadata in interface SupportsReadingMetadata

applyReadableMetadata

public void applyReadableMetadata(List<String> metadataKeys,
                                  org.apache.flink.table.types.DataType producedDataType)

Description copied from interface: SupportsReadingMetadata

Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.

Note: Use the passed data type instead of TableSchema.toPhysicalRowDataType() for describing the final output data type when creating TypeInformation. If the source implements SupportsProjectionPushDown, the projection is already considered in the given output data type.

Specified by:

applyReadableMetadata in interface SupportsReadingMetadata

Parameters:

metadataKeys - a subset of the keys returned by SupportsReadingMetadata.listReadableMetadata(), ordered by the iteration order of returned map

producedDataType - the final output type of the source