org.apache.flink.streaming.connectors.pulsar

Class PulsarTableSource

java.lang.Object

org.apache.flink.streaming.connectors.pulsar.PulsarTableSource

All Implemented Interfaces:

SupportsReadingMetadata, org.apache.flink.table.sources.DefinedFieldMapping, org.apache.flink.table.sources.DefinedProctimeAttribute, org.apache.flink.table.sources.DefinedRowtimeAttributes, org.apache.flink.table.sources.StreamTableSource<org.apache.flink.types.Row>, org.apache.flink.table.sources.TableSource<org.apache.flink.types.Row>

public class PulsarTableSource
extends Object
implements org.apache.flink.table.sources.StreamTableSource<org.apache.flink.types.Row>, org.apache.flink.table.sources.DefinedProctimeAttribute, org.apache.flink.table.sources.DefinedRowtimeAttributes, org.apache.flink.table.sources.DefinedFieldMapping, SupportsReadingMetadata

Pulsar .

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PulsarTableSource.ReadableMetadata readable metadata in source.

Field Summary

Fields

Modifier and Type	Field and Description
protected List<String>	metadataKeys Metadata that is appended at the end of a physical source row.
protected org.apache.flink.table.types.DataType	physicalDataType Data type to configure the format.
protected org.apache.flink.table.types.DataType	producedDataType Data type that describes the final output of the source.
protected org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row>	producedTypeInfo
protected boolean	useExtendField

Constructor Summary

Constructors

Constructor and Description
PulsarTableSource(Optional<org.apache.flink.table.api.TableSchema> providedSchema, Optional<String> proctimeAttribute, List<org.apache.flink.table.sources.RowtimeAttributeDescriptor> rowtimeAttributeDescriptors, Optional<Map<String,String>> fieldMapping, String serviceUrl, String adminUrl, Properties properties, org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.types.Row> deserializationSchema, StartupMode startupMode, Map<String,org.apache.pulsar.client.api.MessageId> specificStartupOffsets, String externalSubscriptionName)
PulsarTableSource(String serviceUrl, String adminUrl, Properties properties)

Method Summary

Modifier and Type	Method and Description
void	applyReadableMetadata(List<String> metadataKeys, org.apache.flink.table.types.DataType producedDataType) Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.
org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.types.Row>	getDataStream(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment execEnv)
PulsarDeserializationSchema<org.apache.flink.types.Row>	getDeserializationSchema()
Map<String,String>	getFieldMapping()
String	getProctimeAttribute()
org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row>	getReturnType()
List<org.apache.flink.table.sources.RowtimeAttributeDescriptor>	getRowtimeAttributeDescriptors()
org.apache.flink.table.api.TableSchema	getTableSchema()
Map<String,org.apache.flink.table.types.DataType>	listReadableMetadata() Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.sources.StreamTableSource

isBounded

Methods inherited from interface org.apache.flink.table.sources.TableSource

explainSource, getProducedDataType

Field Detail

metadataKeys

protected List<String> metadataKeys

Metadata that is appended at the end of a physical source row.

useExtendField

protected final boolean useExtendField

producedDataType

protected org.apache.flink.table.types.DataType producedDataType

Data type that describes the final output of the source.

physicalDataType

protected final org.apache.flink.table.types.DataType physicalDataType

Data type to configure the format.

producedTypeInfo

protected org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> producedTypeInfo

Constructor Detail

PulsarTableSource

public PulsarTableSource(Optional<org.apache.flink.table.api.TableSchema> providedSchema,
                         Optional<String> proctimeAttribute,
                         List<org.apache.flink.table.sources.RowtimeAttributeDescriptor> rowtimeAttributeDescriptors,
                         Optional<Map<String,String>> fieldMapping,
                         String serviceUrl,
                         String adminUrl,
                         Properties properties,
                         org.apache.flink.api.common.serialization.DeserializationSchema<org.apache.flink.types.Row> deserializationSchema,
                         StartupMode startupMode,
                         Map<String,org.apache.pulsar.client.api.MessageId> specificStartupOffsets,
                         String externalSubscriptionName)

PulsarTableSource

public PulsarTableSource(String serviceUrl,
                         String adminUrl,
                         Properties properties)

Method Detail

getProctimeAttribute

public String getProctimeAttribute()

Specified by:

getProctimeAttribute in interface org.apache.flink.table.sources.DefinedProctimeAttribute

getRowtimeAttributeDescriptors

public List<org.apache.flink.table.sources.RowtimeAttributeDescriptor> getRowtimeAttributeDescriptors()

Specified by:

getRowtimeAttributeDescriptors in interface org.apache.flink.table.sources.DefinedRowtimeAttributes

getReturnType

public org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.types.Row> getReturnType()

Specified by:

getReturnType in interface org.apache.flink.table.sources.TableSource<org.apache.flink.types.Row>

getTableSchema

public org.apache.flink.table.api.TableSchema getTableSchema()

Specified by:

getTableSchema in interface org.apache.flink.table.sources.TableSource<org.apache.flink.types.Row>

getFieldMapping

public Map<String,String> getFieldMapping()

Specified by:

getFieldMapping in interface org.apache.flink.table.sources.DefinedFieldMapping

getDataStream

public org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.types.Row> getDataStream(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment execEnv)

Specified by:

getDataStream in interface org.apache.flink.table.sources.StreamTableSource<org.apache.flink.types.Row>

getDeserializationSchema

public PulsarDeserializationSchema<org.apache.flink.types.Row> getDeserializationSchema()

listReadableMetadata

public Map<String,org.apache.flink.table.types.DataType> listReadableMetadata()

Description copied from interface: SupportsReadingMetadata

Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

The returned map will be used by the planner for validation and insertion of explicit casts (see LogicalTypeCasts.supportsExplicitCast(LogicalType, LogicalType)) if necessary.

The iteration order of the returned map determines the order of metadata keys in the list passed in SupportsReadingMetadata.applyReadableMetadata(List, DataType). Therefore, it might be beneficial to return a LinkedHashMap if a strict metadata column order is required.

If a source forwards metadata from one or more formats, we recommend the following column order for consistency:

   KEY FORMAT METADATA COLUMNS + VALUE FORMAT METADATA COLUMNS + SOURCE METADATA COLUMNS
 

Metadata key names follow the same pattern as mentioned in Factory. In case of duplicate names in format and source keys, format keys shall have higher precedence.

Regardless of the returned DataTypes, a metadata column is always represented using internal data structures (see RowData).

Specified by:

listReadableMetadata in interface SupportsReadingMetadata

applyReadableMetadata

public void applyReadableMetadata(List<String> metadataKeys,
                                  org.apache.flink.table.types.DataType producedDataType)

Description copied from interface: SupportsReadingMetadata

Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.

Note: Use the passed data type instead of TableSchema.toPhysicalRowDataType() for describing the final output data type when creating TypeInformation. If the source implements SupportsProjectionPushDown, the projection is already considered in the given output data type.

Specified by:

applyReadableMetadata in interface SupportsReadingMetadata

Parameters:

metadataKeys - a subset of the keys returned by SupportsReadingMetadata.listReadableMetadata(), ordered by the iteration order of returned map

producedDataType - the final output type of the source