org.apache.flink.streaming.connectors.gcp.pubsub

Class PubSubSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<OUT>

org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class PubSubSource<OUT>
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<OUT>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

PubSub Source, this Source will consume PubSub messages from a subscription and Acknowledge them on the next checkpoint. This ensures every message will get acknowledged at least once.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PubSubSource.DeserializationSchemaBuilder Part of PubSubSource.PubSubSourceBuilder to set required fields.
static interface	PubSubSource.ProjectNameBuilder<OUT> Part of PubSubSource.PubSubSourceBuilder to set required fields.
static class	PubSubSource.PubSubSourceBuilder<OUT> Builder to create PubSubSource.
static interface	PubSubSource.SubscriptionNameBuilder<OUT> Part of PubSubSource.PubSubSourceBuilder to set required fields.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected AcknowledgeOnCheckpoint<String>	acknowledgeOnCheckpoint
protected org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory	acknowledgeOnCheckpointFactory
protected com.google.auth.Credentials	credentials
protected PubSubDeserializationSchema<OUT>	deserializationSchema
protected boolean	isRunning
protected int	messagePerSecondRateLimit
protected PubSubSubscriberFactory	pubSubSubscriberFactory
protected org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter	rateLimiter
protected PubSubSubscriber	subscriber

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	close()
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getProducedType()
static PubSubSource.DeserializationSchemaBuilder	newBuilder()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration configuration)
void	restoreState(List<AcknowledgeIdsForCheckpoint<String>> state)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> sourceContext)
List<AcknowledgeIdsForCheckpoint<String>>	snapshotState(long checkpointId, long timestamp)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

open

Field Detail

deserializationSchema

protected final PubSubDeserializationSchema<OUT> deserializationSchema

pubSubSubscriberFactory

protected final PubSubSubscriberFactory pubSubSubscriberFactory

credentials

protected final com.google.auth.Credentials credentials

acknowledgeOnCheckpointFactory

protected final org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory acknowledgeOnCheckpointFactory

rateLimiter

protected final org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter rateLimiter

messagePerSecondRateLimit

protected final int messagePerSecondRateLimit

acknowledgeOnCheckpoint

protected transient AcknowledgeOnCheckpoint<String> acknowledgeOnCheckpoint

subscriber

protected transient PubSubSubscriber subscriber

isRunning

protected transient volatile boolean isRunning

Method Detail

open

public void open(org.apache.flink.configuration.Configuration configuration)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> sourceContext)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>

newBuilder

public static PubSubSource.DeserializationSchemaBuilder newBuilder()

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

Specified by:

notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

snapshotState

public List<AcknowledgeIdsForCheckpoint<String>> snapshotState(long checkpointId,
                                                               long timestamp)
                                                        throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

Throws:

Exception

restoreState

public void restoreState(List<AcknowledgeIdsForCheckpoint<String>> state)
                  throws Exception

Specified by:

restoreState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

Throws:

Exception