Package org.apache.beam.sdk.io.pulsar

Class ReadFromPulsarDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PulsarSourceDescriptor,PulsarMessage>

org.apache.beam.sdk.io.pulsar.ReadFromPulsarDoFn

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

@UnboundedPerElement
public class ReadFromPulsarDoFn
extends org.apache.beam.sdk.transforms.DoFn<PulsarSourceDescriptor,PulsarMessage>

Transform for reading from Apache Pulsar. Support is currently incomplete, and there may be bugs; see https://github.com/apache/beam/issues/31078 for more info, and comment in that issue if you run into issues with this IO.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T extends java.lang.Object>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor	Description
ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Summary

Modifier and Type	Method	Description
org.apache.beam.sdk.io.range.OffsetRange	getInitialRestriction(PulsarSourceDescriptor pulsarSource)
org.joda.time.Instant	getInitialWatermarkEstimatorState(org.joda.time.Instant currentElementTimestamp)
org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.io.range.OffsetRange>	getRestrictionCoder()
double	getSize(PulsarSourceDescriptor pulsarSource, org.apache.beam.sdk.io.range.OffsetRange range)
void	initPulsarClients()
org.apache.pulsar.client.api.Reader<byte[]>	newReader(org.apache.pulsar.client.api.PulsarClient client, java.lang.String topicPartition)
org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimator<org.joda.time.Instant>	newWatermarkEstimator(org.joda.time.Instant watermarkEstimatorState)
org.apache.beam.sdk.transforms.DoFn.ProcessContinuation	processElement(PulsarSourceDescriptor pulsarSourceDescriptor, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker, org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimator watermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PulsarMessage> output)
org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker	restrictionTracker(PulsarSourceDescriptor pulsarSource, org.apache.beam.sdk.io.range.OffsetRange restriction)
void	teardown()

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReadFromPulsarDoFn

public ReadFromPulsarDoFn(PulsarIO.Read transform)

Method Detail

initPulsarClients

@Setup
public void initPulsarClients()
                       throws java.lang.Exception

Throws:

java.lang.Exception

teardown

@Teardown
public void teardown()
              throws java.lang.Exception

Throws:

java.lang.Exception

getInitialRestriction

@GetInitialRestriction
public org.apache.beam.sdk.io.range.OffsetRange getInitialRestriction(@Element
                                                                      PulsarSourceDescriptor pulsarSource)

getSize

@GetSize
public double getSize(@Element
                      PulsarSourceDescriptor pulsarSource,
                      @Restriction
                      org.apache.beam.sdk.io.range.OffsetRange range)

newReader

public org.apache.pulsar.client.api.Reader<byte[]> newReader(org.apache.pulsar.client.api.PulsarClient client,
                                                             java.lang.String topicPartition)
                                                      throws org.apache.pulsar.client.api.PulsarClientException

Throws:

org.apache.pulsar.client.api.PulsarClientException

getRestrictionCoder

@GetRestrictionCoder
public org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.io.range.OffsetRange> getRestrictionCoder()

processElement

@ProcessElement
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation processElement(@Element
                                                                              PulsarSourceDescriptor pulsarSourceDescriptor,
                                                                              org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker,
                                                                              org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimator watermarkEstimator,
                                                                              org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PulsarMessage> output)
                                                                       throws java.io.IOException

Throws:

java.io.IOException

getInitialWatermarkEstimatorState

@GetInitialWatermarkEstimatorState
public org.joda.time.Instant getInitialWatermarkEstimatorState(@Timestamp
                                                               org.joda.time.Instant currentElementTimestamp)

newWatermarkEstimator

@NewWatermarkEstimator
public org.apache.beam.sdk.transforms.splittabledofn.WatermarkEstimator<org.joda.time.Instant> newWatermarkEstimator(@WatermarkEstimatorState
                                                                                                                     org.joda.time.Instant watermarkEstimatorState)

restrictionTracker

@NewTracker
public org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker restrictionTracker(@Element
                                                                                           PulsarSourceDescriptor pulsarSource,
                                                                                           @Restriction
                                                                                           org.apache.beam.sdk.io.range.OffsetRange restriction)