org.apache.beam.io.debezium

Class KafkaSourceConsumerFn<T>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<java.util.Map<java.lang.String,java.lang.String>,T>

org.apache.beam.io.debezium.KafkaSourceConsumerFn<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class KafkaSourceConsumerFn<T>
extends org.apache.beam.sdk.transforms.DoFn<java.util.Map<java.lang.String,java.lang.String>,T>

Quick Overview

SDF used to process records fetched from supported Debezium Connectors.

Currently it has a time limiter (see OffsetTracker) which, if set, it will stop automatically after the specified elapsed minutes. Otherwise, it will keep running until the user explicitly interrupts it.

It might be initialized either as:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper)

Or with a time limiter:

KafkaSourceConsumerFn(connectorClass, SourceRecordMapper, minutesToRun)

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaSourceConsumerFn.DebeziumSDFDatabaseHistory

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BEAM_INSTANCE_PROPERTY

Constructor Summary

Constructors

Constructor and Description
KafkaSourceConsumerFn(java.lang.Class<?> connectorClass, SourceRecordMapper<T> fn, java.lang.Integer maxRecords) Initializes the SDF to be run indefinitely.
KafkaSourceConsumerFn(java.lang.Class<?> connectorClass, SourceRecordMapper<T> fn, long minutesToRun) Initializes the SDF with a time limit.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getHashCode()
org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder	getInitialRestriction(java.util.Map<java.lang.String,java.lang.String> unused)
org.apache.beam.sdk.coders.Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder>	getRestrictionCoder()
org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>>	newTracker(org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)
org.apache.beam.sdk.transforms.DoFn.ProcessContinuation	process(java.util.Map<java.lang.String,java.lang.String> element, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T> receiver) Process the retrieved element.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BEAM_INSTANCE_PROPERTY

public static final java.lang.String BEAM_INSTANCE_PROPERTY

See Also:

Constant Field Values

Constructor Detail

KafkaSourceConsumerFn

public KafkaSourceConsumerFn(java.lang.Class<?> connectorClass,
                             SourceRecordMapper<T> fn,
                             long minutesToRun)

Initializes the SDF with a time limit.

Parameters:

connectorClass - Supported Debezium connector class

fn - a SourceRecordMapper

minutesToRun - Maximum time to run (in minutes)

KafkaSourceConsumerFn

public KafkaSourceConsumerFn(java.lang.Class<?> connectorClass,
                             SourceRecordMapper<T> fn,
                             java.lang.Integer maxRecords)

Initializes the SDF to be run indefinitely.

Parameters:

connectorClass - Supported Debezium connector class

fn - a SourceRecordMapper

Method Detail

getInitialRestriction

@DoFn.GetInitialRestriction
public org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder getInitialRestriction(@DoFn.Element
                                                                                                                        java.util.Map<java.lang.String,java.lang.String> unused)
                                                                                                                 throws java.io.IOException

Throws:

java.io.IOException

newTracker

@DoFn.NewTracker
public org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> newTracker(@DoFn.Restriction
                                                                                                                                                                                                                     org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder restriction)

getRestrictionCoder

@DoFn.GetRestrictionCoder
public org.apache.beam.sdk.coders.Coder<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder> getRestrictionCoder()

process

@DoFn.ProcessElement
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation process(@DoFn.Element
                                                                                            java.util.Map<java.lang.String,java.lang.String> element,
                                                                                            org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.io.debezium.KafkaSourceConsumerFn.OffsetHolder,java.util.Map<java.lang.String,java.lang.Object>> tracker,
                                                                                            org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T> receiver)
                                                                                     throws java.lang.Exception

Process the retrieved element. Currently it just logs the retrieved record as JSON.

Parameters:

element - Record retrieved

tracker - Restriction Tracker

receiver - Output Receiver

Returns:

Throws:

java.lang.Exception

getHashCode

public java.lang.String getHashCode()