Class SparkReceiverIO
- java.lang.Object
-
- org.apache.beam.sdk.io.sparkreceiver.SparkReceiverIO
-
public class SparkReceiverIO extends java.lang.ObjectStreaming sources for SparkReceiver.Reading using
SparkReceiverIOYou will need to pass a
ReceiverBuilderwhich is responsible for instantiating newReceiverobjects.Receiverthat will be used should implementHasOffsetinterface. You will need to passgetOffsetFnwhich is aSerializableFunctionthat defines how to getLong offsetfromV record.Optionally you can pass
timestampFnwhich is aSerializableFunctionthat defines how to getInstant timestampfromV record, you can passstartOffsetwhich is inclusive start offset from which the reading should be started.Optionally you can pass
pullFrequencySecwhich is a delay in seconds between polling for new records updates. Also, you can passstartPollTimeoutSecwhich is delay in seconds before start polling.Example of
read()usage:Pipeline p = ...; // Create pipeline. // Create ReceiverBuilder for CustomReceiver ReceiverBuilder<String, CustomReceiverWithOffset> receiverBuilder = new ReceiverBuilder<>(CustomReceiver.class).withConstructorArgs(); //Read from CustomReceiver p.apply("Spark Receiver Read", SparkReceiverIO.Read<String> reader = SparkReceiverIO.<String>read() .withGetOffsetFn(Long::valueOf) .withTimestampFn(Instant::parse) .withPullFrequencySec(1L) .withStartPollTimeoutSec(2L) .withStartOffset(10L) .withSparkReceiverBuilder(receiverBuilder);
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classSparkReceiverIO.Read<V>APTransformto read from SparkReceiver.
-
Constructor Summary
Constructors Constructor Description SparkReceiverIO()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static <V> SparkReceiverIO.Read<V>read()
-
-
-
Method Detail
-
read
public static <V> SparkReceiverIO.Read<V> read()
-
-