Package org.apache.beam.sdk.io.sparkreceiver

Class SparkReceiverIO

  • @Experimental(SOURCE_SINK)
public class SparkReceiverIO
extends java.lang.Object
    Streaming sources for Spark Receiver.

    Reading using SparkReceiverIO

    You will need to pass a ReceiverBuilder which is responsible for instantiating new Receiver objects.

    Receiver that will be used should implement HasOffset interface. You will need to pass getOffsetFn which is a SerializableFunction that defines how to get Long offset from V record.

    Optionally you can pass timestampFn which is a SerializableFunction that defines how to get Instant timestamp from V record.

    Example of read() usage: 

    
 Pipeline p = ...; // Create pipeline.

 // Create ReceiverBuilder for CustomReceiver
 ReceiverBuilder<String, CustomReceiverWithOffset> receiverBuilder =
         new ReceiverBuilder<>(CustomReceiver.class).withConstructorArgs();

 //Read from CustomReceiver
 p.apply("Spark Receiver Read",
  SparkReceiverIO.Read<String> reader =
    SparkReceiverIO.<String>read()
      .withGetOffsetFn(Long::valueOf)
      .withTimestampFn(Instant::parse)
      .withSparkReceiverBuilder(receiverBuilder);

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  SparkReceiverIO.Read<V>
      A PTransform to read from Spark Receiver.

    • Constructor Summary

      Constructors 
      Constructor Description
      SparkReceiverIO()  

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static <V> SparkReceiverIO.Read<V> read()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • SparkReceiverIO

        public SparkReceiverIO()