Class SparkReceiverIO


  • public class SparkReceiverIO
    extends java.lang.Object
    Streaming sources for Spark Receiver.

    Reading using SparkReceiverIO

    You will need to pass a ReceiverBuilder which is responsible for instantiating new Receiver objects.

    Receiver that will be used should implement HasOffset interface. You will need to pass getOffsetFn which is a SerializableFunction that defines how to get Long offset from V record.

    Optionally you can pass timestampFn which is a SerializableFunction that defines how to get Instant timestamp from V record, you can pass startOffset which is inclusive start offset from which the reading should be started.

    Optionally you can pass pullFrequencySec which is a delay in seconds between polling for new records updates. Also, you can pass startPollTimeoutSec which is delay in seconds before start polling.

    Example of read() usage:

    
     Pipeline p = ...; // Create pipeline.
    
     // Create ReceiverBuilder for CustomReceiver
     ReceiverBuilder<String, CustomReceiverWithOffset> receiverBuilder =
             new ReceiverBuilder<>(CustomReceiver.class).withConstructorArgs();
    
     //Read from CustomReceiver
     p.apply("Spark Receiver Read",
      SparkReceiverIO.Read<String> reader =
        SparkReceiverIO.<String>read()
          .withGetOffsetFn(Long::valueOf)
          .withTimestampFn(Instant::parse)
          .withPullFrequencySec(1L)
          .withStartPollTimeoutSec(2L)
          .withStartOffset(10L)
          .withSparkReceiverBuilder(receiverBuilder);
     
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  SparkReceiverIO.Read<V>
      A PTransform to read from Spark Receiver.
    • Constructor Summary

      Constructors 
      Constructor Description
      SparkReceiverIO()  
    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static <V> SparkReceiverIO.Read<V> read()  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • SparkReceiverIO

        public SparkReceiverIO()