org.apache.beam.sdk.io

Class CountingSource

java.lang.Object

org.apache.beam.sdk.io.CountingSource

public class CountingSource
extends Object

A source that produces longs. When used as a BoundedSource, CountingSource starts at 0 and counts up to a specified maximum. When used as an UnboundedSource, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

The bounded CountingSource is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

To produce a bounded PCollection<Long>, use upTo(long):

 Pipeline p = ...
 PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
 PCollection<Long> bounded = p.apply(producer);
 

To produce an unbounded PCollection<Long>, use CountingInput.unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

 Pipeline p = ...

 // To create an unbounded PCollection that uses processing time as the element timestamp.
 PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
 // Or, to create an unbounded source that uses a provided function to set the element timestamp.
 PCollection<Long> unboundedWithTimestamps =
     p.apply(CountingInput.unbounded().withTimestampFn(someFn));

 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountingSource.CounterMark The checkpoint for an unbounded CountingSource is simply the last value produced.

Method Summary

Modifier and Type	Method and Description
static UnboundedSource<Long,CountingSource.CounterMark>	unbounded() Deprecated. use CountingInput.unbounded() instead
static UnboundedSource<Long,CountingSource.CounterMark>	unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn) Deprecated. use CountingInput.unbounded() and call CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) instead
static BoundedSource<Long>	upTo(long numElements) Deprecated. use CountingInput.upTo(long) instead

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

upTo

@Deprecated
public static BoundedSource<Long> upTo(long numElements)

Deprecated. use CountingInput.upTo(long) instead

Creates a BoundedSource that will produce the specified number of elements, from 0 to numElements - 1.

unbounded

@Deprecated
public static UnboundedSource<Long,CountingSource.CounterMark> unbounded()

Deprecated. use CountingInput.unbounded() instead

Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE.

After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

Elements in the resulting PCollection<Long> will have timestamps corresponding to processing time at element generation, provided by Instant.now().

unboundedWithTimestampFn

@Deprecated
public static UnboundedSource<Long,CountingSource.CounterMark> unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn)

Deprecated. use CountingInput.unbounded() and call CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) instead

Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE, with element timestamps supplied by the specified function.

After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

Note that the timestamps produced by timestampFn may not decrease.