org.apache.beam.sdk.io

Class BoundedSource<T>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<T>

Type Parameters:

T - Type of records read by the source.

All Implemented Interfaces:

Serializable, HasDisplayData

Direct Known Subclasses:

OffsetBasedSource

public abstract class BoundedSource<T>
extends Source<T>

A Source that reads a finite amount of input and, because of that, supports some additional operations.

The operations are:

• Splitting into bundles of given size: splitIntoBundles(long, org.apache.beam.sdk.options.PipelineOptions);
• Size estimation: getEstimatedSizeBytes(org.apache.beam.sdk.options.PipelineOptions);
• Telling whether or not this source produces key/value pairs in sorted order: producesSortedKeys(org.apache.beam.sdk.options.PipelineOptions);
• The accompanying reader has additional functionality to enable runners to dynamically adapt based on runtime conditions.
  • Progress estimation (BoundedSource.BoundedReader.getFractionConsumed())
  • Tracking of parallelism, to determine whether the current source can be split (BoundedSource.BoundedReader.getSplitPointsConsumed() and BoundedSource.BoundedReader.getSplitPointsRemaining()).
  • Dynamic splitting of the current source (BoundedSource.BoundedReader.splitAtFraction(double)).

To use this class for supporting your custom input type, derive your class class from it, and override the abstract methods. For an example, see DatastoreIO.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BoundedSource.BoundedReader<T> A Reader that reads a bounded amount of input and supports some additional operations, such as progress estimation and dynamic work rebalancing.

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
BoundedSource()

Method Summary

Modifier and Type	Method and Description
abstract BoundedSource.BoundedReader<T>	createReader(PipelineOptions options) Returns a new BoundedSource.BoundedReader that reads from this source.
abstract long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
abstract boolean	producesSortedKeys(PipelineOptions options) Whether this source is known to produce key/value pairs sorted by lexicographic order on the bytes of the encoded key.
abstract List<? extends BoundedSource<T>>	splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately desiredBundleSizeBytes.

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BoundedSource

public BoundedSource()

Method Detail

splitIntoBundles

public abstract List<? extends BoundedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                                  PipelineOptions options)
                                                           throws Exception

Splits the source into bundles of approximately desiredBundleSizeBytes.

Throws:

Exception

getEstimatedSizeBytes

public abstract long getEstimatedSizeBytes(PipelineOptions options)
                                    throws Exception

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Throws:

Exception

producesSortedKeys

public abstract boolean producesSortedKeys(PipelineOptions options)
                                    throws Exception

Whether this source is known to produce key/value pairs sorted by lexicographic order on the bytes of the encoded key.

Throws:

Exception

createReader

public abstract BoundedSource.BoundedReader<T> createReader(PipelineOptions options)
                                                     throws IOException

Returns a new BoundedSource.BoundedReader that reads from this source.

Throws:

IOException