org.apache.beam.sdk.io.synthetic

Class SyntheticBoundedSource

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.sdk.io.BoundedSource<T>

org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

@Experimental(value=SOURCE_SINK)
public class SyntheticBoundedSource
extends org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

A SyntheticBoundedSource that reads KV<byte[], byte[]>.

The SyntheticBoundedSource generates a PCollection of KV<byte[], byte[]>. A fraction of the generated records KV<byte[], byte[]> are associated with "hot" keys, which are uniformly distributed over a fixed number of hot keys. The remaining generated records are associated with "random" keys. Each record will be slowed down by a certain sleep time generated based on the specified sleep time distribution when the SyntheticBoundedSource.SyntheticSourceReader reads each record. The record KV<byte[], byte[]> is generated deterministically based on the record's position in the source, which enables repeatable execution for debugging. The SyntheticBoundedInput configurable parameters are defined in SyntheticSourceOptions.*

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.OffsetBasedSource

org.apache.beam.sdk.io.OffsetBasedSource.OffsetBasedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

org.apache.beam.sdk.io.BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

org.apache.beam.sdk.io.Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
SyntheticBoundedSource(long startOffset, long endOffset, SyntheticSourceOptions sourceOptions)
SyntheticBoundedSource(SyntheticSourceOptions sourceOptions)

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource.SyntheticSourceReader	createReader(org.apache.beam.sdk.options.PipelineOptions pipelineOptions)
SyntheticBoundedSource	createSourceForSubrange(long start, long end)
long	getBytesPerOffset()
org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<byte[],byte[]>>	getDefaultOutputCoder()
long	getMaxEndOffset(org.apache.beam.sdk.options.PipelineOptions options)
java.util.List<SyntheticBoundedSource>	split(long desiredBundleSizeBytes, org.apache.beam.sdk.options.PipelineOptions options)
java.lang.String	toString()
void	validate()

Methods inherited from class org.apache.beam.sdk.io.OffsetBasedSource

getEndOffset, getEstimatedSizeBytes, getMinBundleSize, getStartOffset, populateDisplayData

Methods inherited from class org.apache.beam.sdk.io.Source

getOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SyntheticBoundedSource

public SyntheticBoundedSource(SyntheticSourceOptions sourceOptions)

SyntheticBoundedSource

public SyntheticBoundedSource(long startOffset,
                              long endOffset,
                              SyntheticSourceOptions sourceOptions)

Method Detail

getDefaultOutputCoder

public org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<byte[],byte[]>> getDefaultOutputCoder()

Overrides:

getDefaultOutputCoder in class org.apache.beam.sdk.io.Source<org.apache.beam.sdk.values.KV<byte[],byte[]>>

getBytesPerOffset

public long getBytesPerOffset()

Overrides:

getBytesPerOffset in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

validate

public void validate()

Overrides:

validate in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

toString

public java.lang.String toString()

Overrides:

toString in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

createSourceForSubrange

public final SyntheticBoundedSource createSourceForSubrange(long start,
                                                            long end)

Specified by:

createSourceForSubrange in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

getMaxEndOffset

public long getMaxEndOffset(org.apache.beam.sdk.options.PipelineOptions options)

Specified by:

getMaxEndOffset in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

createReader

public org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource.SyntheticSourceReader createReader(org.apache.beam.sdk.options.PipelineOptions pipelineOptions)

Specified by:

createReader in class org.apache.beam.sdk.io.BoundedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

split

public java.util.List<SyntheticBoundedSource> split(long desiredBundleSizeBytes,
                                                    org.apache.beam.sdk.options.PipelineOptions options)
                                             throws java.lang.Exception

Overrides:

split in class org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>>

Throws:

java.lang.Exception