org.apache.beam.sdk.io.synthetic

Class SyntheticBoundedIO.SyntheticSourceOptions

java.lang.Object

org.apache.beam.sdk.io.synthetic.SyntheticOptions

org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO.SyntheticSourceOptions

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

SyntheticBoundedIO

public static class SyntheticBoundedIO.SyntheticSourceOptions
extends SyntheticOptions

Synthetic bounded source options. These options are all JSON, see documentations of individual fields for details. SyntheticSourceOptions uses jackson annotations which PipelineOptionsFactory can use to parse and construct an instance.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SyntheticBoundedIO.SyntheticSourceOptions.Record Record generated by genRecord(long).

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

SyntheticOptions.DelayType, SyntheticOptions.Sampler

Field Summary

Fields

Modifier and Type	Field and Description
SyntheticOptions.Sampler	bundleSizeDistribution Distribution for generating initial split bundles.
java.lang.Integer	forceNumInitialBundles If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.
long	numRecords Total number of generated records.
SyntheticBoundedIO.ProgressShape	progressShape See SyntheticBoundedIO.ProgressShape.
long	splitPointFrequencyRecords Only records whose index is a multiple of this will be split points.

Fields inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

bytesPerRecord, cpuUtilizationInMixedDelay, delayType, hotKeyFraction, keySizeBytes, largeKeyFraction, largeKeySizeBytes, numHotKeys, seed, valueSizeBytes

Constructor Summary

Constructors

Constructor and Description
SyntheticSourceOptions()

Method Summary

Modifier and Type	Method and Description
SyntheticBoundedIO.SyntheticSourceOptions.Record	genRecord(long position)
void	validate()

Methods inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

fromIntegerDistribution, fromRealDistribution, genKvPair, hashFunction, nextDelay, setSeed, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

numRecords

public long numRecords

Total number of generated records.

splitPointFrequencyRecords

public long splitPointFrequencyRecords

Only records whose index is a multiple of this will be split points. 0 means the source is not dynamically splittable (but is perfectly statically splittable). In that case it also doesn't report progress at all.

bundleSizeDistribution

public SyntheticOptions.Sampler bundleSizeDistribution

Distribution for generating initial split bundles.

When splitting into "desiredBundleSizeBytes", we'll compute the desired number of bundles N, then sample this many numbers from this distribution, normalize their sum to 1, and use that as the boundaries of generated bundles.

The Zipf distribution is expected to be particularly useful here.

E.g., empirically, with 100 bundles, the Zipf distribution with a parameter of 3.5 will generate bundles where the largest is about 3x-10x larger than the median; with a parameter of 3.0 this ratio will be about 5x-50x; with 2.5, 5x-100x (i.e. 1 bundle can be as large as all others combined).

forceNumInitialBundles

public java.lang.Integer forceNumInitialBundles

If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.

progressShape

public SyntheticBoundedIO.ProgressShape progressShape

See SyntheticBoundedIO.ProgressShape.

Constructor Detail

SyntheticSourceOptions

public SyntheticSourceOptions()

Method Detail

validate

public void validate()

Overrides:

validate in class SyntheticOptions

genRecord

public SyntheticBoundedIO.SyntheticSourceOptions.Record genRecord(long position)