org.apache.beam.sdk.io.synthetic

Class SyntheticSourceOptions

java.lang.Object

org.apache.beam.sdk.io.synthetic.SyntheticOptions

org.apache.beam.sdk.io.synthetic.SyntheticSourceOptions

All Implemented Interfaces:

java.io.Serializable

public class SyntheticSourceOptions
extends SyntheticOptions

Synthetic bounded source options. These options are all JSON, see documentations of individual fields for details. SyntheticSourceOptions uses jackson annotations which PipelineOptionsFactory can use to parse and construct an instance.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SyntheticSourceOptions.ProgressShape Shape of the progress reporting curve as a function of the current offset in the SyntheticBoundedSource.
static class	SyntheticSourceOptions.Record Record generated by genRecord(long).

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

SyntheticOptions.DelayType, SyntheticOptions.Sampler

Field Summary

Fields

Modifier and Type	Field and Description
SyntheticOptions.Sampler	bundleSizeDistribution Distribution for generating initial split bundles.
java.lang.Integer	forceNumInitialBundles If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.
long	numRecords Total number of generated records.
SyntheticSourceOptions.ProgressShape	progressShape See SyntheticSourceOptions.ProgressShape.
long	splitPointFrequencyRecords Only records whose index is a multiple of this will be split points.
java.lang.Integer	watermarkDriftMillis Could be either positive and negative.
java.lang.Integer	watermarkSearchInAdvanceCount Defines how many elements should the watermark function check in advance to "predict" how the record distribution will look like.

Fields inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

bytesPerRecord, cpuUtilizationInMixedDelay, delayType, hotKeyFraction, keySizeBytes, largeKeyFraction, largeKeySizeBytes, numHotKeys, seed, valueSizeBytes

Constructor Summary

Constructors

Constructor and Description
SyntheticSourceOptions()

Method Summary

Modifier and Type	Method and Description
SyntheticSourceOptions.Record	genRecord(long position)
org.joda.time.Duration	nextInitializeDelay(long seed) Generates a random delay value for the synthetic source initialization using the distribution defined by initializeDelayDistribution.
org.joda.time.Duration	nextProcessingTimeDelay(long seed) Generates a random delay value between event and processing time using the distribution defined by processingTimeDelayDistribution.
void	validate()

Methods inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

fromIntegerDistribution, fromJsonString, fromRealDistribution, genKvPair, hashFunction, nextDelay, setSeed, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

numRecords

public long numRecords

Total number of generated records.

splitPointFrequencyRecords

public long splitPointFrequencyRecords

Only records whose index is a multiple of this will be split points. 0 means the source is not dynamically splittable (but is perfectly statically splittable). In that case it also doesn't report progress at all.

bundleSizeDistribution

public SyntheticOptions.Sampler bundleSizeDistribution

Distribution for generating initial split bundles.

When splitting into "desiredBundleSizeBytes", we'll compute the desired number of bundles N, then sample this many numbers from this distribution, normalize their sum to 1, and use that as the boundaries of generated bundles.

The Zipf distribution is expected to be particularly useful here.

E.g., empirically, with 100 bundles, the Zipf distribution with a parameter of 3.5 will generate bundles where the largest is about 3x-10x larger than the median; with a parameter of 3.0 this ratio will be about 5x-50x; with 2.5, 5x-100x (i.e. 1 bundle can be as large as all others combined).

forceNumInitialBundles

public java.lang.Integer forceNumInitialBundles

If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.

progressShape

public SyntheticSourceOptions.ProgressShape progressShape

See SyntheticSourceOptions.ProgressShape.

watermarkSearchInAdvanceCount

public java.lang.Integer watermarkSearchInAdvanceCount

Defines how many elements should the watermark function check in advance to "predict" how the record distribution will look like.

watermarkDriftMillis

public java.lang.Integer watermarkDriftMillis

Could be either positive and negative. Positive drift will "push away" the watermark from the actual records event times. Negative will bring it closer, possibly causing some events to be "late".

By default there is no drift at all.

Constructor Detail

SyntheticSourceOptions

public SyntheticSourceOptions()

Method Detail

nextInitializeDelay

public org.joda.time.Duration nextInitializeDelay(long seed)

Generates a random delay value for the synthetic source initialization using the distribution defined by initializeDelayDistribution.

nextProcessingTimeDelay

public org.joda.time.Duration nextProcessingTimeDelay(long seed)

Generates a random delay value between event and processing time using the distribution defined by processingTimeDelayDistribution.

validate

public void validate()

Overrides:

validate in class SyntheticOptions

genRecord

public SyntheticSourceOptions.Record genRecord(long position)