Serialized Form

Package org.apache.beam.sdk.io.synthetic

Class org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO.SyntheticBoundedSource extends org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable

serialVersionUID:

0L

Serialized Fields

sourceOptions

SyntheticBoundedIO.SyntheticSourceOptions sourceOptions

Class org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO.SyntheticSourceOptions extends SyntheticOptions implements Serializable

serialVersionUID:

0L

Serialized Fields

numRecords

long numRecords

Total number of generated records.

splitPointFrequencyRecords

long splitPointFrequencyRecords

Only records whose index is a multiple of this will be split points. 0 means the source is not dynamically splittable (but is perfectly statically splittable). In that case it also doesn't report progress at all.

bundleSizeDistribution

SyntheticOptions.Sampler bundleSizeDistribution

Distribution for generating initial split bundles.

When splitting into "desiredBundleSizeBytes", we'll compute the desired number of bundles N, then sample this many numbers from this distribution, normalize their sum to 1, and use that as the boundaries of generated bundles.

The Zipf distribution is expected to be particularly useful here.

E.g., empirically, with 100 bundles, the Zipf distribution with a parameter of 3.5 will generate bundles where the largest is about 3x-10x larger than the median; with a parameter of 3.0 this ratio will be about 5x-50x; with 2.5, 5x-100x (i.e. 1 bundle can be as large as all others combined).

forceNumInitialBundles

java.lang.Integer forceNumInitialBundles

If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.

progressShape

SyntheticBoundedIO.ProgressShape progressShape

See SyntheticBoundedIO.ProgressShape.

initializeDelayDistribution

SyntheticOptions.Sampler initializeDelayDistribution

The distribution for the delay when reading from synthetic source starts. This delay is independent of the per-record delay and uses the same types of distributions as SyntheticOptions.delayDistribution.

Class org.apache.beam.sdk.io.synthetic.SyntheticOptions extends java.lang.Object implements Serializable

serialVersionUID:

0L

Serialized Fields

keySizeBytes

long keySizeBytes

The key size in bytes.

valueSizeBytes

long valueSizeBytes

The value size in bytes.

bytesPerRecord

long bytesPerRecord

The size of a single record used for size estimation in bytes. If less than zero, keySizeBytes + valueSizeBytes is used.

numHotKeys

long numHotKeys

The number of distinct "hot" keys.

hotKeyFraction

double hotKeyFraction

The fraction of records associated with "hot" keys, which are uniformly distributed over a fixed number of hot keys.

largeKeyFraction

double largeKeyFraction

The fraction of keys that should be larger than others.

largeKeySizeBytes

double largeKeySizeBytes

The size of large keys.

seed

int seed

The seed is used for generating a hash function implementing the 128-bit murmur3 algorithm.

delayDistribution

SyntheticOptions.Sampler delayDistribution

SyntheticOptions supports several delay distributions including uniform, normal, exponential, and constant delay per record. The delay is either sleep or CPU spinning for the duration.

• The uniform delay distribution is specified through "delayDistribution":{"type":"uniform","lower":lower_bound,"upper":upper_bound}, where lower_bound and upper_bound are non-negative numbers representing the delay range in milliseconds.
• The normal delay distribution is specified through "delayDistribution":{"type":"normal","mean":mean,"stddev":stddev}, where mean is a non-negative number representing the mean of this normal distributed delay in milliseconds and stddev is a positive number representing its standard deviation.
• The exponential delay distribution is specified through "delayDistribution":{"type":"exp","mean":mean}, where mean is a positive number representing the mean of this exponentially distributed delay in milliseconds.
• The zipf distribution is specified through "delayDistribution":{"type":"zipf","param":param,"multiplier":multiplier}, where param is a number > 1 and multiplier just scales the output of the distribution. By default, the multiplier is 1. Parameters closer to 1 produce dramatically more skewed results. E.g. given 100 samples, the min will almost always be 1, while max with param 3 will usually be below 10; with param 2 max will usually be between several dozen and several hundred; with param 1.5, thousands to millions.
• The constant sleep time per record is specified through "delayDistribution":{"type":"const","const":const} where const is a non-negative number representing the constant sleep time in milliseconds.

The field delayDistribution is not used in the synthetic unbounded source. The synthetic unbounded source uses RateLimiter to control QPS.

delayType

SyntheticOptions.DelayType delayType

When 'delayDistribution' is configured, this indicates how the delay enforced ("SLEEP", "CPU", or "MIXED").

cpuUtilizationInMixedDelay

double cpuUtilizationInMixedDelay

CPU utilization when delayType is 'MIXED'. This determines the fraction of processing time spent spinning. The remaining time is spent sleeping. For each millisecond of processing time we choose to spin with probability equal to this fraction.

Class org.apache.beam.sdk.io.synthetic.SyntheticStep extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<byte[],byte[]>,org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable

Serialized Fields

options

SyntheticStep.Options options

idAndThroughput

org.apache.beam.sdk.values.KV<K,V> idAndThroughput

throttlingCounter

org.apache.beam.sdk.metrics.Counter throttlingCounter

Class org.apache.beam.sdk.io.synthetic.SyntheticStep.Options extends SyntheticOptions implements Serializable

Serialized Fields

outputRecordsPerInputRecord

double outputRecordsPerInputRecord

Amplification/filtering ratio: the number of output records should be emitted on average for each input record.

preservesInputKeyDistribution

boolean preservesInputKeyDistribution

If false, the DoFn generates a different distribution of KV pairs according to the parameters in SyntheticOptions, and input records are merely used as a “clock”; If true, the shape of the input distribution is preserved, and the DoFn only does sleeping and amplification/filtering.

maxWorkerThroughput

long maxWorkerThroughput

An upper limit on throughput across the worker for this step. In a streaming job, it is not easy to tightly control parallelism of a DoFn. It depends on various factors. As a result, it is much harder to control throughput preconfigured cpu or sleep delay. When max throughput is set, SyntheticStep delays beyond the configured delay (either cpu or sleep) in order to keep the overall throughput below the limit.

perBundleDelay

long perBundleDelay

Number of milliseconds to delay for in each bundle. Cannot be enabled simultaneously with maxWorkerThroughput >= 0.

perBundleDelayType

SyntheticOptions.DelayType perBundleDelayType

Type of per bundle delay to use ("SLEEP", "CPU", "MIXED").

reportThrottlingMicros

boolean reportThrottlingMicros

If true, reports time spent sleeping as 'cumulativeThrottlingMicros' metric. This enables Dataflow to detect throttled stages, which would influence scaling decisions.