org.apache.beam.sdk.io.synthetic

Class SyntheticStep.Options

java.lang.Object

org.apache.beam.sdk.io.synthetic.SyntheticOptions

org.apache.beam.sdk.io.synthetic.SyntheticStep.Options

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

SyntheticStep

public static class SyntheticStep.Options
extends SyntheticOptions

Synthetic step options. These options are all JSON, see documentations of individual fields for details. Options uses jackson annotations which PipelineOptionsFactory can use to parse and construct an instance.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

SyntheticOptions.DelayType, SyntheticOptions.Sampler

Field Summary

Fields

Modifier and Type	Field and Description
long	maxWorkerThroughput An upper limit on throughput across the worker for this step.
double	outputRecordsPerInputRecord Amplification/filtering ratio: the number of output records should be emitted on average for each input record.
long	perBundleDelay Number of milliseconds to delay for in each bundle.
SyntheticOptions.DelayType	perBundleDelayType Type of per bundle delay to use ("SLEEP", "CPU", "MIXED").
boolean	preservesInputKeyDistribution If false, the DoFn generates a different distribution of KV pairs according to the parameters in SyntheticOptions, and input records are merely used as a “clock”; If true, the shape of the input distribution is preserved, and the DoFn only does sleeping and amplification/filtering.
boolean	reportThrottlingMicros If true, reports time spent sleeping as 'cumulativeThrottlingMicros' metric.

Fields inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

bytesPerRecord, cpuUtilizationInMixedDelay, delayType, hotKeyFraction, keySizeBytes, largeKeyFraction, largeKeySizeBytes, numHotKeys, seed, valueSizeBytes

Constructor Summary

Constructors

Constructor and Description
Options()

Method Summary

Modifier and Type	Method and Description
void	validate()

Methods inherited from class org.apache.beam.sdk.io.synthetic.SyntheticOptions

fromIntegerDistribution, fromJsonString, fromRealDistribution, genKvPair, hashFunction, nextDelay, setSeed, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

outputRecordsPerInputRecord

public double outputRecordsPerInputRecord

Amplification/filtering ratio: the number of output records should be emitted on average for each input record.

preservesInputKeyDistribution

public boolean preservesInputKeyDistribution

If false, the DoFn generates a different distribution of KV pairs according to the parameters in SyntheticOptions, and input records are merely used as a “clock”; If true, the shape of the input distribution is preserved, and the DoFn only does sleeping and amplification/filtering.

maxWorkerThroughput

public long maxWorkerThroughput

An upper limit on throughput across the worker for this step. In a streaming job, it is not easy to tightly control parallelism of a DoFn. It depends on various factors. As a result, it is much harder to control throughput preconfigured cpu or sleep delay. When max throughput is set, SyntheticStep delays beyond the configured delay (either cpu or sleep) in order to keep the overall throughput below the limit.

perBundleDelay

public long perBundleDelay

Number of milliseconds to delay for in each bundle. Cannot be enabled simultaneously with maxWorkerThroughput >= 0.

perBundleDelayType

public SyntheticOptions.DelayType perBundleDelayType

Type of per bundle delay to use ("SLEEP", "CPU", "MIXED").

reportThrottlingMicros

public boolean reportThrottlingMicros

If true, reports time spent sleeping as 'cumulativeThrottlingMicros' metric. This enables Dataflow to detect throttled stages, which would influence scaling decisions.

Constructor Detail

Options

public Options()

Method Detail

validate

public void validate()

Overrides:

validate in class SyntheticOptions