Serialized Form (synthetic 2.30.0 API)

Package org.apache.beam.sdk.io.synthetic
- Class org.apache.beam.sdk.io.synthetic.BundleSplitter extends java.lang.Object implements Serializable
  - Serialized Fields
    - options
```
SyntheticSourceOptions options
```
- Class org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource extends org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable
  
  serialVersionUID:
  
  0L
  - Serialized Fields
    - sourceOptions
```
SyntheticSourceOptions sourceOptions
```
    - bundleSplitter
```
BundleSplitter bundleSplitter
```
- Class org.apache.beam.sdk.io.synthetic.SyntheticOptions extends java.lang.Object implements Serializable
  
  serialVersionUID:
  
  0L
  - Serialized Fields
    - keySizeBytes
```
long keySizeBytes
```
      The key size in bytes.
    - valueSizeBytes
```
long valueSizeBytes
```
      The value size in bytes.
    - bytesPerRecord
```
long bytesPerRecord
```
      The size of a single record used for size estimation in bytes. If less than zero, keySizeBytes + valueSizeBytes is used.
    - numHotKeys
```
long numHotKeys
```
      The number of distinct "hot" keys.
    - hotKeyFraction
```
double hotKeyFraction
```
      The fraction of records associated with "hot" keys, which are uniformly distributed over a fixed number of hot keys.
    - largeKeyFraction
```
double largeKeyFraction
```
      The fraction of keys that should be larger than others.
    - largeKeySizeBytes
```
double largeKeySizeBytes
```
      The size of large keys.
    - seed
```
int seed
```
      The seed is used for generating a hash function implementing the 128-bit murmur3 algorithm.
    - delayDistribution
```
SyntheticOptions.Sampler delayDistribution
```
      SyntheticOptions supports several delay distributions including uniform, normal, exponential, and constant delay per record. The delay is either sleep or CPU spinning for the duration.
      
      The uniform delay distribution is specified through "delayDistribution":{"type":"uniform","lower":lower_bound,"upper":upper_bound}, where lower_bound and upper_bound are non-negative numbers representing the delay range in milliseconds.
      The normal delay distribution is specified through "delayDistribution":{"type":"normal","mean":mean,"stddev":stddev}, where mean is a non-negative number representing the mean of this normal distributed delay in milliseconds and stddev is a positive number representing its standard deviation.
      The exponential delay distribution is specified through "delayDistribution":{"type":"exp","mean":mean}, where mean is a positive number representing the mean of this exponentially distributed delay in milliseconds.
      The zipf distribution is specified through "delayDistribution":{"type":"zipf","param":param,"multiplier":multiplier}, where param is a number > 1 and multiplier just scales the output of the distribution. By default, the multiplier is 1. Parameters closer to 1 produce dramatically more skewed results. E.g. given 100 samples, the min will almost always be 1, while max with param 3 will usually be below 10; with param 2 max will usually be between several dozen and several hundred; with param 1.5, thousands to millions.
      The constant sleep time per record is specified through "delayDistribution":{"type":"const","const":const} where const is a non-negative number representing the constant sleep time in milliseconds.
      
      The field delayDistribution is not used in the synthetic unbounded source. The synthetic unbounded source uses RateLimiter to control QPS.
    - delayType
```
SyntheticOptions.DelayType delayType
```
      When 'delayDistribution' is configured, this indicates how the delay enforced ("SLEEP", "CPU", or "MIXED").
    - cpuUtilizationInMixedDelay
```
double cpuUtilizationInMixedDelay
```
      CPU utilization when delayType is 'MIXED'. This determines the fraction of processing time spent spinning. The remaining time is spent sleeping. For each millisecond of processing time we choose to spin with probability equal to this fraction.
- Class org.apache.beam.sdk.io.synthetic.SyntheticSourceOptions extends SyntheticOptions implements Serializable
  
  serialVersionUID:
  
  0L
  - Serialized Fields
    - numRecords
```
long numRecords
```
      Total number of generated records.
    - splitPointFrequencyRecords
```
long splitPointFrequencyRecords
```
      Only records whose index is a multiple of this will be split points. 0 means the source is not dynamically splittable (but is perfectly statically splittable). In that case it also doesn't report progress at all.
    - bundleSizeDistribution
```
SyntheticOptions.Sampler bundleSizeDistribution
```
      Distribution for generating initial split bundles.
      When splitting into "desiredBundleSizeBytes", we'll compute the desired number of bundles N, then sample this many numbers from this distribution, normalize their sum to 1, and use that as the boundaries of generated bundles.
      The Zipf distribution is expected to be particularly useful here.
      E.g., empirically, with 100 bundles, the Zipf distribution with a parameter of 3.5 will generate bundles where the largest is about 3x-10x larger than the median; with a parameter of 3.0 this ratio will be about 5x-50x; with 2.5, 5x-100x (i.e. 1 bundle can be as large as all others combined).
    - forceNumInitialBundles
```
java.lang.Integer forceNumInitialBundles
```
      If specified, this source will split into exactly this many bundles regardless of the hints provided by the service.
    - progressShape
```
SyntheticSourceOptions.ProgressShape progressShape
```
      See SyntheticSourceOptions.ProgressShape.
    - initializeDelayDistribution
```
SyntheticOptions.Sampler initializeDelayDistribution
```
      The distribution for the delay when reading from synthetic source starts. This delay is independent of the per-record delay and uses the same types of distributions as SyntheticOptions.delayDistribution.
    - processingTimeDelayDistribution
```
SyntheticOptions.Sampler processingTimeDelayDistribution
```
      The delay between event and processing time. uses same types of distributions as any other delay in SyntheticSourceOptions.
      Example: we can use ConstantRealDistribution(10) to simulate constant 10 millis delay between event and processing times for each record generated by UnboundedSyntheticSource.
    - watermarkSearchInAdvanceCount
```
java.lang.Integer watermarkSearchInAdvanceCount
```
      Defines how many elements should the watermark function check in advance to "predict" how the record distribution will look like.
    - watermarkDriftMillis
```
java.lang.Integer watermarkDriftMillis
```
      Could be either positive and negative. Positive drift will "push away" the watermark from the actual records event times. Negative will bring it closer, possibly causing some events to be "late".
      By default there is no drift at all.
- Class org.apache.beam.sdk.io.synthetic.SyntheticStep extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<byte[],byte[]>,org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable
  - Serialized Fields
    - options
```
SyntheticStep.Options options
```
    - idAndThroughput
```
org.apache.beam.sdk.values.KV<K,V> idAndThroughput
```
    - throttlingCounter
```
org.apache.beam.sdk.metrics.Counter throttlingCounter
```
- Class org.apache.beam.sdk.io.synthetic.SyntheticStep.Options extends SyntheticOptions implements Serializable
  - Serialized Fields
    - outputRecordsPerInputRecord
```
double outputRecordsPerInputRecord
```
      Amplification/filtering ratio: the number of output records should be emitted on average for each input record.
    - preservesInputKeyDistribution
```
boolean preservesInputKeyDistribution
```
      If false, the DoFn generates a different distribution of KV pairs according to the parameters in SyntheticOptions, and input records are merely used as a “clock”; If true, the shape of the input distribution is preserved, and the DoFn only does sleeping and amplification/filtering.
    - maxWorkerThroughput
```
long maxWorkerThroughput
```
      An upper limit on throughput across the worker for this step. In a streaming job, it is not easy to tightly control parallelism of a DoFn. It depends on various factors. As a result, it is much harder to control throughput preconfigured cpu or sleep delay. When max throughput is set, SyntheticStep delays beyond the configured delay (either cpu or sleep) in order to keep the overall throughput below the limit.
    - perBundleDelay
```
long perBundleDelay
```
      Number of milliseconds to delay for in each bundle. Cannot be enabled simultaneously with maxWorkerThroughput >= 0.
    - perBundleDelayType
```
SyntheticOptions.DelayType perBundleDelayType
```
      Type of per bundle delay to use ("SLEEP", "CPU", "MIXED").
    - reportThrottlingMicros
```
boolean reportThrottlingMicros
```
      If true, reports time spent sleeping as 'cumulativeThrottlingMicros' metric. This enables Dataflow to detect throttled stages, which would influence scaling decisions.
- Class org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource extends org.apache.beam.sdk.io.UnboundedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>,org.apache.beam.sdk.io.synthetic.SyntheticRecordsCheckpoint> implements Serializable
  
  serialVersionUID:
  
  0L
  - Serialized Fields
    - sourceOptions
```
SyntheticSourceOptions sourceOptions
```
    - bundleSplitter
```
BundleSplitter bundleSplitter
```
    - startOffset
```
long startOffset
```
    - endOffset
```
long endOffset
```

Serialized Form

Package org.apache.beam.sdk.io.synthetic

Class org.apache.beam.sdk.io.synthetic.BundleSplitter extends java.lang.Object implements Serializable

Serialized Fields

options

Class org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource extends org.apache.beam.sdk.io.OffsetBasedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable

Serialized Fields

sourceOptions

bundleSplitter

Class org.apache.beam.sdk.io.synthetic.SyntheticOptions extends java.lang.Object implements Serializable

Serialized Fields

keySizeBytes

valueSizeBytes

bytesPerRecord

numHotKeys

hotKeyFraction

largeKeyFraction

largeKeySizeBytes

seed

delayDistribution

delayType

cpuUtilizationInMixedDelay

Class org.apache.beam.sdk.io.synthetic.SyntheticSourceOptions extends SyntheticOptions implements Serializable

Serialized Fields

numRecords

splitPointFrequencyRecords

bundleSizeDistribution

forceNumInitialBundles

progressShape

initializeDelayDistribution

processingTimeDelayDistribution

watermarkSearchInAdvanceCount

watermarkDriftMillis

Class org.apache.beam.sdk.io.synthetic.SyntheticStep extends org.apache.beam.sdk.transforms.DoFn<org.apache.beam.sdk.values.KV<byte[],byte[]>,org.apache.beam.sdk.values.KV<byte[],byte[]>> implements Serializable

Serialized Fields

options

idAndThroughput

throttlingCounter

Class org.apache.beam.sdk.io.synthetic.SyntheticStep.Options extends SyntheticOptions implements Serializable

Serialized Fields

outputRecordsPerInputRecord

preservesInputKeyDistribution

maxWorkerThroughput

perBundleDelay

perBundleDelayType

reportThrottlingMicros

Class org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource extends org.apache.beam.sdk.io.UnboundedSource<org.apache.beam.sdk.values.KV<byte[],byte[]>,org.apache.beam.sdk.io.synthetic.SyntheticRecordsCheckpoint> implements Serializable

Serialized Fields

sourceOptions

bundleSplitter

startOffset

endOffset