Package org.apache.beam.sdk.extensions.sketching

Class ApproximateDistinct.ApproximateDistinctFn<InputT>

java.lang.Object

org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

org.apache.beam.sdk.extensions.sketching.ApproximateDistinct.ApproximateDistinctFn<InputT>

Type Parameters:

InputT - the type of the elements in the input PCollection

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

ApproximateDistinct

public static class ApproximateDistinct.ApproximateDistinctFn<InputT>
extends org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

Implements the Combine.CombineFn of ApproximateDistinct transforms.

See Also:

Serialized Form

Method Summary

Modifier and Type	Method	Description
com.clearspring.analytics.stream.cardinality.HyperLogLogPlus	addInput(com.clearspring.analytics.stream.cardinality.HyperLogLogPlus acc, InputT record)
static <InputT> ApproximateDistinct.ApproximateDistinctFn<InputT>	create(org.apache.beam.sdk.coders.Coder<InputT> coder)	Returns an ApproximateDistinct.ApproximateDistinctFn combiner with the given input coder.
com.clearspring.analytics.stream.cardinality.HyperLogLogPlus	createAccumulator()
com.clearspring.analytics.stream.cardinality.HyperLogLogPlus	extractOutput(com.clearspring.analytics.stream.cardinality.HyperLogLogPlus accumulator)	Output the whole structure so it can be queried, reused or stored easily.
java.lang.reflect.TypeVariable<?>	getAccumTVariable()
org.apache.beam.sdk.coders.Coder<AccumT>	getAccumulatorCoder(org.apache.beam.sdk.coders.CoderRegistry arg0, org.apache.beam.sdk.coders.Coder<@UnknownKeyFor @NonNull @Initialized InputT> arg1)
org.apache.beam.sdk.coders.Coder<OutputT>	getDefaultOutputCoder(org.apache.beam.sdk.coders.CoderRegistry arg0, org.apache.beam.sdk.coders.Coder<@UnknownKeyFor @NonNull @Initialized InputT> arg1)
java.lang.String	getIncompatibleGlobalWindowErrorMessage()
java.lang.reflect.TypeVariable<?>	getInputTVariable()
java.lang.reflect.TypeVariable<?>	getOutputTVariable()
com.clearspring.analytics.stream.cardinality.HyperLogLogPlus	mergeAccumulators(java.lang.Iterable<com.clearspring.analytics.stream.cardinality.HyperLogLogPlus> accumulators)
void	populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
ApproximateDistinct.ApproximateDistinctFn<InputT>	withPrecision(int p)	Returns an ApproximateDistinct.ApproximateDistinctFn combiner with a new precision p.
ApproximateDistinct.ApproximateDistinctFn<InputT>	withSparseRepresentation(int sp)	Returns an ApproximateDistinct.ApproximateDistinctFn combiner with a new sparse representation's precision sp.

Methods inherited from class org.apache.beam.sdk.transforms.Combine.CombineFn

apply, compact, defaultValue, getInputType, getOutputType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static <InputT> ApproximateDistinct.ApproximateDistinctFn<InputT> create(org.apache.beam.sdk.coders.Coder<InputT> coder)

Returns an ApproximateDistinct.ApproximateDistinctFn combiner with the given input coder.

Parameters:

coder - the coder that encodes the elements' type

withPrecision

public ApproximateDistinct.ApproximateDistinctFn<InputT> withPrecision(int p)

Returns an ApproximateDistinct.ApproximateDistinctFn combiner with a new precision p.

Keep in mind that p cannot be lower than 4, because the estimation would be too inaccurate.

See ApproximateDistinct.precisionForRelativeError(double) and ApproximateDistinct.relativeErrorForPrecision(int) to have more information about the relationship between precision and relative error.

Parameters:

p - the precision value for the normal representation

withSparseRepresentation

public ApproximateDistinct.ApproximateDistinctFn<InputT> withSparseRepresentation(int sp)

Returns an ApproximateDistinct.ApproximateDistinctFn combiner with a new sparse representation's precision sp.

Values above 32 are not yet supported by the AddThis version of HyperLogLog+.

Fore more information about the sparse representation, read Google's paper available here.

Parameters:

sp - the precision of HyperLogLog+' sparse representation

createAccumulator

public com.clearspring.analytics.stream.cardinality.HyperLogLogPlus createAccumulator()

Specified by:

createAccumulator in class org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

addInput

public com.clearspring.analytics.stream.cardinality.HyperLogLogPlus addInput(com.clearspring.analytics.stream.cardinality.HyperLogLogPlus acc,
                                                                             InputT record)

Specified by:

addInput in class org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

extractOutput

public com.clearspring.analytics.stream.cardinality.HyperLogLogPlus extractOutput(com.clearspring.analytics.stream.cardinality.HyperLogLogPlus accumulator)

Output the whole structure so it can be queried, reused or stored easily.

Specified by:

extractOutput in class org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

mergeAccumulators

public com.clearspring.analytics.stream.cardinality.HyperLogLogPlus mergeAccumulators(java.lang.Iterable<com.clearspring.analytics.stream.cardinality.HyperLogLogPlus> accumulators)

Specified by:

mergeAccumulators in class org.apache.beam.sdk.transforms.Combine.CombineFn<InputT,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus,com.clearspring.analytics.stream.cardinality.HyperLogLogPlus>

populateDisplayData

public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)

Specified by:

populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData

getAccumulatorCoder

public org.apache.beam.sdk.coders.Coder<AccumT> getAccumulatorCoder(org.apache.beam.sdk.coders.CoderRegistry arg0,
                                                                    org.apache.beam.sdk.coders.Coder<@UnknownKeyFor @NonNull @Initialized InputT> arg1)
                                                             throws org.apache.beam.sdk.coders.CannotProvideCoderException

Specified by:

getAccumulatorCoder in interface org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT extends java.lang.Object,AccumT extends java.lang.Object,OutputT extends java.lang.Object>

Throws:

org.apache.beam.sdk.coders.CannotProvideCoderException

getDefaultOutputCoder

public org.apache.beam.sdk.coders.Coder<OutputT> getDefaultOutputCoder(org.apache.beam.sdk.coders.CoderRegistry arg0,
                                                                       org.apache.beam.sdk.coders.Coder<@UnknownKeyFor @NonNull @Initialized InputT> arg1)
                                                                throws org.apache.beam.sdk.coders.CannotProvideCoderException

Specified by:

getDefaultOutputCoder in interface org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT extends java.lang.Object,AccumT extends java.lang.Object,OutputT extends java.lang.Object>

Throws:

org.apache.beam.sdk.coders.CannotProvideCoderException

getIncompatibleGlobalWindowErrorMessage

public java.lang.String getIncompatibleGlobalWindowErrorMessage()

Specified by:

getIncompatibleGlobalWindowErrorMessage in interface org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT extends java.lang.Object,AccumT extends java.lang.Object,OutputT extends java.lang.Object>

getInputTVariable

public java.lang.reflect.TypeVariable<?> getInputTVariable()

getAccumTVariable

public java.lang.reflect.TypeVariable<?> getAccumTVariable()

getOutputTVariable

public java.lang.reflect.TypeVariable<?> getOutputTVariable()