Package jasima.core.experiment

Class MultipleReplicationExperiment

  • All Implemented Interfaces:
    Notifier<Experiment,​Experiment.ExperimentEvent>, ValueStore, Serializable, Cloneable
    public class MultipleReplicationExperiment
extends AbstractMultiExperiment

    Runs an arbitrary baseExperiment multiple times (determined by maxReplications). All numeric results of the base experiment are averaged over the runs, other result types are returned as an array containing all values over the runs.

    Optionally the maximum number of experiments run can be determined by a confidence interval (t-test). To use this feature you have to tell addConfIntervalMeasure(String) which result(s) of the base experiment to use.

    In case of dynamic runs the following procedure is followed:

    1. getMinReplications() replications are performed
    2. no further runs are performed if the confidence interval is less than a certain allowance value
      1. width of the confidence interval is determined by setErrorProb(double), default is 0.05
      2. allowance value is computed by the runMean * allowancePercentage() (default 1%)
    3. if there is another run (i.e., confidence interval too large), another batch of getMinReplications() is performed, i.e., go back to step 1
    Author:
    Torsten Hildebrandt
    See Also:
    OCBAExperiment, Serialized Form

    • Constructor Detail

      • MultipleReplicationExperiment

        public MultipleReplicationExperiment()

      • MultipleReplicationExperiment

        public MultipleReplicationExperiment​(Experiment e,
                                     int numReps)

    • Method Detail

      • getMinReplications

        public int getMinReplications()

      • setMinReplications

        public void setMinReplications​(int minReplications)
        Sets the minimum number of replications to perform if the total number of replications is dynamic (i.e., if at least 1 result name is given in confIntervalMeasure).

        If the number of runs is not dynamic, this setting has no effect. A value of 0 uses the number of available cpu cores.

        Parameters:
        minReplications - Minimum number of replications.

      • getMaxReplications

        public int getMaxReplications()

      • setMaxReplications

        public void setMaxReplications​(int maxReplications)
        Sets the maximum number of replications to perform. If the number of runs is not dynamic, this sets the total number of replications to perform. If the total number of replications is dynamic (i.e., if at least 1 result name is given in confIntervalMeasures), this sets the maximum number of replications to perform.
        Parameters:
        maxReplications - The number of replications to perform.

      • getErrorProb

        public double getErrorProb()

      • setErrorProb

        public void setErrorProb​(double errorProb)

        Sets the error probability used when computing the width of the confidence interval of confIntervalMeasures. The closer this setting is to 0, the more replications will be performed (with less uncertain results).

        This setting only has an effect if the total number of runs is dynamic. Its default value is 0.05.

        Parameters:
        errorProb - Desired maximum error probability for computing the confidence intervals.

      • getAllowancePercentage

        public double getAllowancePercentage()

      • setAllowancePercentage

        public void setAllowancePercentage​(double allowancePercentage)
        Sets the desired target quality of results as a percentage of the mean across all replications performed so far. Its default value is 0.01, i.e., 1%. This setting has no effect if the total number of runs is static.

        Let's say after 5 replications the observed mean of a result in confIntervalMeasure is 987.6 with some standard deviation (SD). Say this SD together with a certain errorProb leads to a confidence interval of ±10.2. No further replications are performed, if the total width of this interval (2*10.2=20.4) is smaller than the observed mean multiplied by allowancePercentage (987.6*0.01=9.876). In the example, the result is not yet precise enough, as the observed uncertainty is not smaller than the target value (20.4 > 9.876). Therefore further replications would be performed to further reduce the uncertainty of results.

        Parameters:
        allowancePercentage - The desired maximum result uncertainty as a percentage of the mean value.

      • addConfIntervalMeasure

        public void addConfIntervalMeasure​(String name)

      • removeConfIntervalMeasure

        public boolean removeConfIntervalMeasure​(String name)

      • getConfIntervalMeasures

        public String[] getConfIntervalMeasures()

      • setConfIntervalMeasures

        public void setConfIntervalMeasures​(String... confIntervalMeasures)
        Sets the list of results that will be used when the total number of replications to perform is dynamic. If any result name is given here as a confIntervalMeasure, then at least minReplication replications are performed. After this, additional replications are performed until all confIntervalMeasures are precise enough or a total of maxReplications was performed. The meaning of "precise enough" is determined by the settings allowancePercentage and errorProb.
        Parameters:
        confIntervalMeasures - A list of all result names that should be checked when the number of runs is dynamic.

      • getBaseExperiment

        public Experiment getBaseExperiment()

      • setBaseExperiment

        public void setBaseExperiment​(Experiment baseExperiment)
        Sets the base experiment that is executed multiple times in various configurations. Before experiment execution, a copy (clone) of baseExperiment is created and run. Therefore the specific experiment instance passed as the baseExperiment is never actually executed.
        Parameters:
        baseExperiment - The base experiment to use.