com.aliasi.stats

Class AnnealingSchedule

java.lang.Object

com.aliasi.stats.AnnealingSchedule

public abstract class AnnealingSchedule
extends Object

An AnnealingSchedule instance implements a method to return the learning rate for a specified epoch. It also has a method to receive callback reports on the resulting error rate for an epoch so that implementations may adapt the learning rate to received error.

This class provides an abstract base class implementation along with static factory methods to create the two most popular annealing schedules, constant, exponential decay and inverse scaling.

Constant Learning Rate

The contant learning rate schedule always returns the same learning rate, which is fixed at construction time.

Exponential Decay

The exponential decay annealing schedule sets the learning rate as follows based on a specfieid exponential base:

 learningRate(epoch) = initialLearningRate * java.lang.Math.pow(base,epoch)

Under this schedule, the learning rate undergoes exponential decay starting at the initial learning rate and decaying exponentially at a rate determined by the base of the exponent.

The exponential learning rate can find solutions quickly in fairly well-behaved spaces, but may stop short of the minimum error solution due to too much decay in later epochs.

Inverse Scaling

The inverse learning rate scaling sets the learning rate as:

 learningRate(epoch) = initialLearningRate / (1 + epoch/annealingRate)

The inverse scaling annealing schedule lowers the rate more quickly than the exponential rates initially and then more slowly for later epochs.

This is a popular learning rate because it is guaranteed to converge in the limit. It can be slower to converge once it gets near a solution than exponential decay.

Rejecting Updates

The return value of the receivedError() method is a boolean flag indicating whether to accept the updates to the underlying vectors or not. This allows sampling-based annealing schedules to be implemented that evaluate several learning rates and accept just the one with the most error reduction.

The method allowsRejection() should be overridden to return false if the annealing schedule never rejects updates; this will save a coefficient vector copy per epoch in logistic regression.

Convergence Guarantees in the Limit

The inverse scale metric is popular because it is theoretically guaranteed to converge in the limit.

An annealing rate will converge in the limit within arbitrary precision of a solution if the learning rate satisfies:

 Σepoch learningRate(epoch) = ∞

and:

 Σepoch learningRate(epoch)2 < ∞

References

• Wikipedia: Simulated Annealing

Since:

LingPipe3.5

Version:

4.0.1

Author:

Bob Carpenter

Constructor Summary

Constructors

Constructor and Description
AnnealingSchedule() Do-nothing constructor to be used by concrete implementations.

Method Summary

Modifier and Type	Method and Description
boolean	allowsRejection() Return true if this annealing schedule allows updates to be rejected.
static AnnealingSchedule	constant(double learningRate) Return the annealing schedule for the specified constant learning rate.
static AnnealingSchedule	exponential(double initialLearningRate, double base) Return the exponential annealing schedule with the specified initial learning rate and exponent.
static AnnealingSchedule	inverse(double initialLearningRate, double annealingRate) Return the inverse annealing schedule with the specified initial learning rate and annealing rate.
abstract double	learningRate(int epoch) Return the learning rate for the specified epoch.
boolean	receivedError(int epoch, double rate, double error) Receive a report from an optimizer about the effect of the specified learning rate in the specified epoch and return true if the update producing the error should be accepted or rejected.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AnnealingSchedule

public AnnealingSchedule()

Do-nothing constructor to be used by concrete implementations.

Method Detail

allowsRejection

public boolean allowsRejection()

Return true if this annealing schedule allows updates to be rejected. The implementation in this class always returns true for backward compatibility reasons. Implementations that never reject updates should override this method to return false.

Returns:

true if this schedule allows update rejections.

learningRate

public abstract double learningRate(int epoch)

Return the learning rate for the specified epoch.

Parameters:

epoch - Epoch being evaluated.

Returns:

Learning rate for that epoch.

receivedError

public boolean receivedError(int epoch,
                             double rate,
                             double error)

Receive a report from an optimizer about the effect of the specified learning rate in the specified epoch and return true if the update producing the error should be accepted or rejected.

This abstract class's implementation of this method is to do nothing. Concrete subclasses which adapt learning rates based on empirical error reports from the optimizer must override this method.

Parameters:

epoch - Training epoch.

rate - Training rate.

error - Training error.

inverse

public static AnnealingSchedule inverse(double initialLearningRate,
                                        double annealingRate)

Return the inverse annealing schedule with the specified initial learning rate and annealing rate. The inverse annealing schedule is not adaptive. adaptation.

Parameters:

initialLearningRate - Initial learning rate for epoch zero.

annealingRate - Rate at which initial learning rate anneals.

Returns:

The inverse annealing schedule for the specified initial learning rate and annealing rate.

Throws:

IllegalArgumentException - If the initial learning rate or the annealing rates are not finite and positive.

exponential

public static AnnealingSchedule exponential(double initialLearningRate,
                                            double base)

Return the exponential annealing schedule with the specified initial learning rate and exponent. The exponential annealing schedule is not adaptive.

Parameters:

initialLearningRate - Initial learning rate for epoch 0.

base - Base of the exponential decay.

Returns:

The exponential annealing schedule for the specified initial learning rate and exponent.

Throws:

IllegalArgumentException - If the initial learning rate is not finite and positive, or if the exponent is not between 0.0 (exclusive) and 1.0 (inclusive).

constant

public static AnnealingSchedule constant(double learningRate)

Return the annealing schedule for the specified constant learning rate. The constant annealing schedule is not adaptive.

Parameters:

learningRate - The constant rate returned by this annealing schedule.

Returns:

The annealing schedule returning the specified constant learning rate.

Throws:

IllegalArgumentException - If the learning rate is not finite and positive.