Skip to content

deepbox/optim

Schedulers

Learning-rate schedulers: step, exponential, cosine, one-cycle, cyclic, linear, polynomial, warmup, plateau and chained schedules.

  • Scheduling

Overview

  • Call scheduler.step() once per epoch, after optimizer.step(). A scheduler writes the new rate into the optimizer's parameter groups, and getLastLr() reads the current rates.
  • Creating a scheduler performs the first step, so epoch 0 already runs at the scheduled rate, as in PyTorch. A second scheduler built on the same optimizer starts from the optimizer's original rate, not from where the first one left off.
  • WarmupLR(optimizer, afterScheduler, { warmupEpochs }) ramps the rate up linearly and then hands over to afterScheduler, which you create first on the same optimizer. SequentialLR(optimizer, { schedulers, milestones }) switches schedulers at the milestones.
  • ReduceLROnPlateau is not an LRScheduler. Call step(metric) with the value it should monitor. Option names are camelCase (tMax, stepSize, warmupEpochs), and stateDict() and loadStateDict() checkpoint a scheduler.

Types

2

PlateauStateDict

type

Serializable snapshot of a ReduceLROnPlateau scheduler.

  • best: number

    Best metric value seen so far (Infinity / -Infinity before the first step).

  • numBadEpochs: number

    Number of consecutive epochs without improvement.

  • cooldownCounter: number

    Remaining cooldown epochs.

  • lastLr: number[]

    Learning rate of every parameter group at the time of the snapshot.

SchedulerStateDict

type

Serializable snapshot of a scheduler, returned by stateDict().

  • lastEpoch: number

    Index of the last completed epoch (-1 before the first step).

  • baseLrs: number[]

    Base learning rate of every parameter group.

  • lastLr: number[]

    Learning rate of every parameter group at the time of the snapshot.

  • children?: SchedulerStateDict[]

    Snapshots of wrapped schedulers (WarmupLR, SequentialLR).

Classes

14

CosineAnnealingLR

extends LRScheduler

Cosine annealing learning rate scheduler.

Sets the learning rate using a cosine annealing schedule. lr = etaMin + (baseLr - etaMin) * (1 + cos(π * epoch / T_max)) / 2

Example
const scheduler = new CosineAnnealingLR(optimizer, { T_max: 100, etaMin: 0.001 });
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { T_max?: number; tMax?: number; etaMin?: number; lastEpoch?: number; })
  • getLr(): number[]

CosineAnnealingWarmRestarts

extends LRScheduler

Cosine annealing with warm restarts.

The learning rate follows a cosine curve from the base lr to etaMin over T_0 epochs, then restarts. After each restart the period is multiplied by T_mult (and rounded down to a whole number of epochs).

lr = etaMin + 0.5 * (baseLr - etaMin) * (1 + cos(pi * T_cur / T_i))

t0 and tMult are accepted as aliases of T_0 and T_mult.

Example
const scheduler = new CosineAnnealingWarmRestarts(optimizer, { T_0: 10, T_mult: 2 });// restarts at epochs 10, 30, 70, ...
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { T_0?: number; t0?: number; T_mult?: number; tMult?: number; etaMin?: number; lastEpoch?: number; })
  • getLr(): number[]

CyclicLR

extends LRScheduler

Cyclic learning rate scheduler.

Cycles the learning rate between baseLr and maxLr. Each cycle rises over stepSizeUp steps and falls over stepSizeDown steps. Modes: - "triangular": constant amplitude. - "triangular2": the amplitude halves every cycle. - "exp_range": the amplitude is scaled by gamma ** step.

With several parameter groups, baseLr and maxLr apply to the first group and the other groups keep their base lr ratio to it.

Example
const scheduler = new CyclicLR(optimizer, { baseLr: 0.001, maxLr: 0.01, stepSizeUp: 100 });
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { baseLr: number; maxLr: number; stepSizeUp?: number; stepSizeDown?: number; mode?: "triangular" | "triangular2" | "exp_range"; gamma?: number; lastEpoch?: number; })
  • getLr(): number[]

ExponentialLR

extends LRScheduler

Exponential learning rate scheduler.

Decays the learning rate exponentially every epoch. lr = baseLr * gamma^epoch

Example
const scheduler = new ExponentialLR(optimizer, { gamma: 0.95 });// lr *= 0.95 each epoch
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { gamma: number; lastEpoch?: number; })
  • getLr(): number[]

LambdaLR

extends LRScheduler

Lambda learning rate scheduler.

Sets the learning rate to baseLr * lrLambda(epoch) for each parameter group. Each parameter group can have its own lambda function if an array is provided.

Example
// Halve the learning rate every 10 epochsconst scheduler = new LambdaLR(optimizer, {  lrLambda: (epoch) => 0.5 ** Math.floor(epoch / 10)});
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { lrLambda: ((epoch: number) => number) | Array<(epoch: number) => number>; lastEpoch?: number; })
  • getLr(): number[]

LinearLR

extends LRScheduler

Linear learning rate scheduler.

Linearly interpolates the learning rate multiplicative factor from startFactor to endFactor over totalIters epochs. After totalIters, the factor remains at endFactor. startFactor must be > 0 (default 1/3); endFactor may be 0 to decay to zero (default 1).

lr = baseLr * (startFactor + (endFactor - startFactor) * min(epoch, totalIters) / totalIters)

Example
const scheduler = new LinearLR(optimizer, {  startFactor: 0.1,  endFactor: 0.01,  totalIters: 100});
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { startFactor?: number; endFactor?: number; totalIters: number; lastEpoch?: number; })
  • getLr(): number[]

LRScheduler

Base class for learning rate schedulers.

Learning rate schedulers adjust the learning rate during training according to a predefined schedule. This can help improve convergence and prevent overshooting optimal solutions.

Constructing a scheduler performs the first step, so epoch 0 already runs at the scheduled rate (the same as PyTorch). The base learning rate of each group is the rate the optimizer was created with: a second scheduler built on the same optimizer starts from that original rate, not from the value the first scheduler left behind, unless the rate was changed by hand in between.

Schedulers write the rate to paramGroups[i].options.lr (or paramGroups[i].lr). Optimizers that use the learning rate only to initialise their state (for example Rprop, whose step sizes start at lr) are not affected by later changes.

Example
import { SGD, StepLR } from 'deepbox/optim'; const optimizer = new SGD(model.parameters(), { lr: 0.1 });const scheduler = new StepLR(optimizer, { stepSize: 10, gamma: 0.1 }); for (let epoch = 0; epoch < 100; epoch++) {  train();  scheduler.step();}
Members (7)
  • constructor(optimizer: SchedulerOptimizer, lastEpoch?: number)
  • epoch: number

    Get current epoch number.

  • abstract getLr(): number[]

    Compute the learning rate for the current epoch.

  • step(): void

    Perform a scheduler step, updating learning rates.

  • getLastLr(): number[]

    Get the current learning rates for all parameter groups.

  • stateDict(): SchedulerStateDict

    Snapshot of the scheduler for checkpointing: the epoch counter, the base learning rates and the current learning rates.

  • loadStateDict(state: SchedulerStateDict): void

    Restore a snapshot produced by stateDict().

MultiStepLR

extends LRScheduler

Multi-step learning rate scheduler.

Decays the learning rate by gamma once the epoch reaches one of the milestones.

Example
const scheduler = new MultiStepLR(optimizer, { milestones: [30, 80], gamma: 0.1 });// lr = 0.1 for epochs 0-29// lr = 0.01 for epochs 30-79// lr = 0.001 for epochs 80+
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { milestones: number[]; gamma?: number; lastEpoch?: number; })
  • getLr(): number[]

OneCycleLR

extends LRScheduler

One-cycle learning rate scheduler.

Implements the 1cycle policy as PyTorch does: the rate starts at maxLr / divFactor, rises to maxLr at step pctStart * totalSteps - 1, then falls to maxLr / (divFactor * finalDivFactor) at step totalSteps - 1. Both phases use the same annealing curve ("cos" or "linear"). After the last step the rate stays at the minimum.

With several parameter groups, maxLr applies to the first group and the other groups keep their base lr ratio to it.

Example
const scheduler = new OneCycleLR(optimizer, {  maxLr: 0.1,  totalSteps: 1000,  pctStart: 0.3});
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { maxLr: number; totalSteps: number; pctStart?: number; divFactor?: number; finalDivFactor?: number; annealStrategy?: "cos" | "linear"; lastEpoch?: number; })
  • getLr(): number[]

PolynomialLR

extends LRScheduler

Polynomial learning rate scheduler.

Decays the learning rate using a polynomial function from the initial lr to a target value over a given number of epochs.

lr = (baseLr - endLr) * (1 - epoch/totalIters)^power + endLr

After totalIters, lr stays at endLr.

Example
const scheduler = new PolynomialLR(optimizer, {  totalIters: 100,  power: 2.0});
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { totalIters: number; power?: number; endLr?: number; lastEpoch?: number; })
  • getLr(): number[]

ReduceLROnPlateau

Reduce learning rate on plateau.

Reduces learning rate when a metric has stopped improving. This scheduler reads a metric value and if no improvement is seen for 'patience' epochs, the learning rate is reduced.

A reduction is skipped when it would change the rate by no more than eps. minLr may be one number for all groups or one number per group.

Example
const scheduler = new ReduceLROnPlateau(optimizer, {  mode: 'min',  factor: 0.1,  patience: 10}); for (let epoch = 0; epoch < 100; epoch++) {  const valLoss = validate();  scheduler.step(valLoss);}
Members (5)
  • constructor(optimizer: SchedulerOptimizer, options?: { mode?: "min" | "max"; factor?: number; patience?: number; threshold?: number; thresholdMode?: "rel" | "abs"; cooldown?: number; minLr?: number | readonly number[]; eps?: number; })
  • step(metric: number): void

    Perform a scheduler step based on the metric value.

  • getLastLr(): number[]

    Get the current learning rates for all parameter groups.

  • stateDict(): PlateauStateDict

    Snapshot of the scheduler for checkpointing: the best metric, the bad-epoch and cooldown counters and the current learning rates.

  • loadStateDict(state: PlateauStateDict): void

    Restore a snapshot produced by stateDict().

SequentialLR

extends LRScheduler

Sequential learning rate scheduler.

Chains several schedulers. Scheduler i is active from epoch milestones[i - 1] (the first one from epoch 0) up to the next milestone. When a milestone is reached, the next scheduler is restarted at its own epoch 0.

Create all schedulers on the same optimizer; they share its original learning rates. The steps they performed while being constructed are undone, so the first scheduler's epoch 0 is the epoch 0 of the chain.

Example
const scheduler1 = new LinearLR(optimizer, { startFactor: 0.1, totalIters: 5 });const scheduler2 = new ExponentialLR(optimizer, { gamma: 0.9 });const scheduler = new SequentialLR(optimizer, {  schedulers: [scheduler1, scheduler2],  milestones: [5]});
Members (3)
  • constructor(optimizer: SchedulerOptimizer, options: { schedulers: LRScheduler[]; milestones: number[]; lastEpoch?: number; })
  • getLr(): number[]
  • step(): void

StepLR

extends LRScheduler

Step learning rate scheduler.

Decays the learning rate by gamma every stepSize epochs. lr = baseLr * gamma^(epoch // stepSize)

Example
const scheduler = new StepLR(optimizer, { stepSize: 30, gamma: 0.1 });// lr = 0.1 for epochs 0-29// lr = 0.01 for epochs 30-59// lr = 0.001 for epochs 60-89
Members (2)
  • constructor(optimizer: SchedulerOptimizer, options: { stepSize: number; gamma?: number; lastEpoch?: number; })
  • getLr(): number[]

WarmupLR

extends LRScheduler

Warmup scheduler that wraps another scheduler.

Linearly increases the learning rate from baseLr / warmupEpochs (epoch 0) to the base lr (epoch warmupEpochs - 1), then delegates to the wrapped scheduler, which continues from its own epoch 1 at epoch warmupEpochs. Without a wrapped scheduler the rate stays at the base lr after warmup.

Create the wrapped scheduler first on the same optimizer; both then share the same base learning rates.

Example
const baseScheduler = new CosineAnnealingLR(optimizer, { T_max: 100 });const scheduler = new WarmupLR(optimizer, baseScheduler, { warmupEpochs: 5 });
Members (3)
  • constructor(optimizer: SchedulerOptimizer, afterScheduler: LRScheduler | null, options: { warmupEpochs: number; lastEpoch?: number; })
  • getLr(): number[]
  • step(): void
optim-schedulers.ts
import { Linear } from "deepbox/nn";import { Adam, CosineAnnealingLR, ReduceLROnPlateau, WarmupLR } from "deepbox/optim"; const layer = new Linear(4, 2);const optimizer = new Adam(layer.parameters(), { lr: 0.001 }); const cosine = new CosineAnnealingLR(optimizer, { tMax: 20 });const scheduler = new WarmupLR(optimizer, cosine, { warmupEpochs: 5 }); for (let epoch = 0; epoch < 8; epoch++) {  // Train for one epoch here, then update the rate.  optimizer.step();  scheduler.step();  console.log(epoch, scheduler.getLastLr());} const plateauOptimizer = new Adam(layer.parameters(), { lr: 0.01 });const plateau = new ReduceLROnPlateau(plateauOptimizer, { patience: 2 });plateau.step(0.5); // pass the monitored value, such as the validation loss

Found a mistake? Open an issue.