deepbox/optim

Optimizers

First-order, adaptive, quasi-Newton, sparse, and large-batch optimizers for deepbox/nn parameters.

Parameter updates

type ParamGroup

export type ParamGroup<Options extends Record<string, unknown>> = { readonly params: Iterable<GradTensor>; } & Partial<Options>;

Base class for all optimizers.

Optimizer

Abstract base class for all optimization algorithms.

AdaDelta

AdaDelta optimizer.

Adagrad

Adagrad (Adaptive Gradient Algorithm) optimizer.

Adam

Adam (Adaptive Moment Estimation) optimizer.

Adamax

Adamax optimizer — variant of Adam using the infinity norm.

AdamW

AdamW (Adam with decoupled Weight decay) optimizer.

ASGD

Averaged Stochastic Gradient Descent (ASGD) optimizer.

LAMB

LAMB (Layer-wise Adaptive Moments optimizer for Batch training) optimizer.

LARS

LARS (Layer-wise Adaptive Rate Scaling) optimizer.

LBFGS

L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) optimizer.

Lion

Lion is part of the deepbox/optim public API.

Nadam

Nadam (Nesterov-accelerated Adam) optimizer.

RAdam

RAdam (Rectified Adam) optimizer.

RMSprop

RMSprop (Root Mean Square Propagation) optimizer.

Rprop

Rprop (Resilient Backpropagation) optimizer.

SGD

Stochastic Gradient Descent (SGD) optimizer.

SparseAdam

SparseAdam optimizer — a variant of Adam designed for sparse gradients.

optim-optimizers.ts

import { Adam, LAMB, SGD } from "deepbox/optim";import { Linear } from "deepbox/nn";const layer = new Linear(4, 2);console.log(new SGD(layer.parameters(), { lr: 0.1 }));console.log(new Adam(layer.parameters(), { lr: 0.001 }));console.log(new LAMB(layer.parameters(), { lr: 0.001 }));

Losses & Initialization

Schedulers