deepbox/nn

Losses & Initialization

Regression and classification losses, ranking and sequence losses, and in-place weight initializers.

Losses
Initializers

Overview

Losses take (predictions, targets) and often a reduction of "mean" (default), "sum" or "none". When the input is a GradTensor, which is what forward returns during training, the loss is a GradTensor, so you can call backward() and read the number with loss.item().
With plain tensors, mseLoss and most others return a Tensor, but crossEntropyLoss and binaryCrossEntropyWithLogitsLoss return a number. ctcLoss, gaussianNLLLoss and poissonNLLLoss take and return plain tensors, so they cannot drive training.
crossEntropyLoss(logits, target) expects raw logits, so do not apply softmax first. The target is either integer class indices of shape [n] or class probabilities of shape [n, nClasses]. Options are weight, ignoreIndex and labelSmoothing.
Initializers such as xavierUniform_, kaimingNormal_ and zeros_ fill a tensor in place and return it. The trailing underscore follows PyTorch and is part of the name. kaimingNormal_ is the camelCase form of the deprecated kaiming_normal_. They draw from the global generator, so setSeed makes them repeatable. Gradient clipping is on the Module, Containers & Trainer page.

Types

BinaryCrossEntropyWithLogitsOptions

interface

Options for binaryCrossEntropyWithLogitsLoss.

readonly reduction?: LossReduction
"mean" (default), "sum", or "none" to return the element-wise loss.
readonly posWeight?: number | Tensor
Weight of the positive class.
readonly weight?: Tensor
Element-wise rescaling weight that broadcasts against the logits.

CrossEntropyLossOptions

interface

Options for crossEntropyLoss.

readonly reduction?: LossReduction
"mean" (default), "sum", or "none" to return one loss per sample.
readonly weight?: Tensor | readonly number[]
Per-class rescaling weights, length n_classes.
readonly ignoreIndex?: number
Class index to skip (class-index targets only).
readonly labelSmoothing?: number
Label smoothing in [0, 1].

CtcLossOptions

interface

Options for ctcLoss.

readonly blank?: number
Index of the blank label (default: 0).
readonly reduction?: LossReduction
"mean" (default) divides each sample's loss by its target length (at least 1) and averages over the batch, as PyTorch does.
readonly zeroInfinity?: boolean
Replace infinite losses (inputs too short for the target) with zero (default: false).

LossReduction

type

How per-sample losses are combined: average, total, or kept per sample.

type LossReduction = "mean" | "sum" | "none";

NllLossOptions

interface

Options for nllLoss.

readonly reduction?: LossReduction
"mean" (default), "sum", or "none" to return one loss per sample.
readonly weight?: Tensor | readonly number[]
Per-class rescaling weights, length n_classes.
readonly ignoreIndex?: number
Class index to skip: such samples add no loss and are not counted by "mean".

Functions

calculateFanInOut

function calculateFanInOut(tensor: AnyTensor): { fanIn: number; fanOut: number; };

Parameters

tensor AnyTensor

Returns

{ fanIn: number; fanOut: number; }

calculateGain

function calculateGain(nonlinearity: string, param?: number): number;

Recommended gain for a nonlinearity (mirrors torch.nn.init.calculate_gain).

Parameters

nonlinearity string: One of linear, conv1d, conv2d, conv3d, conv_transpose1d, conv_transpose2d, conv_transpose3d, sigmoid, tanh, relu, leaky_relu, selu
paramoptional number: Negative slope for leaky_relu (default: 0.01)

Returns

numberThe gain factor

Throws

InvalidParameterErrorFor an unknown nonlinearity or a non-finite slope

constant

function constant<T extends AnyTensor>(tensor: T, val: number): T;

Fill tensor with a constant value.

Works for every numeric dtype. Integer and bool tensors store the truncated value (bool: non-zero is true); int64 converts through BigInt. Half-precision tensors store the rounded value.

Parameters

tensor T: Tensor to fill in-place
val number: Value to fill with

Returns

TThe modified tensor

Throws

DTypeErrorFor string tensors
InvalidParameterErrorIf val is not finite for an integer or int64 tensor

constant_

function constant_<T extends AnyTensor>(tensor: T, val: number): T;

Fill tensor with a constant value.

Works for every numeric dtype. Integer and bool tensors store the truncated value (bool: non-zero is true); int64 converts through BigInt. Half-precision tensors store the rounded value.

Parameters

tensor T: Tensor to fill in-place
val number: Value to fill with

Returns

TThe modified tensor

Throws

DTypeErrorFor string tensors
InvalidParameterErrorIf val is not finite for an integer or int64 tensor

eye_

function eye_<T extends AnyTensor>(tensor: T): T;

Fill a 2-D tensor with the identity matrix (ones on the main diagonal, zeros elsewhere). The tensor may be rectangular.

Parameters

tensor T: 2-D tensor to fill in-place

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf the tensor is not 2-D

kaimingNormal

function kaimingNormal<T extends AnyTensor>(tensor: T, a?: number, mode?: "fan_in" | "fan_out", nonlinearity?: string): T;

Fill tensor using Kaiming normal initialization (He initialization).

Draws from N(0, std²) where std = gain / sqrt(fan). Recommended for ReLU activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
aoptional number: Negative slope of rectifier (used when nonlinearity is 'leaky_relu', default: 0)
modeoptional "fan_in" | "fan_out": 'fan_in' (default) or 'fan_out'
nonlinearityoptional string: Activation function name (default: 'leaky_relu')

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor, an unknown mode or an unknown nonlinearity

kaimingNormal_

function kaimingNormal_<T extends AnyTensor>(tensor: T, a?: number, mode?: "fan_in" | "fan_out", nonlinearity?: string): T;

Fill tensor using Kaiming normal initialization (He initialization).

Draws from N(0, std²) where std = gain / sqrt(fan). Recommended for ReLU activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
aoptional number: Negative slope of rectifier (used when nonlinearity is 'leaky_relu', default: 0)
modeoptional "fan_in" | "fan_out": 'fan_in' (default) or 'fan_out'
nonlinearityoptional string: Activation function name (default: 'leaky_relu')

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor, an unknown mode or an unknown nonlinearity

kaimingUniform

function kaimingUniform<T extends AnyTensor>(tensor: T, a?: number, mode?: "fan_in" | "fan_out", nonlinearity?: string): T;

Fill tensor using Kaiming uniform initialization (He initialization).

Draws from U(-bound, bound) where bound = gain * sqrt(3 / fan). Recommended for ReLU activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
aoptional number: Negative slope of rectifier (used when nonlinearity is 'leaky_relu', default: 0)
modeoptional "fan_in" | "fan_out": 'fan_in' (default) or 'fan_out'
nonlinearityoptional string: Activation function name (default: 'leaky_relu')

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor, an unknown mode or an unknown nonlinearity

kaimingUniform_

function kaimingUniform_<T extends AnyTensor>(tensor: T, a?: number, mode?: "fan_in" | "fan_out", nonlinearity?: string): T;

Fill tensor using Kaiming uniform initialization (He initialization).

Draws from U(-bound, bound) where bound = gain * sqrt(3 / fan). Recommended for ReLU activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
aoptional number: Negative slope of rectifier (used when nonlinearity is 'leaky_relu', default: 0)
modeoptional "fan_in" | "fan_out": 'fan_in' (default) or 'fan_out'
nonlinearityoptional string: Activation function name (default: 'leaky_relu')

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor, an unknown mode or an unknown nonlinearity

normal_

function normal_<T extends AnyTensor>(tensor: T, mean?: number, std?: number): T;

Fill tensor with values drawn from a normal distribution N(mean, std²).

Parameters

tensor T: Floating-point tensor to fill in-place
meanoptional number: Mean of the distribution (default: 0)
stdoptional number: Standard deviation, must be >= 0 (default: 1)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf mean is not finite or std is negative or not finite
DTypeErrorIf the tensor is not floating-point

ones

function ones<T extends AnyTensor>(tensor: T): T;

Fill tensor with ones.

Parameters

tensor T: Tensor to fill in-place

Returns

TThe modified tensor

ones_

function ones_<T extends AnyTensor>(tensor: T): T;

Fill tensor with ones.

Parameters

tensor T: Tensor to fill in-place

Returns

TThe modified tensor

orthogonal

function orthogonal<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor with a (semi-)orthogonal matrix.

A normal random matrix is orthonormalized with Gram-Schmidt (applied twice for accuracy). With more columns than rows the rows are orthonormal; with more rows than columns the columns are. Tensors with more than two dimensions are treated as a [shape[0], prod(shape[1:])] matrix, like PyTorch. The result is scaled by gain.

Parameters

tensor T: Floating-point tensor of at least 2 dimensions, filled in-place
gainoptional number: Scaling factor (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf the tensor has fewer than 2 dimensions or gain is not finite

orthogonal_

function orthogonal_<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor with a (semi-)orthogonal matrix.

Parameters

tensor T: Floating-point tensor of at least 2 dimensions, filled in-place
gainoptional number: Scaling factor (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf the tensor has fewer than 2 dimensions or gain is not finite

sparse_

function sparse_<T extends AnyTensor>(tensor: T, sparsity?: number, std?: number): T;

Fill a 2-D tensor as a sparse matrix with normally distributed non-zero entries.

In every column, ceil(sparsity * rows) randomly chosen entries are set to zero and the rest are drawn from N(0, std²), as in torch.nn.init.sparse_.

Parameters

tensor T: Floating-point 2-D tensor to fill in-place
sparsityoptional number: Fraction of elements in each column to be zero, in [0, 1] (default: 0.1)
stdoptional number: Standard deviation of the non-zero entries, must be >= 0 (default: 0.01)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf the tensor is not 2-D or an argument is out of range

trunc_normal_

function trunc_normal_<T extends AnyTensor>(tensor: T, mean?: number, std?: number, a?: number, b?: number): T;

Fill tensor with values drawn from a normal distribution N(mean, std²) truncated to [a, b] (mirrors torch.nn.init.trunc_normal_).

Uses rejection sampling, so the result is exactly the truncated normal.

Parameters

tensor T: Floating-point tensor to fill in-place
meanoptional number: Mean of the underlying normal (default: 0)
stdoptional number: Standard deviation of the underlying normal, must be >= 0 (default: 1)
aoptional number: Lower cut-off (default: -2)
boptional number: Upper cut-off (default: 2)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf a parameter is not finite, std < 0, a > b, or the interval lies so far in the tail that sampling would not finish
DTypeErrorIf the tensor is not floating-point

truncNormal_

function truncNormal_<T extends AnyTensor>(tensor: T, mean?: number, std?: number, a?: number, b?: number): T;

Fill tensor with values drawn from a normal distribution N(mean, std²) truncated to [a, b] (mirrors torch.nn.init.trunc_normal_).

Uses rejection sampling, so the result is exactly the truncated normal.

Parameters

tensor T: Floating-point tensor to fill in-place
meanoptional number: Mean of the underlying normal (default: 0)
stdoptional number: Standard deviation of the underlying normal, must be >= 0 (default: 1)
aoptional number: Lower cut-off (default: -2)
boptional number: Upper cut-off (default: 2)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf a parameter is not finite, std < 0, a > b, or the interval lies so far in the tail that sampling would not finish
DTypeErrorIf the tensor is not floating-point

uniform_

function uniform_<T extends AnyTensor>(tensor: T, low?: number, high?: number): T;

Fill tensor with values drawn from a uniform distribution U(low, high).

Parameters

tensor T: Floating-point tensor to fill in-place
lowoptional number: Lower bound (default: 0)
highoptional number: Upper bound (default: 1)

Returns

TThe modified tensor

Throws

InvalidParameterErrorIf a bound is not finite or low > high
DTypeErrorIf the tensor is not floating-point

xavierNormal

function xavierNormal<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor using Xavier normal initialization.

Draws from N(0, std²) where std = gain * sqrt(2 / (fan_in + fan_out)). Recommended for sigmoid/tanh activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
gainoptional number: Scaling factor, must be >= 0 (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor or an invalid gain

xavierNormal_

function xavierNormal_<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor using Xavier normal initialization.

Draws from N(0, std²) where std = gain * sqrt(2 / (fan_in + fan_out)). Recommended for sigmoid/tanh activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
gainoptional number: Scaling factor, must be >= 0 (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor or an invalid gain

xavierUniform

function xavierUniform<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor using Xavier uniform initialization.

Draws from U(-a, a) where a = gain * sqrt(6 / (fan_in + fan_out)). Recommended for sigmoid/tanh activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
gainoptional number: Scaling factor, must be >= 0 (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor or an invalid gain

xavierUniform_

function xavierUniform_<T extends AnyTensor>(tensor: T, gain?: number): T;

Fill tensor using Xavier uniform initialization.

Draws from U(-a, a) where a = gain * sqrt(6 / (fan_in + fan_out)). Recommended for sigmoid/tanh activations.

Parameters

tensor T: Floating-point tensor to fill in-place (at least 1-D)
gainoptional number: Scaling factor, must be >= 0 (default: 1.0)

Returns

TThe modified tensor

Throws

InvalidParameterErrorFor a scalar tensor or an invalid gain

zeros

function zeros<T extends AnyTensor>(tensor: T): T;

Fill tensor with zeros.

Parameters

tensor T: Tensor to fill in-place

Returns

TThe modified tensor

zeros_

function zeros_<T extends AnyTensor>(tensor: T): T;

Fill tensor with zeros.

Parameters

tensor T: Tensor to fill in-place

Returns

TThe modified tensor

binaryCrossEntropyLoss

function binaryCrossEntropyLoss(predictions: AnyTensor, targets: AnyTensor, reduction?: LossReduction): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor
reductionoptional LossReduction

Returns

AnyTensor

binaryCrossEntropyWithLogitsLoss

function binaryCrossEntropyWithLogitsLoss(input: AnyTensor, target: AnyTensor, options?: BinaryCrossEntropyWithLogitsOptions): number | Tensor | GradTensor;

Parameters

input AnyTensor
target AnyTensor
optionsoptional BinaryCrossEntropyWithLogitsOptions

Returns

number | Tensor | GradTensor

cosineEmbeddingLoss

function cosineEmbeddingLoss(x1: Tensor, x2: Tensor, y: Tensor, margin?: number, reduction?: LossReduction): Tensor;

Cosine Embedding Loss for similarity/contrastive learning.

Formula:

loss = 1 - cos(x1, x2)           if y == 1     = max(0, cos(x1, x2) - margin) if y == -1

The cosine is dot(x1, x2) / sqrt((|x1|^2 + 1e-12) * (|x2|^2 + 1e-12)), as in PyTorch.

Parameters

x1 Tensor: First input, shape (N, D) or (D,)
x2 Tensor: Second input, same shape as x1
y Tensor: Labels: 1 (similar) or -1 (dissimilar), one per row of x1
marginoptional number: Margin for dissimilar pairs (default: 0)
reductionoptional LossReduction: How to reduce the loss

Returns

Tensor

Throws

ShapeErrorIf x1 and x2 differ in shape, have more than 2 dimensions, or y has the wrong length
InvalidParameterErrorIf a label is neither 1 nor -1

crossEntropyLoss

function crossEntropyLoss(input: AnyTensor, target: AnyTensor, options?: CrossEntropyLossOptions): number | Tensor | GradTensor;

Parameters

input AnyTensor
target AnyTensor
optionsoptional CrossEntropyLossOptions

Returns

number | Tensor | GradTensor

ctcLoss

function ctcLoss(logProbs: Tensor, targets: Tensor, inputLengths: Tensor, targetLengths: Tensor, options?: CtcLossOptions): Tensor;

Connectionist Temporal Classification (CTC) Loss.

Details

CTC loss for sequence-to-sequence models where the alignment between input and output is unknown (e.g., speech recognition, OCR).

Algorithm: Forward (alpha) computation in log-space over an extended label sequence with interleaved blanks.

With the default "mean" reduction each sample's loss is divided by its target length (at least 1) before averaging over the batch, matching PyTorch's CTCLoss. A sample whose input is too short to emit its target has an infinite loss unless zeroInfinity is set. Target length 0 is allowed (the loss is then the negative log-probability of emitting only blanks).

Parameters

logProbs Tensor: Log-probabilities of shape (T, N, C) where T = input length (time steps), N = batch size, C = number of classes (including blank)
targets Tensor: Either the concatenated target sequences (1D, length at least the sum of all target lengths) or a padded matrix of shape (N, S). Values must be in [0, C) and must not equal blank.
inputLengths Tensor: Lengths of each input sequence, shape (N,)
targetLengths Tensor: Lengths of each target sequence, shape (N,)
optionsoptional CtcLossOptions: Configuration options

Returns

TensorLoss value

gaussianNLLLoss

function gaussianNLLLoss(input: Tensor, target: Tensor, variance: Tensor, options?: { full?: boolean; eps?: number; reduction?: LossReduction; }): Tensor;

Gaussian Negative Log Likelihood Loss.

Details

Formula:

loss = 0.5 * (log(var) + (input - target)^2 / var + log(2π))

Use Cases: - Probabilistic regression where the model predicts both mean and variance - Heteroscedastic regression (varying noise)

variance may have the same shape as input, the shape of input with the last dimension removed, or the shape of input with the last dimension set to 1 (one variance shared by all outputs of a sample). Variances below eps are clamped to eps.

Parameters

input Tensor: Predicted means
target Tensor: Target values
variance Tensor: Predicted variances (must be non-negative)
optionsoptional { full?: boolean; eps?: number; reduction?: LossReduction; }: Configuration options

Returns

TensorLoss value

Throws

InvalidParameterErrorIf variance contains a negative value
ShapeErrorIf variance has an incompatible shape

huberLoss

function huberLoss(predictions: AnyTensor, targets: AnyTensor, delta?: number, reduction?: LossReduction): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor
deltaoptional number
reductionoptional LossReduction

Returns

AnyTensor

klDivLoss

function klDivLoss(input: AnyTensor, target: AnyTensor, reduction?: LossReduction | "batchmean", logTarget?: boolean): AnyTensor;

Parameters

input AnyTensor
target AnyTensor
reductionoptional LossReduction | "batchmean"
logTargetoptional boolean

Returns

AnyTensor

maeLoss

function maeLoss(predictions: AnyTensor, targets: AnyTensor, reduction?: LossReduction): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor
reductionoptional LossReduction

Returns

AnyTensor

marginRankingLoss

function marginRankingLoss(x1: AnyTensor, x2: AnyTensor, y: AnyTensor, margin?: number, reduction?: LossReduction): AnyTensor;

Parameters

x1 AnyTensor
x2 AnyTensor
y AnyTensor
marginoptional number
reductionoptional LossReduction

Returns

AnyTensor

mseLoss

function mseLoss(predictions: AnyTensor, targets: AnyTensor, reduction?: LossReduction): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor
reductionoptional LossReduction

Returns

AnyTensor

nllLoss

function nllLoss(logProbs: AnyTensor, targets: AnyTensor, reductionOrOptions?: LossReduction | NllLossOptions): AnyTensor;

Parameters

logProbs AnyTensor
targets AnyTensor
reductionOrOptionsoptional LossReduction | NllLossOptions

Returns

AnyTensor

poissonNLLLoss

function poissonNLLLoss(input: Tensor, target: Tensor, options?: { logInput?: boolean; full?: boolean; eps?: number; reduction?: LossReduction; }): Tensor;

Poisson Negative Log Likelihood Loss.

Formula:

loss = exp(input) - target * input

When logInput=false:

loss = input - target * log(input + eps)

Use Cases: - Count data modeling (Poisson regression) - Event rate prediction

Parameters

input Tensor: Log of expected rate (or rate itself if logInput=false)
target Tensor: Target counts (non-negative)
optionsoptional { logInput?: boolean; full?: boolean; eps?: number; reduction?: LossReduction; }: Configuration options

Returns

TensorLoss value

rmseLoss

function rmseLoss(predictions: AnyTensor, targets: AnyTensor): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor

Returns

AnyTensor

smoothL1Loss

function smoothL1Loss(predictions: AnyTensor, targets: AnyTensor, beta?: number, reduction?: LossReduction): AnyTensor;

Parameters

predictions AnyTensor
targets AnyTensor
betaoptional number
reductionoptional LossReduction

Returns

AnyTensor

tripletMarginLoss

function tripletMarginLoss(anchor: AnyTensor, positive: AnyTensor, negative: AnyTensor, marginOrOptions?: number | TripletMarginLossOptions, reduction?: LossReduction): AnyTensor;

Parameters

anchor AnyTensor
positive AnyTensor
negative AnyTensor
marginOrOptionsoptional number | TripletMarginLossOptions
reductionoptional LossReduction

Returns

AnyTensor

Deprecated names

These older spellings still work in 1.x. New code should use the names above.

kaiming_normal_Prefer kaimingNormal_.
kaiming_uniform_Prefer kaimingUniform_.
xavier_normal_Prefer xavierNormal_.
xavier_uniform_Prefer xavierUniform_.

nn-losses.tsTypeScript

import { parameter, tensor } from "deepbox/ndarray";import { Linear, crossEntropyLoss, mseLoss, xavierUniform_, zeros_ } from "deepbox/nn"; const labels = tensor([0, 1]); // Plain tensors: crossEntropyLoss returns a number.console.log(crossEntropyLoss(tensor([[2, 0.5, 0.1], [0.1, 1.8, 0.1]]), labels)); // A GradTensor input gives a differentiable loss.const scores = parameter([[2, 0.5, 0.1], [0.1, 1.8, 0.1]]);const loss = crossEntropyLoss(scores, labels, { labelSmoothing: 0.1 });loss.backward();console.log(loss.item(), scores.grad?.shape); console.log(mseLoss(tensor([0.9, 0.1, 0.8]), tensor([1, 0, 1])).toString()); const layer = new Linear(4, 2);xavierUniform_(layer.getWeight());const bias = layer.getBias();if (bias) zeros_(bias);

Found a mistake? Open an issue.