Skip to content

deepbox/ndarray

Activation Functions

Element-wise activation functions on tensors, such as relu, sigmoid, softmax and gelu.

  • Activations

Overview

  • These functions take a Tensor. The common ones are also GradTensor methods with the same names (x.relu(), x.softmax(-1)), which record gradients. For the others, or to put an activation inside a model, use the layers in Activation Layers.
  • Float input keeps its float dtype, computed in float64 and rounded back. Integer input gives float32, except relu, relu6 and hardtanh, which keep integer dtypes.
  • gelu uses the tanh approximation by default, while PyTorch defaults to the exact form. Pass { approximate: "none" } for the exact erf version.
  • softmax and logSoftmax default to the last axis. 1.5.0 added relu6, selu, celu, softsign, hardsigmoid, hardswish, logSigmoid, hardshrink and softshrink.

Types

2

GeluApproximation

type

Which GELU formula to use: the tanh approximation, or the exact erf form ("none").

type GeluApproximation = "tanh" | "none";

GeluOptions

type

Options of gelu.

  • approximate?: GeluApproximation

    "tanh" (default) for the tanh approximation, "none" for the exact erf form.

Functions

21

celu

function celu(t: Tensor, alpha?: number): Tensor;

Continuously differentiable Exponential Linear Unit activation.

Applies element-wise: celu(x) = max(0, x) + min(0, alpha * (exp(x / alpha) - 1)).

Parameters

t
Tensor
Input tensor
alphaoptional
number
Scale of the negative branch; must be finite and non-zero (default: 1.0)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf alpha is zero or not a finite number
  • DTypeErrorFor string tensors
Example
import { celu, tensor } from 'deepbox/ndarray'; celu(tensor([-1, 0, 1]), 2);  // [-0.7869..., 0, 1]

elu

function elu(t: Tensor, alpha?: number): Tensor;

Exponential Linear Unit activation.

Applies element-wise: - elu(x) = x if x > 0 - elu(x) = alpha * (exp(x) - 1) if x <= 0

The negative branch uses expm1, so values near zero keep full precision.

Parameters

t
Tensor
Input tensor
alphaoptional
number
Scale for negative values (default: 1.0)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf alpha is not a finite number
  • DTypeErrorFor string tensors
Example
import { elu, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = elu(x);  // [-0.632..., 0, 1]

gelu

function gelu(t: Tensor, options?: GeluApproximation | GeluOptions): Tensor;

Gaussian Error Linear Unit activation.

Details

With approximate: "tanh" (the default) this applies element-wise gelu(x) ~ 0.5 x (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))), the approximation from Hendrycks & Gimpel (2016). It matches PyTorch's gelu(approximate="tanh"), the device kernels and the autograd gradient. With approximate: "none" it is the exact form x * Phi(x) = 0.5 x (1 + erf(x / sqrt(2))), which is what PyTorch's gelu and nn.GELU() compute by default. The two differ by up to about 5e-4. gelu(-Infinity) is 0.

Tensors on a kernel device run the tanh form as a single kernel and the exact form from the erf kernel.

Parameters

t
Tensor
Input tensor
optionsoptional
GeluApproximation | GeluOptions
{ approximate?: "tanh" | "none" }. A bare "tanh" or "none" string is also accepted, for backward compatibility

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf approximate is neither "tanh" nor "none"
  • DTypeErrorFor string tensors
Example
import { gelu, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = gelu(x);                          // [-0.158..., 0, 0.841...]const exact = gelu(x, { approximate: "none" });  // [-0.1586..., 0, 0.8413...]

hardshrink

function hardshrink(t: Tensor, lambd?: number): Tensor;

Hard shrinkage: x where |x| > lambd, otherwise 0.

Values exactly at +-lambd become 0. NaN propagates.

Parameters

t
Tensor
Input tensor
lambdoptional
number
Threshold, must be non-negative (default: 0.5)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf lambd is negative or NaN
  • DTypeErrorFor string tensors
Example
import { hardshrink, tensor } from 'deepbox/ndarray'; hardshrink(tensor([-2, -0.3, 0.3, 2]));  // [-2, 0, 0, 2]

hardsigmoid

function hardsigmoid(t: Tensor): Tensor;

Hard sigmoid activation: relu6(x + 3) / 6.

Piecewise linear: 0 for x <= -3, 1 for x >= 3 and x / 6 + 1/2 in between.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { hardsigmoid, tensor } from 'deepbox/ndarray'; hardsigmoid(tensor([-4, 0, 1.5, 4]));  // [0, 0.5, 0.75, 1]

hardswish

function hardswish(t: Tensor): Tensor;

Hard swish activation: x * relu6(x + 3) / 6.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { hardswish, tensor } from 'deepbox/ndarray'; hardswish(tensor([-4, 0, 1, 4]));  // [0, 0, 0.666..., 4]

hardtanh

function hardtanh(t: Tensor, minVal?: number, maxVal?: number): Tensor;

Applies the HardTanh activation function element-wise.

HardTanh(x) = max(minVal, min(maxVal, x)). This is a clip, so the input dtype is kept (see clip for integer-tensor rules).

Parameters

t
Tensor
Input tensor
minValoptional
number
Lower bound (default: -1)
maxValoptional
number
Upper bound (default: 1)

Returns

TensorTensor of the same shape

Throws

  • InvalidParameterErrorIf minVal > maxVal
  • DTypeErrorFor string tensors
Example
import { hardtanh, tensor } from 'deepbox/ndarray'; hardtanh(tensor([-2, 0.5, 3]));  // [-1, 0.5, 1]

leakyRelu

function leakyRelu(t: Tensor, alpha?: number): Tensor;

Leaky ReLU activation.

Applies element-wise: leaky_relu(x) = x for x > 0, alpha * x otherwise. NaN propagates.

Parameters

t
Tensor
Input tensor
alphaoptional
number
Slope for negative values (default: 0.01)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf alpha is not a finite number
  • DTypeErrorFor string tensors
Example
import { leakyRelu, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = leakyRelu(x, 0.1);  // [-0.1, 0, 1]

logSigmoid

function logSigmoid(t: Tensor): Tensor;

Log-sigmoid activation: log(1 / (1 + exp(-x))).

Computed as min(x, 0) - log1p(exp(-|x|)), so it neither overflows for very negative x nor loses precision for large positive x.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { logSigmoid, tensor } from 'deepbox/ndarray'; logSigmoid(tensor([-1, 0, 1]));  // [-1.3132..., -0.6931..., -0.3132...]

logSoftmax

function logSoftmax(t: Tensor, axis?: Axis): Tensor;

Log-Softmax activation function.

Computes log(softmax(x)) in a numerically stable way using log_softmax(x) = x - max(x) - log(sum(exp(x - max(x)))).

Properties: - More accurate than computing log(softmax(x)) directly (no log of a value that underflowed to 0) - Output values are log probabilities (<= 0; exp of them sums to 1) - Time O(n), extra memory O(1)

Parameters

t
Tensor
Input tensor of any dimensionality
axisoptional
Axis
Axis along which to compute log-softmax (default: -1, the last axis)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf axis is out of range
  • DTypeErrorFor string tensors
Example
import { logSoftmax, tensor } from 'deepbox/ndarray'; const x = tensor([[1, 2, 3]]);const result = logSoftmax(x, 1); const x3d = tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]]);const result3d = logSoftmax(x3d, -1);  // Log-softmax along last axis

mish

function mish(t: Tensor): Tensor;

Mish activation function.

Applies element-wise: mish(x) = x * tanh(softplus(x)) where softplus(x) = log(1 + exp(x)). The limit at -Infinity is 0.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { mish, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = mish(x);  // [-0.303..., 0, 0.865...]

relu

function relu(t: Tensor): Tensor;

Rectified Linear Unit activation.

Applies element-wise: relu(x) = max(0, x). NaN propagates.

Properties: - Output range: [0, ∞) - Non-linear but simple - Can suffer from dying ReLU problem

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape and dtype (bool input gives int32)

Throws

  • DTypeErrorFor string tensors
Example
import { relu, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = relu(x);  // [0, 0, 1]

relu6

function relu6(t: Tensor): Tensor;

ReLU6 activation: relu capped at 6.

Applies element-wise: relu6(x) = min(max(0, x), 6). NaN propagates.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape and dtype (bool input gives int32)

Throws

  • DTypeErrorFor string tensors
Example
import { relu6, tensor } from 'deepbox/ndarray'; relu6(tensor([-2, 3, 8]));  // [0, 3, 6]

selu

function selu(t: Tensor): Tensor;

Scaled Exponential Linear Unit activation.

Applies element-wise: selu(x) = scale * (x if x > 0, else alpha * (exp(x) - 1)) with alpha = 1.6732632423543772 and scale = 1.0507009873554805 (PyTorch's constants).

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { selu, tensor } from 'deepbox/ndarray'; selu(tensor([-1, 0, 1]));  // [-1.1113..., 0, 1.0507...]

sigmoid

function sigmoid(t: Tensor): Tensor;

Sigmoid activation function.

Applies element-wise: sigmoid(x) = 1 / (1 + exp(-x)), evaluated so that large negative inputs do not lose the (denormal) tail.

Properties: - Output range: (0, 1) - Smooth gradient - Can suffer from vanishing gradients

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { sigmoid, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = sigmoid(x);  // [0.268..., 0.5, 0.731...]

softmax

function softmax(t: Tensor, axis?: Axis): Tensor;

Softmax activation function.

Details

Normalizes input to a probability distribution along axis: softmax(x)_i = exp(x_i - max(x)) / sum(exp(x - max(x))).

Subtracting the maximum along the axis before exponentiating keeps large inputs from overflowing. A slice that contains NaN, or consists only of infinities, yields NaN for that slice (as in PyTorch).

Properties: - Output sums to 1 along the specified axis - Output values in [0, 1] - Supports tensors of any dimensionality, including 0-d (result 1)

Parameters

t
Tensor
Input tensor of any dimensionality
axisoptional
Axis
Axis along which to compute softmax (default: -1, the last axis)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf axis is out of range
  • DTypeErrorFor string tensors
Example
import { softmax, tensor } from 'deepbox/ndarray'; const x = tensor([[1, 2, 3], [1, 2, 3]]);const result = softmax(x, 1);  // Each row sums to 1 const x3d = tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]]);const result3d = softmax(x3d, -1);  // Softmax along last axis

softplus

function softplus(t: Tensor): Tensor;

Softplus activation function.

Smooth approximation of ReLU: softplus(x) = log(1 + exp(x)), computed without overflow for large x or cancellation for very negative x.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32; device tensors keep their dtype)

Throws

  • DTypeErrorFor string tensors
Example
import { softplus, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = softplus(x);  // [0.313..., 0.693..., 1.313...]

softshrink

function softshrink(t: Tensor, lambd?: number): Tensor;

Soft shrinkage: x - lambd for x > lambd, x + lambd for x < -lambd, otherwise 0.

NaN propagates.

Parameters

t
Tensor
Input tensor
lambdoptional
number
Threshold, must be non-negative (default: 0.5)

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • InvalidParameterErrorIf lambd is negative or NaN
  • DTypeErrorFor string tensors
Example
import { softshrink, tensor } from 'deepbox/ndarray'; softshrink(tensor([-2, -0.3, 0.3, 2]));  // [-1.5, 0, 0, 1.5]

softsign

function softsign(t: Tensor): Tensor;

Softsign activation: x / (1 + |x|).

Output range is (-1, 1). Infinite inputs give NaN, as in PyTorch.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { softsign, tensor } from 'deepbox/ndarray'; softsign(tensor([-2, 0, 2]));  // [-0.666..., 0, 0.666...]

swish

function swish(t: Tensor): Tensor;

Swish activation function (also known as SiLU).

Applies element-wise: swish(x) = x * sigmoid(x). The limit at -Infinity is 0.

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { swish, tensor } from 'deepbox/ndarray'; const x = tensor([-1, 0, 1]);const result = swish(x);  // [-0.268..., 0, 0.731...]

tanhshrink

function tanhshrink(t: Tensor): Tensor;

Applies the Tanhshrink activation element-wise.

Tanhshrink(x) = x - tanh(x). Small inputs use a Taylor series because the direct subtraction cancels almost every digit (the true value is about x³ / 3).

Parameters

t
Tensor
Input tensor

Returns

TensorTensor of the same shape (float dtypes are kept, integer and bool input gives float32)

Throws

  • DTypeErrorFor string tensors
Example
import { tanhshrink, tensor } from 'deepbox/ndarray'; tanhshrink(tensor([-1, 0, 1]));  // [-0.238..., 0, 0.238...]
ndarray-activations.ts
import { gelu, logSoftmax, relu, sigmoid, softmax, tensor } from "deepbox/ndarray"; const logits = tensor([[1.2, -0.4, 0.3], [0.1, 0.2, 0.3]]);console.log(relu(logits).toString());console.log(sigmoid(logits).toString());console.log(softmax(logits).toString()); // over the last axisconsole.log(softmax(logits, 0).toString()); // over the first axisconsole.log(logSoftmax(logits).toString());console.log(gelu(logits).toString()); // tanh approximationconsole.log(gelu(logits, { approximate: "none" }).toString()); // exact, as in PyTorch

Found a mistake? Open an issue.