deepbox/metrics

Classification Metrics

Accuracy, precision, recall, F-scores, ROC and precision-recall curves, confusion matrices and classification reports.

Evaluation

Overview

Labels are 1-D tensors of numbers, bool, int64 or strings. Mismatched lengths throw ShapeError, and NaN labels throw DataValidationError. rocAucScore and logLoss take scores or probabilities instead of labels.
precision, recall, f1Score, fbetaScore and jaccardScore accept a positional average or an options object { average, labels, zeroDivision, sampleWeight }. Without average, two classes use "binary" (positive class 1) and more than two use "weighted", where scikit-learn raises an error for multiclass input. zeroDivision defaults to 0, and to 1 for jaccardScore.
rocAucScore takes 1-D scores for binary problems. For multiclass problems pass an [n, nClasses] probability matrix and multiClass: "ovr" | "ovo". The defaults are "ovr" and average: "macro".
confusionMatrix returns a float64 tensor with true labels in rows and predicted labels in columns. It takes labels, normalize ("true", "pred" or "all") and sampleWeight. classificationReport returns a string.
accuracy, confusionMatrix and the averaged scores take sampleWeight, one weight per sample.

Types

AveragedMetricOptions

type

Options object accepted by precision, recall, f1Score, fbetaScore and jaccardScore in place of the positional average.

readonly average?: Average | null
Averaging strategy: "binary" (positive class 1), "micro", "macro", "weighted" (by support) or null for one value per class in label order.
readonly labels?: ReadonlyArray<number | string | bigint>
Classes to score, in the order they are reported for average: null.
readonly zeroDivision?: number | "warn"
Value returned when a score is undefined because its denominator is zero (a class that is never predicted for precision, has no true samples for recall, and so on).
readonly sampleWeight?: SampleWeightInput
One weight per sample; every count and support is a weighted sum.

ConfusionMatrixOptions

type

Options for confusionMatrix.

readonly labels?: ReadonlyArray<number | string | bigint>
Labels that index the rows and columns, in the order given.
readonly normalize?: "true" | "pred" | "all" | null
Normalize the counts over the true labels ("true", each row sums to 1), the predicted labels ("pred", each column sums to 1) or all samples ("all").
readonly sampleWeight?: SampleWeightInput
One weight per sample.

LogLossOptions

type

Options for logLoss.

type LogLossOptions = WeightedMetricOptions & {    readonly labels?: ReadonlyArray<number | string | bigint>;    readonly normalize?: boolean;};

RocAucOptions

type

Options for rocAucScore.

readonly multiClass?: "ovr" | "ovo"
How a 2-D score matrix is turned into one number: "ovr" (one class against the rest) or "ovo" (every pair of classes, Hand and Till 2001).
readonly average?: "micro" | "macro" | "weighted" | null
Averaging over the classes (or class pairs): "macro" (default), "weighted" (by class prevalence), "micro" ("ovr" only, all one-vs-rest problems pooled) or null for the per-class AUCs ("ovr" only).
readonly labels?: ReadonlyArray<number | string | bigint>
Class labels in the order of the columns of a 2-D yScore, in ascending order.
readonly sampleWeight?: SampleWeightInput
One weight per sample.

Functions

accuracy

function accuracy(yTrue: Tensor, yPred: Tensor, options?: WeightedMetricOptions): number;

Calculates the accuracy classification score.

Details

Accuracy is the fraction of predictions that match the true labels. It's the most intuitive performance measure but can be misleading for imbalanced datasets.

Formula: accuracy = (correct predictions) / (total predictions)

Time Complexity: O(n) where n is the number of samples Space Complexity: O(n) for the dense copies of the label vectors

With options.sampleWeight each sample counts with its weight: Σ w_i * [y_i == p_i] / Σ w_i.

int64 labels may be compared with integer-valued numeric labels.

Parameters

yTrue Tensor: Ground truth (correct) target values
yPred Tensor: Estimated targets as returned by a classifier
optionsoptional WeightedMetricOptions: Optional sampleWeight, one weight per sample

Returns

numberAccuracy score in range [0, 1], where 1 is perfect accuracy. Returns 0 for empty input.

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors, or sampleWeight has the wrong length
DTypeErrorIf yTrue and yPred use incompatible label types
InvalidParameterErrorIf sampleWeight sums to zero
DataValidationErrorIf numeric labels contain NaN or infinite values

ExampleTypeScript

import { accuracy } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const acc = accuracy(yTrue, yPred); // 0.8 (4 out of 5 correct)accuracy(yTrue, yPred, { sampleWeight: [1, 1, 3, 1, 1] }); // 0.5714285714285714

averagePrecisionScore

function averagePrecisionScore(yTrue: Tensor, yScore: Tensor): number;

Average precision score.

Computes the average precision (AP) from prediction scores. AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.

Range: [0, 1], where 1 is perfect.

Time Complexity: O(n log n) due to sorting Space Complexity: O(n)

Parameters

yTrue Tensor: Ground truth binary labels (0 or 1)
yScore Tensor: Target scores (higher score = more likely positive class)

Returns

numberAverage precision score in range [0, 1]. Returns 0 for empty input or when yTrue has no positive sample.

Throws

ShapeErrorIf yTrue and yScore have different sizes or are not 1D/column vectors
DTypeErrorIf inputs are string tensors
InvalidParameterErrorIf yTrue contains non-binary values
DataValidationErrorIf inputs contain NaN or infinite values

ExampleTypeScript

import { averagePrecisionScore } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 0, 1, 1]);const yScore = tensor([0.1, 0.4, 0.35, 0.8]);const ap = averagePrecisionScore(yTrue, yScore); // 0.8333...

balancedAccuracyScore

function balancedAccuracyScore(yTrue: Tensor, yPred: Tensor): number;

Balanced accuracy score.

Computes the balanced accuracy, which is the macro-averaged recall over the classes that occur in yTrue. It is useful for imbalanced datasets where regular accuracy can be misleading.

Formula: balanced_accuracy = (1/n_classes) * Σ(recall_per_class)

Range: [0, 1], where 1 is perfect.

Time Complexity: O(n + k) Space Complexity: O(n + k)

Parameters

yTrue Tensor: Ground truth labels (numeric, int64 or string)
yPred Tensor: Predicted labels (same label type as yTrue)

Returns

numberBalanced accuracy score in range [0, 1]. Returns 0 for empty input.

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors
DTypeErrorIf yTrue and yPred use incompatible label types
DataValidationErrorIf numeric labels contain NaN or infinite values

ExampleTypeScript

import { balancedAccuracyScore } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 0, 0, 0, 1]); // Imbalancedconst yPred = tensor([0, 0, 0, 0, 0]); // Predicts all 0const bacc = balancedAccuracyScore(yTrue, yPred); // 0.5 (not 0.8!)

classificationReport

function classificationReport(yTrue: Tensor, yPred: Tensor): string;

Generates a text classification report showing main classification metrics.

The report lists precision, recall, F1-score and support for each class, followed by accuracy and the macro and weighted averages. Only binary labels (0 or 1) are supported. int64 labels may be paired with numeric labels.

Time Complexity: O(n) Space Complexity: O(n)

Parameters

yTrue Tensor: Ground truth (correct) binary target values (0 or 1)
yPred Tensor: Estimated binary targets as returned by a classifier (0 or 1)

Returns

stringFormatted string report with per-class and aggregate classification metrics

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors
DataValidationErrorIf labels contain NaN or infinite values
InvalidParameterErrorIf labels are strings or are not binary (0 or 1)

ExampleTypeScript

import { classificationReport } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);console.log(classificationReport(yTrue, yPred));// Classification Report:// Class         Precision   Recall      F1-Score    Support// ---------------------------------------------------------// 0             0.6667      1.0000      0.8000      2// 1             1.0000      0.6667      0.8000      3//// Accuracy                              0.8000      5// Macro Avg     0.8333      0.8333      0.8000      5// Weighted Avg  0.8667      0.8000      0.8000      5

cohenKappaScore

function cohenKappaScore(yTrue: Tensor, yPred: Tensor, weights?: "linear" | "quadratic" | null): number;

Cohen's kappa score.

Details

Computes Cohen's kappa, a statistic that measures inter-annotator agreement. Unlike plain percent agreement it corrects for the agreement expected by chance.

Formula: kappa = (p_o - p_e) / (1 - p_e) - p_o: observed agreement - p_e: expected agreement by chance

With weights the disagreement between classes i and j (in ascending label order) is weighted by |i - j| ("linear") or (i - j)² ("quadratic"), which suits ordinal labels.

Range: [-1, 1], where 1 is perfect agreement, 0 is chance, <0 is worse than chance. When both vectors hold the same single class the score is 1.

Time Complexity: O(n) unweighted, O(n + k²) weighted Space Complexity: O(n + k) unweighted, O(n + k²) weighted

Parameters

yTrue Tensor: Ground truth labels (numeric, int64 or string)
yPred Tensor: Predicted labels (same label type as yTrue)
weightsoptional "linear" | "quadratic" | null: Optional disagreement weighting: "linear", "quadratic" or null (default)

Returns

numberKappa score in range [-1, 1]

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors
DTypeErrorIf yTrue and yPred use incompatible label types
DataValidationErrorIf numeric labels contain NaN or infinite values
InvalidParameterErrorIf weights is not 'linear', 'quadratic' or null

ExampleTypeScript

import { cohenKappaScore } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const kappa = cohenKappaScore(yTrue, yPred); // ~0.615

confusionMatrix

function confusionMatrix(yTrue: Tensor, yPred: Tensor, options?: ConfusionMatrixOptions): Tensor;

Computes the confusion matrix to evaluate classification accuracy.

Details

A confusion matrix is a table showing the counts of correct and incorrect predictions broken down by each class. Rows represent true labels, columns represent predicted labels. Classes are listed in ascending order (strings by code unit, like NumPy), unless options.labels fixes the order.

Time Complexity: O(n + k²) where n is number of samples, k is number of classes Space Complexity: O(n + k²)

Parameters

yTrue Tensor: Ground truth (correct) target values
yPred Tensor: Estimated targets as returned by a classifier
optionsoptional ConfusionMatrixOptions: Optional labels (row/column order and subset), normalize and sampleWeight

Returns

TensorConfusion matrix as a float64 tensor of shape [n_classes, n_classes] int64 labels may be paired with integer-valued numeric labels (the int64 values must be safe integers); they are then compared as numbers.

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors, or sampleWeight has the wrong length
DTypeErrorIf yTrue and yPred use incompatible label types, or labels does not match the label type
DataValidationErrorIf numeric labels contain NaN or infinite values
InvalidParameterErrorIf labels is empty or has duplicates, or normalize is invalid

ExampleTypeScript

import { confusionMatrix } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const cm = confusionMatrix(yTrue, yPred);// [[2, 0],//  [1, 2]] const rowRates = confusionMatrix(yTrue, yPred, { normalize: "true" });// [[1, 0],//  [1/3, 2/3]] const weighted = confusionMatrix(yTrue, yPred, { sampleWeight: [1, 1, 2, 1, 1] });// [[2, 0],//  [2, 2]]

f1Score

function f1Score(yTrue: Tensor, yPred: Tensor, average?: Average | null | AveragedMetricOptions): number | number[];

Parameters

yTrue Tensor
yPred Tensor
averageoptional Average | null | AveragedMetricOptions

Returns

number | number[]

fbetaScore

function fbetaScore(yTrue: Tensor, yPred: Tensor, beta: number, average?: Average | null | AveragedMetricOptions): number | number[];

Parameters

yTrue Tensor
yPred Tensor
beta number
averageoptional Average | null | AveragedMetricOptions

Returns

number | number[]

hammingLoss

function hammingLoss(yTrue: Tensor, yPred: Tensor): number;

Hamming loss.

Computes the fraction of labels that are incorrectly predicted. For single-label classification this equals 1 - accuracy.

Formula: hamming_loss = (incorrect predictions) / (total predictions)

Range: [0, 1], where 0 is perfect.

Time Complexity: O(n) Space Complexity: O(n)

Parameters

yTrue Tensor: Ground truth target values
yPred Tensor: Estimated targets as returned by a classifier

Returns

numberHamming loss in range [0, 1]. Returns 0 for empty input.

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors
DTypeErrorIf yTrue and yPred use incompatible label types
DataValidationErrorIf numeric labels contain NaN or infinite values

ExampleTypeScript

import { hammingLoss } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const loss = hammingLoss(yTrue, yPred); // 0.2

jaccardScore

function jaccardScore(yTrue: Tensor, yPred: Tensor, average?: Average | null | AveragedMetricOptions): number | number[];

Parameters

yTrue Tensor
yPred Tensor
averageoptional Average | null | AveragedMetricOptions

Returns

number | number[]

logLoss

function logLoss(yTrue: Tensor, yPred: Tensor, options?: LogLossOptions): number;

Log loss (logistic loss, cross-entropy loss).

Details

Measures the performance of a classification model where the prediction is a probability value between 0 and 1. Lower log loss indicates better predictions.

Binary: yPred is a 1-D vector of positive-class probabilities and yTrue holds 0/1 labels. Loss: -1/n * Σ(y * log(p) + (1 - y) * log(1 - p)).

Multiclass: yPred is a matrix of shape [n_samples, n_classes] of class probabilities whose columns follow the sorted class labels (a two-column matrix is the binary case). Loss: -1/n * Σ log(p[i, y_i]). Rows are not renormalized. options.labels names the columns when yTrue does not contain every class. Labels may be numbers, strings or int64 values.

Edge Cases: - Predictions are clipped to [eps, 1 - eps] to avoid log(0), where eps is the machine epsilon of yPred's dtype (2.2e-16 for float64, 1.2e-7 for float32), as in scikit-learn - Returns 0 for empty inputs

Parameters

yTrue Tensor: Ground truth labels
yPred Tensor: Predicted probabilities: 1-D positive-class probabilities, or a [n_samples, n_classes] matrix (each value in [0, 1])
optionsoptional LogLossOptions: Optional labels, sampleWeight and normalize (see LogLossOptions)

Returns

numberLog loss value (lower is better, 0 is perfect)

Throws

ShapeErrorIf yTrue and yPred have different sizes, are not 1D/column vectors (or a 2-D probability matrix), or sampleWeight has the wrong length
DTypeErrorIf yPred is a string tensor, or yTrue is a string tensor for a 1-D yPred
InvalidParameterErrorIf a 1-D yTrue is not binary or yPred is outside [0, 1], the number of classes does not match the columns of yPred, labels is invalid, or sampleWeight sums to zero
DataValidationErrorIf inputs contain NaN or infinite values

ExampleTypeScript

import { logLoss } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0]);const yPred = tensor([0.1, 0.9, 0.8, 0.2], { dtype: "float64" });const loss = logLoss(yTrue, yPred); // 0.164252... // Multiclass probabilities, one column per classconst labels = tensor([0, 1, 2]);const proba = tensor(  [    [0.7, 0.2, 0.1],    [0.2, 0.6, 0.2],    [0.1, 0.3, 0.6],  ],  { dtype: "float64" });logLoss(labels, proba); // 0.4594420638235713

matthewsCorrcoef

function matthewsCorrcoef(yTrue: Tensor, yPred: Tensor, options?: WeightedMetricOptions): number;

Matthews correlation coefficient (MCC).

Details

Computes the Matthews correlation coefficient, a balanced measure that can be used even if the classes are of very different sizes. MCC is considered one of the best metrics for classification.

Binary formula: MCC = (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))

Multiclass: the generalization from the confusion matrix (Gorodkin 2004), identical to scikit-learn's matthews_corrcoef: MCC = (c*s - Σ t_k p_k) / sqrt((s² - Σ p_k²)(s² - Σ t_k²)) with c the number of correct predictions, s the number of samples and t_k, p_k the true and predicted counts of class k. Labels may be numbers, strings or int64 values.

Returns 0 when the coefficient is undefined (a denominator term is zero, for example when only one class occurs).

Range: [-1, 1], where 1 is perfect, 0 is random, -1 is inverse.

Time Complexity: O(n + k) Space Complexity: O(n + k)

Parameters

yTrue Tensor: Ground truth labels
yPred Tensor: Predicted labels (same label type as yTrue)
optionsoptional WeightedMetricOptions: Optional sampleWeight, one weight per sample

Returns

numberMCC score in range [-1, 1]

Throws

ShapeErrorIf yTrue and yPred have different sizes or are not 1D/column vectors, or sampleWeight has the wrong length
DTypeErrorIf yTrue and yPred use incompatible label types
DataValidationErrorIf numeric labels contain NaN or infinite values

ExampleTypeScript

import { matthewsCorrcoef } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const mcc = matthewsCorrcoef(yTrue, yPred); // ~0.667 const multiTrue = tensor([0, 1, 2, 2, 1, 0]);const multiPred = tensor([0, 1, 2, 1, 1, 2]);matthewsCorrcoef(multiTrue, multiPred); // 0.5222329678670935

precision

function precision(yTrue: Tensor, yPred: Tensor, average?: Average | null | AveragedMetricOptions): number | number[];

Parameters

yTrue Tensor
yPred Tensor
averageoptional Average | null | AveragedMetricOptions

Returns

number | number[]

precisionRecallCurve

function precisionRecallCurve(yTrue: Tensor, yScore: Tensor): [ Tensor, Tensor, Tensor ];

Precision-Recall curve.

Details

Computes precision-recall pairs for different probability thresholds. Useful for evaluating classifiers on imbalanced datasets where ROC curves may be overly optimistic.

Returns: [precision, recall, thresholds] as a tuple of tensors - precision: Precision values at each threshold - recall: Recall values at each threshold - thresholds: Decision thresholds (in descending order). The first entry is Infinity and belongs to the starting point (precision 1, recall 0).

Unlike scikit-learn, the three arrays have the same length and are ordered by descending threshold (increasing recall).

Time Complexity: O(n log n) due to sorting Space Complexity: O(n)

Parameters

yTrue Tensor: Ground truth binary labels (0 or 1)
yScore Tensor: Target scores (higher score = more likely positive class)

Returns

[ Tensor, Tensor, Tensor ]Tuple of [precision, recall, thresholds] float64 tensors; empty tensors when yTrue has no positive sample

Throws

ShapeErrorIf yTrue and yScore have different sizes or are not 1D/column vectors
DTypeErrorIf inputs are string tensors
InvalidParameterErrorIf yTrue contains non-binary values
DataValidationErrorIf inputs contain NaN or infinite values

ExampleTypeScript

import { precisionRecallCurve } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 0, 1, 1]);const yScore = tensor([0.1, 0.4, 0.35, 0.8]);const [prec, rec, thresh] = precisionRecallCurve(yTrue, yScore);

recall

function recall(yTrue: Tensor, yPred: Tensor, average?: Average | null | AveragedMetricOptions): number | number[];

Parameters

yTrue Tensor
yPred Tensor
averageoptional Average | null | AveragedMetricOptions

Returns

number | number[]

rocAucScore

function rocAucScore(yTrue: Tensor, yScore: Tensor, options?: RocAucOptions): number | number[];

Parameters

yTrue Tensor
yScore Tensor
optionsoptional RocAucOptions

Returns

number | number[]

rocCurve

function rocCurve(yTrue: Tensor, yScore: Tensor): [ Tensor, Tensor, Tensor ];

ROC curve data.

Details

Computes Receiver Operating Characteristic (ROC) curve for binary classification. The ROC curve shows the trade-off between true positive rate and false positive rate at various threshold settings.

Returns: [fpr, tpr, thresholds] - fpr: False positive rates - tpr: True positive rates - thresholds: Decision thresholds (in descending order). The first entry is Infinity and belongs to the starting point (0, 0).

Edge Cases: - Returns empty tensors if yTrue contains only one class - Handles tied scores by grouping them at the same threshold - No intermediate points are dropped (every distinct score is a threshold)

Parameters

yTrue Tensor: Ground truth binary labels (must be 0 or 1)
yScore Tensor: Target scores (higher score = more likely positive class)

Returns

[ Tensor, Tensor, Tensor ]Tuple of [fpr, tpr, thresholds] float64 tensors

Throws

ShapeErrorIf yTrue and yScore have different sizes or are not 1D/column vectors
DTypeErrorIf inputs are string tensors
InvalidParameterErrorIf yTrue contains non-binary values
DataValidationErrorIf inputs contain NaN or infinite values

ExampleTypeScript

import { rocCurve } from 'deepbox/metrics';import { tensor } from 'deepbox/ndarray'; const yTrue = tensor([0, 0, 1, 1]);const yScore = tensor([0.1, 0.4, 0.35, 0.8]);const [fpr, tpr, thresholds] = rocCurve(yTrue, yScore);// fpr: [0, 0, 0.5, 0.5, 1], tpr: [0, 0.5, 0.5, 1, 1]// thresholds: [Infinity, 0.8, 0.4, 0.35, 0.1]

metrics-classification.tsTypeScript

import {  accuracy,  classificationReport,  confusionMatrix,  f1Score,  rocAucScore,} from "deepbox/metrics";import { tensor } from "deepbox/ndarray"; const yTrue = tensor([0, 1, 1, 0, 1]);const yPred = tensor([0, 1, 0, 0, 1]);const yScore = tensor([0.1, 0.8, 0.4, 0.2, 0.9]); console.log(accuracy(yTrue, yPred));console.log(f1Score(yTrue, yPred), f1Score(yTrue, yPred, "macro"));console.log(f1Score(yTrue, yPred, { average: "weighted", sampleWeight: [1, 1, 2, 1, 1] }));console.log(rocAucScore(yTrue, yScore));console.log(confusionMatrix(yTrue, yPred, { normalize: "true" }).toString());console.log(classificationReport(yTrue, yPred));

Found a mistake? Open an issue.