Skip to content

deepbox/datasets

DataLoader, Samplers & Transforms

Mini-batch loading, samplers, dataset subsets and splits, and streaming datasets for data that does not fit in memory.

  • Input pipelines

Overview

  • new DataLoader(X, y, { batchSize, shuffle, seed }) yields [xBatch, yBatch] pairs. Each batch is a copy of the selected rows. shuffle defaults to false, and loader.length is the number of batches. You can loop over the same loader once per epoch.
  • With a seed, every pass uses the same order, unless you set reshuffleEachIteration: true, which gives a different but reproducible order in every epoch. Without a seed, a shuffling loader uses a new order on every pass. sampler sets a custom order and excludes shuffle: true.
  • randomSplit(dataset, lengths, seed?) takes a dataset object such as loadIris(). lengths are counts that add up to the dataset size, or fractions that add up to 1. It returns Subset objects with data and target.
  • For data that does not fit in memory, build a StreamingDataset with iterableDataset or asyncIterableDataset and pass it to DataLoader with shuffleBufferSize and prefetch. Trainer.fitAsync consumes asynchronous batches.

Types

4

CollateFn

type

Function that post-processes a mini-batch.

An in-memory DataLoader gathers each batch itself and calls the collate function with a single-element list holding that batch ([[xBatch]] or [[xBatch, yBatch]]); the returned value is what the loader yields.

type CollateFn<T = [    Tensor] | [    Tensor,    Tensor]> = (batch: T[]) => T;

DataLoaderOptions

type

Configuration options for DataLoader.

  • batchSize?: number

    Samples per batch (positive integer).

  • shuffle?: boolean

    Reshuffle the sample order at the start of every iteration.

  • dropLast?: boolean

    Drop a trailing batch that has fewer than batchSize samples.

  • seed?: number

    Seed for the shuffle.

  • reshuffleEachIteration?: boolean

    With a seed, continue one seeded random stream across iterations, so every epoch has a different order while the sequence of epochs stays reproducible (the first epoch matches the default behavior).

  • sampler?: Sampler

    Custom index order.

  • collateFn?: CollateFn<any>

    Post-process every batch, see CollateFn.

StreamingDataLoaderOptions

type

Configuration options for a DataLoader constructed over a StreamingDataset (out-of-core / lazy source).

  • batchSize?: number

    Samples per batch.

  • shuffleBufferSize?: number

    Window shuffle-buffer size.

  • prefetch?: number

    Read-ahead depth.

  • dropLast?: boolean

    Drop a trailing short batch.

  • seed?: number

    Seed for deterministic shuffle-buffer ordering.

  • reshuffleEachIteration?: boolean

    With a seed, continue one seeded random stream across iterations, so every pass has a different order while the sequence of passes stays reproducible.

  • collateFn?: (samples: any[]) => Batch

    Merge raw samples into a batch.

Sampler

interface

Sampler interface: produces an iterable of integer indices.

  • readonly length: number

    Total number of indices this sampler will produce.

  • [Symbol.iterator](): Iterator<number>

    Return an iterator of sample indices.

Classes

5

DataLoader

extends Tensor | undefined = undefined>

Data loader for batching and shuffling datasets.

Provides efficient iteration over datasets with support for batching, shuffling, and deterministic reproducibility.

Example
import { DataLoader } from 'deepbox/datasets';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1, 2], [3, 4], [5, 6], [7, 8]]);const y = tensor([0, 1, 0, 1]); // Training loop with shufflingconst loader = new DataLoader(X, y, {batchSize: 2,shuffle: true,seed: 42  // Deterministic shuffling}); for (const [xBatch, yBatch] of loader) {// Train on batchconsole.log(xBatch.shape, yBatch.shape);  // [2, 2], [2]}
Example
// Inference without labelsconst testLoader = new DataLoader(X, undefined, {batchSize: 4,shuffle: false}); for (const [xBatch] of testLoader) {// Make predictions}
Members (6)
  • constructor(X: Tensor, y: TTarget, options?: DataLoaderOptions)
  • constructor(X: Tensor, options?: DataLoaderOptions)
  • constructor(dataset: StreamingDataset<unknown>, options?: StreamingDataLoaderOptions)
  • length: number

    Number of batches in the data loader.

  • [Symbol.iterator](): IterableIterator<TTarget extends Tensor ? [ Tensor, Tensor ] : [ Tensor ]>
  • [Symbol.asyncIterator](): AsyncIterableIterator<TTarget extends Tensor ? [ Tensor, Tensor ] : [ Tensor ]>

    Asynchronous iteration over batches.

SequentialSampler

Samples elements sequentially, always in the same order.

Example
const sampler = new SequentialSampler(10);for (const idx of sampler) { ... } // 0, 1, 2, ..., 9
Members (3)
  • constructor(dataSourceLength: number)
  • readonly length: number
  • [Symbol.iterator](): Iterator<number>

SubsetRandomSampler

Samples elements randomly from a given list of indices (subset), without replacement.

Every iteration yields a full permutation of the indices. Without a seed each iteration draws from the global generator, so epochs differ. With a seed, every iteration yields the same permutation.

Example
const sampler = new SubsetRandomSampler([0, 2, 4, 6, 8]);for (const idx of sampler) { ... } // random permutation of [0,2,4,6,8]
Members (3)
  • constructor(indices: ArrayLike<number>, options?: { seed?: number; reshuffleEachIteration?: boolean; })
  • readonly length: number
  • [Symbol.iterator](): Iterator<number>

WeightedRandomSampler

Samples elements according to given weights (probabilities), with or without replacement.

Weights need not sum to one; index i is drawn with probability weights[i] / sum(weights). Indices with weight 0 are never drawn. Without a seed, each iteration draws from the global generator; with a seed, every iteration yields the same sequence.

Example
// Over-sample minority classconst sampler = new WeightedRandomSampler(  [0.1, 0.1, 0.4, 0.4],  { numSamples: 8, replacement: true });
Members (3)
  • constructor(weights: ArrayLike<number>, options?: { numSamples?: number; replacement?: boolean; seed?: number; reshuffleEachIteration?: boolean; })
  • readonly length: number
  • [Symbol.iterator](): Iterator<number>

Subset

A subset of a dataset defined by explicit indices.

data, target (and images, when the dataset has them) hold copies of the selected samples, in the order given. Indices may repeat.

Example
const iris = loadIris();const sub = new Subset(iris, [0, 1, 2, 3, 4]);console.log(sub.data.shape);  // [5, 4]
Members (8)
  • constructor(dataset: Dataset, indices: readonly number[])
  • readonly data: Tensor
  • readonly target: Tensor
  • readonly featureNames: string[]
  • readonly targetNames?: string[]
  • readonly description: string
  • readonly images?: Tensor
  • readonly indices: readonly number[]

Functions

3

filterDataset

function filterDataset(dataset: Dataset, predicate: (data: number[], target: number) => boolean): Dataset;

Filter a dataset by a predicate function, keeping only samples where the predicate returns true.

The predicate receives one sample's features as a flat number[] and its scalar target. Samples keep their original order, dtypes and images. When nothing matches, the result has zero samples but keeps the feature axes (for example data of shape [0, 4]).

Parameters

dataset
Dataset
Source dataset. target must be 1D (one value per sample).
predicate
(data: number[], target: number) => boolean
Function that receives sample data and target, returns boolean

Returns

DatasetA new Dataset containing only matching samples

Throws

  • ShapeErrorIf target is not 1D
Example
const iris = loadIris();// Keep only class 0 and 1const filtered = filterDataset(iris, (_data, target) => target < 2);

mapDataset

function mapDataset(dataset: Dataset, fn: (data: number[], target: number) => { data: number[]; target: number; }): Dataset;

Apply a transformation function to every sample in a dataset, producing a new dataset.

Details

The function receives one sample's features as a flat number[] (the values of data[i] in row-major order) and its scalar target, and must return a { data: number[]; target: number } pair. All returned data arrays must have the same length; the result data has shape [nSamples, length].

The result keeps the float dtype of the source data. The target keeps the dtype of the source target unless a returned value cannot be represented in it (for example a fractional value for an int32 target), in which case the default float dtype is used. The source dataset is not modified.

Parameters

dataset
Dataset
Source dataset. target must be 1D (one value per sample).
fn
(data: number[], target: number) => { data: number[]; target: number; }
Mapping function applied to each sample

Returns

DatasetA new Dataset with transformed data

Throws

  • ShapeErrorIf target is not 1D or fn returns rows of different lengths
  • InvalidParameterErrorIf fn does not return { data, target } with an array and a number
Example
const iris = loadIris();// Scale featuresconst mapped = mapDataset(iris, (data, target) => ({  data: data.map(v => v * 2),  target,}));

randomSplit

function randomSplit(dataset: Dataset, lengths: readonly number[], seed?: number): Subset[];

Randomly split a dataset into non-overlapping subsets of given lengths.

lengths is either absolute sample counts that sum to the dataset size, or fractions in [0, 1] that sum to 1. For fractions, each subset gets floor(fraction * n) samples and the remaining samples are handed out one at a time to the subsets in order, as in PyTorch's random_split.

Parameters

dataset
Dataset
The dataset to split
lengths
readonly number[]
Subset sizes: integers summing to the dataset size, or fractions summing to 1
seedoptional
number
Optional seed for reproducible splits

Returns

Subset[]Array of Subset instances

Throws

  • InvalidParameterErrorIf lengths are negative or do not add up, or seed is not a safe integer
Example
const iris = loadIris();const [train, test] = randomSplit(iris, [120, 30], 42);console.log(train.data.shape);  // [120, 4]console.log(test.data.shape);   // [30, 4] const [a, b] = randomSplit(iris, [0.8, 0.2], 42);console.log(a.data.shape[0], b.data.shape[0]); // 120 30
datasets-dataloader.ts
import { DataLoader, loadIris, randomSplit } from "deepbox/datasets"; const iris = loadIris();const [train, test] = randomSplit(iris, [0.8, 0.2], 42); const loader = new DataLoader(train.data, train.target, {  batchSize: 32,  shuffle: true,  seed: 1,  reshuffleEachIteration: true,}); console.log(loader.length); // number of batches per epochfor (const [xBatch, yBatch] of loader) {  console.log(xBatch.shape, yBatch.shape);} console.log(train.data.shape[0], test.data.shape[0]); // 120 30

Found a mistake? Open an issue.