← Back to API reference

neuralbench.evaluate_model

neuralbench.evaluate_model(model: Module, device: str, task: str | list[str], *, name: str = 'external', overrides: Mapping[str, Any] | None = None, downstream_wrapper: str | list[str] = 'linear_probe_mean', download: bool = True, prepare: bool = True, cluster: str | None = None, debug: bool = False, **selection: Any) → DataFrame[source][source]

Run model on a selection of tasks and return the results.

Runs in-process by default, which is usually what you want in a notebook but means a full suite can take a very long time. Pass cluster="auto" to fan the experiments out to SLURM instead; that call returns once they are queued, so it yields whatever has finished (nothing, at first). Call it again with the same arguments to collect – already-completed experiments are not resubmitted, exactly as with re-running the CLI.

Parameters:

  • model – An instantiated nn.Module accepting (batch, channels, samples) at any width and length, plus channel_positions. A forward that also names ch_names is given the dataset’s channel names, which is the only way to identify electrodes for a model keyed by name (e.g. LaBraM). It is serialized once to the cache folder and reloaded fresh for each experiment. Run check_model() first.

  • device – As on the CLI; task="all" runs every validated task for the device, which with the default dataset=None is the device’s Core suite and with dataset="all" its Full suite.

  • task – As on the CLI; task="all" runs every validated task for the device, which with the default dataset=None is the device’s Core suite and with dataset="all" its Full suite.

  • name – Label for this model in the results frame.

  • overrides – Dotted-key or nested config overrides, applied last and recorded in the frame’s overrides column.

  • downstream_wrapper – Adaptation strategy, which is also what supplies the task’s classifier head; the default freezes the model and trains a linear probe on its mean-pooled output.

  • download – Fetch the datasets first. Nothing downloads on demand, so a missing dataset otherwise fails deep inside Study. Cheap once present.

  • prepare – Warm the preprocessing caches first, by running one cut-down experiment per task/dataset. Hours on a cold cache for a whole device, a status check when already warm. Needs the data, hence after download. On a cluster the warm-up is queued rather than run, so the call returns without submitting the experiments; call it again once it has finished.

  • cluster – None runs in-process; "auto" uses SLURM when available; "slurm" requires it. Applies to the runs and the caches alike.

  • debug – Reduce every experiment (2 epochs, 5 batches, a data subset) and run locally. For checking that a model trains at all, not for results.

  • **selection – Remaining neuralbench.cli.run_benchmark() arguments (dataset, force, …).

← Back to API reference