NeuralBench

NeuralBench¶

NeuralBench is a unified framework to benchmark NeuroAI models. It provides a comprehensive suite of downstream tasks for evaluating pretrained brain models, task-specific architectures, and pretraining strategies across EEG, MEG, and fMRI.

The first release ships two EEG suites – NeuralBench-EEG-Core v1.0 (one dataset per task across the 36 EEG tasks) and NeuralBench-EEG-Full v1.0 (all 94 datasets registered for the same 36 tasks) – evaluated with 8 task-specific and 6 foundation model architectures.

pip install neuralbench

See Installation for more options.

🚀 Quickstart¶

Evaluate a model on a downstream task in three commands:

neuralbench eeg audiovisual_stimulus --download    # 1. Download data
neuralbench eeg audiovisual_stimulus --prepare     # 2. Prepare preprocessing cache
neuralbench eeg audiovisual_stimulus               # 3. Run the task

The audiovisual_stimulus task uses the single-subject MNE-Python sample dataset (~1.5 GB). We use it both as a quick sanity-check task and as a probe of model behaviour in very-low-data regimes (288 trials, 4-class classification). Add --debug for a fast local run.

After experiments complete, re-run with --plot-cached to generate comparison figures and tables from the stored results (no retraining):

neuralbench eeg audiovisual_stimulus --plot-cached

Learn more

Open the quickstart tutorial for a full walkthrough of the CLI, config system, and model selection.

📖 Tutorials¶

Quickstart

Run your first benchmark task: CLI usage, debug mode, model switching, and hyperparameter grids.

Running Your First Benchmark Task

Visualizing Results

Use BenchmarkAggregator to collect results and produce comparison plots, results tables, and rank tables.

Visualizing Benchmark Results

Adding a New Task

Create a config.yaml for a new downstream task.

Adding a New Downstream Task

Adding a New Model

Advanced

Customize the training loop by subclassing BrainModule.

Modifying the Training Loop

EEG Foundation Challenge 2025

Reproduce the NeurIPS 2025 challenge tasks (cross-task reaction time and externalizing-factor prediction) with NeuralBench.

Reproducing tasks from the EEG Foundation Challenge 2025

Running the full EEG benchmark¶

To evaluate a model across all 36 EEG tasks, use the all keyword:

neuralbench eeg all --download    # 1. Download all automatically downloaded datasets (~3.3 TB)
neuralbench eeg all --prepare     # 2. Build preprocessing cache for task-specific models
neuralbench eeg all               # 3. Run all 36 EEG tasks

Full guide

See Running the full EEG benchmark for prerequisites, resource requirements, model and dataset variant options, and computational considerations.

Supported downstream tasks and models¶

Downstream Tasks

Browse the full list of supported tasks with descriptions, datasets, and configurations.

Downstream tasks

Models

See all available task-specific and foundation models for evaluation.

Models

Citation¶

If you use NeuralBench in your research, please cite the NeuralBench white paper:

@misc{banville2026neuralbench,
  title        = {NeuralBench: A Unifying Framework to Benchmark NeuroAI Models},
  author       = {Banville, Hubert and d'Ascoli, St{\'e}phane and Dahan, Simon
                  and Rapin, J{\'e}r{\'e}my and Careil, Marl{\`e}ne
                  and Benchetrit, Yohann and L{\'e}vy, Jarod
                  and Panchavati, Saarang and Ratouchniak, Antoine
                  and Zhang, Mingfang and Cascardi, Elisa and Begany, Katelyn
                  and Brooks, Teon and King, Jean-R{\'e}mi},
  year         = {2026},
  howpublished = {Brain \& AI team, Meta FAIR},
  url          = {https://ai.meta.com/research/publications/neuralbench-a-unifying-framework-to-benchmark-neuroai-models/},
}