Image decoding¶

Name: image
Category: cognitive decoding
Dataset: Chang2019Bold5000 (BOLD5000)
Objective: Retrieval
Split: Predefined

Usage¶

neuralbench fmri image
Show config.yaml
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

data:
  study:
    source:
      name: Chang2019Bold5000
    split:
      name: PredefinedSplit
      test_split_query: null
      col_name: split
      valid_split_by: timeline
      valid_split_ratio: 0.2
      valid_random_state: 33
  target:
    name: HuggingFaceImage
    model_name: facebook/dinov2-giant
    layers: 0.6667
    token_aggregation: mean
    imsize: 518
    aggregation: trigger
    infra:
      cluster: !!python/name:neuralbench.config_manager.CLUSTER
      keep_in_ram: false
      timeout_min: 180
      gpus_per_node: 1
      cpus_per_task: 10
      min_samples_per_job: 64
  trigger_event_type: Image
  start: 2.0
  duration: 7.0
space_name: mni2fsaverage5
available_spaces: [mni, mni2fsaverage4, mni2fsaverage5, mni2fsaverage6]
brain_model_output_size: &brain_model_output_size 1536
brain_model_config:
  output_head_config:
    hidden_sizes:
      - *brain_model_output_size
trainer_config.monitor: val/batch_inv_norm_rank_mean
trainer_config.mode: max
loss:
  name: ClipLoss
  norm_kind: y
  temperature: false
  symmetric: false
metrics: !!python/name:neuralbench.metric_configs.retrieval_metrics
test_full_retrieval_metrics: !!python/name:neuralbench.metric_configs.test_full_retrieval_metrics

Description¶

The fMRI image decoding task involves retrieving the visual stimulus presented to a subject from their BOLD fMRI response. We use BOLD5000 [Chang2019], a 3T fMRI dataset in which 4 subjects each viewed ~5,000 natural images drawn from the SUN, COCO, and ImageNet collections over 9–15 scanning sessions.

BOLD responses are extracted from a 7-second window starting 2 seconds after image onset to account for the hemodynamic delay. Surface-projected activations (fsaverage5 mesh) are used as model input. Target embeddings are DINOv2-giant representations of the presented images (1536-dimensional).

The train/test split follows the dataset-native partition based on the 113 images repeated across sessions for test-retest reliability.

Dataset Notes¶

  • BOLD5000 is distributed via OpenNeuro (ds001499) under a CC0 licence; the preprocessed beta maps (MNI152NLin2009aSym space) and the stimulus images are needed.

Additional Datasets¶

The following additional fMRI image-decoding datasets can also be used with this task:

  • Allen2022MassiveRaw (NSD) – 8 participants each viewing up to 10,000 distinct natural scene images over 30–40 7T scanning sessions (~73,000 presentations total) [Allen2022].

  • Hebart2023ThingsBold (THINGS-fMRI) – 3 participants viewing 1,854 diverse object images from the THINGS database [Hebart2023].

To run with an alternate dataset:

neuralbench fmri image --dataset Allen2022MassiveRaw

References¶

[Allen2022]

Allen, Emily J., et al. “A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.” Nature Neuroscience 25.1 (2022): 116-126.

[Chang2019]

Chang, Nadine, et al. “BOLD5000, a public fMRI dataset while viewing 5000 visual images.” Scientific Data 6.1 (2019): 49.

[Hebart2023]

Hebart, Martin N., et al. “THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior.” eLife 12 (2023): e82580.