Image decoding

Name: image

Category: cognitive decoding

Dataset: Chang2019Bold5000 (BOLD5000)

Objective: Retrieval

Split: Predefined

Usage

neuralbench fmri image

Show config.yaml

# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

data:
  study:
    source:
      name: Chang2019Bold5000
    split:
      name: PredefinedSplit
      test_split_query: null
      col_name: split
      valid_split_by: timeline
      valid_split_ratio: 0.2
      valid_random_state: 33
  neuro:
    =replace=: true
    name: FmriExtractor
    projection:
      name: SurfaceProjector
      mesh: fsaverage5
    from_space: auto
    offset: 0
    frequency: native
    infra:
      mode: cached
      cluster: auto
      folder: !!python/name:neuralbench.config_manager.CACHE_DIR
      keep_in_ram: true
      slurm_partition: !!python/name:neuralbench.config_manager.SLURM_PARTITION
      gpus_per_node: 1
      cpus_per_task: !!python/name:neuralbench.config_manager.N_CPUS
      timeout_min: 180
      min_samples_per_job: 64
  target:
    name: HuggingFaceImage
    model_name: facebook/dinov2-giant
    layers: 0.6667
    token_aggregation: mean
    imsize: 518
    aggregation: trigger
    infra:
      cluster: auto
      keep_in_ram: false
      timeout_min: 180
      gpus_per_node: 1
      cpus_per_task: 10
      min_samples_per_job: 64
  trigger_event_type: Image
  start: 2.0
  duration: 7.0
brain_model_output_size: &brain_model_output_size 1536
brain_model_config:
  =replace=: true
  name: FmriMlp
  hidden: 128
  n_blocks: 2
  n_repetition_times: 4
  output_head_config:
    hidden_sizes:
      - *brain_model_output_size
trainer_config.monitor: val/batch_top5_acc
trainer_config.mode: max
loss:
  name: ClipLoss
  norm_kind: y
  temperature: false
  symmetric: false
metrics: !!python/name:neuralbench.defaults.metrics.retrieval_metrics
test_full_retrieval_metrics: !!python/name:neuralbench.defaults.metrics.test_full_retrieval_metrics

Description

The fMRI image decoding task involves retrieving the visual stimulus presented to a subject from their BOLD fMRI response. We use BOLD5000 [Chang2019], a 3T fMRI dataset in which 4 subjects each viewed ~5,000 natural images drawn from the SUN, COCO, and ImageNet collections over 9–15 scanning sessions.

BOLD responses are extracted from a 7-second window starting 2 seconds after image onset to account for the hemodynamic delay. Surface-projected activations (fsaverage5 mesh) are used as model input. Target embeddings are DINOv2-giant representations of the presented images (1536-dimensional).

The train/test split follows the dataset-native partition based on the 113 images repeated across sessions for test-retest reliability.

Dataset Notes

• BOLD5000 is distributed via OpenNeuro (ds001499) under a CC0 licence; the preprocessed beta maps (MNI152NLin2009aSym space) and the stimulus images are needed.

Additional Datasets

The following additional fMRI image-decoding datasets can also be used with this task:

• Allen2022MassiveRaw (NSD) – 8 participants each viewing up to 10,000 distinct natural scene images over 30–40 7T scanning sessions (~73,000 presentations total) [Allen2022].

• Hebart2023ThingsBold (THINGS-fMRI) – 3 participants viewing 1,854 diverse object images from the THINGS database [Hebart2023].

To run with an alternate dataset:

neuralbench fmri image --dataset Allen2022MassiveRaw

References

[Allen2022]

Allen, Emily J., et al. “A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.” Nature Neuroscience 25.1 (2022): 116-126.

[Chang2019]

Chang, Nadine, et al. “BOLD5000, a public fMRI dataset while viewing 5000 visual images.” Scientific Data 6.1 (2019): 49.

[Hebart2023]

Hebart, Martin N., et al. “THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior.” eLife 12 (2023): e82580.