Image decoding

Name: image

Category: cognitive decoding

Dataset: Allen2022MassiveRaw (NSD, BIDS/deepprep variant)

Objective: Retrieval

Split: Predefined

Note

The Allen2022MassiveRaw study wrapper is not part of the open-source release yet.

Usage

neuralbench fmri image

Show config.yaml

# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

data:
  study:
    source:
      name: Allen2022MassiveRaw
    # Keep a single preprocessed fMRI space per timeline. Allen2022MassiveRaw
    # (and other deepprep studies) emit one Fmri event per
    # {T1w, MNI152NLin2009cAsym, fsnative, fsaverage}; extractors like the
    # subject-id LabelEncoder have no from_space filter, so we narrow to
    # fsaverage (matching the SurfaceProjector(mesh=fsaverage5) below) upstream
    # of segmentation.
    filter_fmri_space:
      name: QueryEvents
      query: 'type != "Fmri" or space == "fsaverage"'
    split:
      name: PredefinedSplit
      test_split_query: null
      col_name: split
      valid_split_by: timeline
      valid_split_ratio: 0.2
      valid_random_state: 33
  neuro:
    =replace=: true
    name: FmriExtractor
    projection:
      name: SurfaceProjector
      mesh: fsaverage5
    from_space: auto
    offset: 0
    frequency: native
    infra:
      mode: cached
      cluster: auto
      folder: !!python/name:neuralbench.config_manager.CACHE_DIR
      keep_in_ram: true
      slurm_partition: !!python/name:neuralbench.config_manager.SLURM_PARTITION
      gpus_per_node: 1
      cpus_per_task: !!python/name:neuralbench.config_manager.N_CPUS
      timeout_min: 180
      min_samples_per_job: 64
  target:
    name: HuggingFaceImage
    model_name: facebook/dinov2-giant
    layers: 0.6667
    token_aggregation: mean
    imsize: 518
    aggregation: trigger
    infra:
      cluster: auto
      keep_in_ram: false
      timeout_min: 180
      gpus_per_node: 1
      cpus_per_task: 10
      min_samples_per_job: 64
  trigger_event_type: Image
  start: 2.0
  duration: 7.0
brain_model_output_size: &brain_model_output_size 1536
brain_model_config:
  =replace=: true
  name: FmriMlp
  hidden: 128
  n_blocks: 2
  n_repetition_times: 4
  output_head_config:
    hidden_sizes:
      - *brain_model_output_size
trainer_config.monitor: val/batch_top5_acc
trainer_config.mode: max
loss:
  name: ClipLoss
  norm_kind: y
  temperature: false
  symmetric: false
metrics: !!python/name:neuralbench.defaults.metrics.retrieval_metrics
test_full_retrieval_metrics: !!python/name:neuralbench.defaults.metrics.test_full_retrieval_metrics

Description

The fMRI image decoding task involves retrieving the visual stimulus presented to a subject from their BOLD fMRI response. We use the Natural Scenes Dataset (NSD) [Allen2022], a large-scale 7T fMRI dataset in which 8 subjects each viewed up to 10,000 distinct natural scene images over 30–40 scanning sessions (~73,000 image presentations total).

BOLD responses are extracted from a 7-second window starting 2 seconds after image onset to account for the hemodynamic delay. Surface-projected activations (fsaverage5 mesh) are used as model input. Target embeddings are DINOv2-giant representations of the presented images (1536-dimensional).

The train/test split follows the dataset-native partition: NSD reserves a set of 1,000 images seen by all 8 subjects as a shared test set.

Dataset Notes

• The NSD is a very large dataset (several TB). Downloading requires AWS credentials and agreeing to the NSD terms of use.

• Only the betas_fithrf_GLMdenoise_RR single-trial beta maps and the stimulus images are needed.

References

[Allen2022]

Allen, Emily J., et al. “A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.” Nature Neuroscience 25.1 (2022): 116-126.