Image decoding

Name: image

Category: cognitive decoding

Dataset: Hebart2023ThingsMeg (THINGS-MEG)

Objective: Retrieval

Split: Predefined

Usage

neuralbench meg image

Show config.yaml

# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

data:
  study:
    source:
      name: Hebart2023ThingsMeg
    split:
      name: PredefinedSplit
      test_split_query: null
      col_name: split
      valid_split_by: timeline
      valid_split_ratio: 0.2
      valid_random_state: 33
  neuro:
    name: MegExtractor
    picks: [meg]
    baseline: [0.0, 0.2]
    allow_maxshield: true
  channel_positions:
    include_ref_eeg: false
    layout_or_montage_name: null
    n_spatial_dims: 3
  target:
    name: HuggingFaceImage
    model_name: facebook/dinov2-giant
    layers: 0.6667
    token_aggregation: mean
    imsize: 518
    aggregation: trigger
    infra:
      cluster: auto
      keep_in_ram: false
      timeout_min: 180
      gpus_per_node: 1
      cpus_per_task: 10
      min_samples_per_job: 64
  trigger_event_type: Image
  start: -0.2
  duration: 1.0
brain_model_output_size: &brain_model_output_size 1536
trainer_config.monitor: val/batch_top5_acc
trainer_config.mode: max
loss:
  name: ClipLoss
  norm_kind: y
  temperature: false
  symmetric: false
metrics: !!python/name:neuralbench.defaults.metrics.retrieval_metrics
test_full_retrieval_metrics: !!python/name:neuralbench.defaults.metrics.test_full_retrieval_metrics

Description

The image decoding task involves decoding visual stimuli from MEG recordings [Benchetrit2023b]. In this task, we use the THINGS-MEG dataset [Hebart2023], which contains EEG data recorded while subjects viewed images from the THINGS database, a large-scale collection of naturalistic object images [Hebart2019b]. The goal is to retrieve the presented image based on the MEG signals and a fixed pretrained image feature extractor.

References

[Benchetrit2023b]

Benchetrit, Yohann, Hubert Banville, and Jean-Rémi King. “Brain decoding: toward real-time reconstruction of visual perception.” arXiv preprint arXiv:2310.19812 (2023).

[Hebart2023]

Hebart, Martin N., et al. “THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior.” Elife 12 (2023): e82580.

[Hebart2019b]

Hebart, Martin N., et al. “THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images.” PloS one 14.10 (2019): e0223792.