Image decoding

Name: image
Category: cognitive decoding
Dataset: Gifford2022Large (THINGS-EEG2)
Objective: Retrieval
Split: Predefined

Usage

neuralbench eeg image
Show config.yaml
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

data:
  study:
    source:
      name: Gifford2022Large
    split:
      name: PredefinedSplit
      test_split_query: null
      col_name: split
      valid_split_by: timeline
      valid_split_ratio: 0.2
      valid_random_state: 33
  neuro.baseline: [0.0, 0.2]
  target:
    name: HuggingFaceImage
    model_name: facebook/dinov2-giant
    layers: 0.6667
    token_aggregation: mean
    imsize: 518
    aggregation: trigger
    infra:
      cluster: auto
      keep_in_ram: false
      timeout_min: 180
      gpus_per_node: 1
      cpus_per_task: 10
      min_samples_per_job: 64
  trigger_event_type: Image
  start: -0.2
  duration: 1.0
  summary_columns: [category, filepath]
brain_model_output_size: &brain_model_output_size 1536
trainer_config.monitor: val/batch_top5_acc
trainer_config.mode: max
loss:
  name: ClipLoss
  norm_kind: y
  temperature: false
  symmetric: false
metrics: !!python/name:neuralbench.defaults.metrics.retrieval_metrics
test_full_retrieval_metrics: !!python/name:neuralbench.defaults.metrics.test_full_retrieval_metrics

Description

The image decoding task involves decoding visual stimuli from EEG recordings [Benchetrit2023]. In this task, we use the Gifford2022Large dataset [Gifford2022Large], which contains EEG data recorded while subjects viewed images from the THINGS database, a large-scale collection of naturalistic object images [Hebart2019]. The goal is to retrieve the presented image based on the EEG signals and a fixed pretrained image feature extractor.

Dataset Notes

  • We use [Gifford2022Large] for evaluation because test images were recorded in separate runs, limiting the potential impact of temporal correlations on decoding performance.

References

[Benchetrit2023]

Benchetrit, Yohann, Hubert Banville, and Jean-Rémi King. “Brain decoding: toward real-time reconstruction of visual perception.” arXiv preprint arXiv:2310.19812 (2023).

[Gifford2022Large] (1,2)

Gifford, Alessandro T., et al. “A large and rich EEG dataset for modeling human visual object recognition.” NeuroImage 264 (2022): 119754.

[Hebart2019]

Hebart, Martin N., et al. “THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images.” PloS one 14.10 (2019): e0223792.