Typing decoding

Name: typing

Category: motor / input decoding

Dataset: Sivakumar2024Emg2qwerty (NM000104, emg2qwerty)

Objective: CTC sequence decoding

Split: Leave-subjects-out (cross-subject)

Usage

# Auto-fetch NM000104 (~239 GB) via eegdash
neuralbench emg typing -m emg2qwerty --download

# Local 2-epoch sanity check
neuralbench emg typing -m emg2qwerty --debug

# Full benchmark run
neuralbench emg typing -m emg2qwerty

Show config.yaml

# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the license found in the
# LICENSE file in the root directory of this source tree.

# emg/typing CTC task: surface-EMG → keystroke sequence (Sivakumar2024Emg2qwerty).
# Cross-subject baseline restricted to the first ~10 subjects
# (timeline_index < 120) to keep benchmark runtime tractable.

# Paper recipe optimizer (Adam + OneCycleLR, lr 1e-8 → 1e-3 → 1e-6 with
# 40 % warmup).  Without the slow warmup CTC collapses to all-blank
# within an epoch.
lightning_optimizer_config:
  =replace=: true
  optimizer:
    name: Adam
    lr: 1.0e-3
    kwargs: {}
  scheduler:
    name: OneCycleLR
    kwargs:
      max_lr: 1.0e-3
      pct_start: 0.4
      anneal_strategy: cos
      div_factor: 1.0e+5
      final_div_factor: 1.0e+3
  interval: step
# Default backbone for this task is the paper's CTC sequence model.  Models
# that don't override ``brain_model_config`` (notably ``chance``) inherit it,
# so ``chance`` becomes an *untrained* EMG2QwertyNet — a sequence-shaped
# (B, T_out, 99) emitter compatible with the CTC head/metric — rather than
# EEGNet, whose single-vector output is incompatible with CTC.
# ``dummy`` and ``emg2qwerty`` replace this via their own ``=replace=``.
brain_model_config:
  =replace=: true
  name: EMG2QwertyNet
  kwargs:
    log_softmax: true
    spec_augment: true
data:
  batch_size: 16
  study:
    source:
      name: Sivakumar2024Emg2qwerty
      query: "timeline_index < 120"
    split:
      name: SklearnSplit
      split_by: subject
      valid_split_ratio: 0.2
      test_split_ratio: 0.2
      valid_random_state: 33
      test_random_state: 33
  target:
    =replace=: true
    # CroppedExtractor restricts label collection to the un-padded 4 s
    # "core" of the padded 5 s window; the inner SequenceLabelEncoder
    # turns the keystrokes in that core into a fixed-length blank-padded
    # CTC target of shape ``(128,)``.
    name: CroppedExtractor
    offset: 0.9
    duration: 4.0
    # Sliding windows that land in typing gaps legitimately contain no
    # Keystroke events; permit them (empty CTC target, handled by
    # ``zero_infinity=True``).  ``CroppedExtractor`` does not inherit
    # ``allow_missing`` from its wrapped extractor, so set it here too.
    allow_missing: true
    extractor:
      # Reads the pre-computed integer ``label`` column written by the
      # study's events-dataframe construction (in
      # ``sivakumar2024emg2qwerty.py``) — no string→int mapping at
      # encode time, no OOV branch (OOV keys are dropped upstream).
      name: SequenceLabelEncoder
      event_types: Keystroke
      event_field: label
      allow_missing: true
      max_length: 128
      pad_value: 98
  # Paper-faithful sliding-window segmentation: 4 s core + 0.9 s left +
  # 0.1 s right padding; stride = core duration so windows abut.
  trigger_event_type: BidsEmg
  start: -0.9
  duration: 5.0
  stride: 4.0
  stride_drop_incomplete: true
  summary_columns: [text]
trainer_config.monitor: val/CER
trainer_config.mode: min
loss:
  =replace=: true
  name: CTCLoss
  kwargs:
    blank: 98
    zero_infinity: true
metrics:
  - name: CharacterErrorRates
    log_name: CER
    blank_idx: 98

Description

Continuous-keystroke decoding from 32-channel surface EMG (two 16-electrode wristbands at 2 kHz) using the CTC framework from [Sivakumar2024]. Each 5-s window (0.9 s + 4 s core + 0.1 s) is mapped to a variable-length keystroke sequence over the 98-key paper vocabulary plus a CTC blank – 99 output classes.

As compared to the original paper, the NeuralBench default configuration restricts training to the first ~10 subjects (timeline_index < 120) and uses a leave-subjects-out split for cross-subject evaluation, keeping turn-around tractable.

Dataset Notes

• Auto-fetch: --download pulls NM000104 from NEMAR (s3://nemar/nm000104) via neuralfetch.download.Eegdash – 1136 files, ~239 GB, under <DATA_DIR>/Sivakumar2024Emg2qwerty/download/nm000104/sub-*/.... Users with an existing BIDS copy should symlink it into download/nm000104/.

• BIDS-aware reader: the Study reads via mne_bids.read_raw_bids() (>= 0.19); channel types and units come from the BIDS sidecars, so BidsEmg._read returns the EMG channels in microvolts directly – no manual rescaling.

• Windowing: 5-s sliding windows with a 4-s stride (0.9 s left + 4-s core + 0.1 s right); CroppedExtractor restricts label collection to the core, so the 4-s cores tile each session non-overlappingly while the EMG signal context overlaps neighbours by 1 s. The paper [Sivakumar2024] trains on 4-s windows but feeds whole sessions at test time; we apply this 5-s padded window across all splits – slightly pessimistic CER, tracked as a follow-up.

References

[Sivakumar2024] (1,2)

Sivakumar, Viswanath, et al. “emg2qwerty: A large dataset with baselines for touch typing using surface electromyography.” Advances in Neural Information Processing Systems 37 (2024): 91373–91389.