← Back to API reference

neuralbench.main.BenchmarkAggregator

pydantic model neuralbench.main.BenchmarkAggregator

Orchestrate multiple Experiment runs and visualise results.

Experiments are submitted (possibly via Slurm) with prepare(), collected with _collect_results(), and plotted/tabled via the functions in neuralbench.plots.benchmark.

Fields:

• collect_max_workers (int)

• debug (bool)

• device (str | None)

• eval_mode (str | None)

• experiments (list['Experiment'])

• loss_to_metric_mapping (dict[str, str])

• max_workers (int)

• output_dir (str)

field experiments: list['Experiment'] [Required]

field max_workers: int = 256

field collect_max_workers: int = 32

field debug: bool = False

field output_dir: str [Optional]

field device: str | None = None

field eval_mode: str | None = None

field loss_to_metric_mapping: dict[str, str] = {'BCEWithLogitsLoss': 'test/f1_score_macro', 'CTCLoss': 'test/CER', 'ClipLoss': 'test/full_retrieval/inv_norm_rank_mean', 'CrossEntropyLoss': 'test/bal_acc', 'L1Loss': 'test/mae', 'MSELoss': 'test/pearsonr', 'MultiLoss': 'test/bmae'}

prepare() → None

property device_output_dir: Path

output_dir/<device>, or output_dir.

Type:

Root of this run’s artefacts

collect(cached_only: bool = True) → list[dict[str, Any]]

Gather results, without the plots, tables and stats files run() writes.

run(cached_only: bool = False) → list[dict[str, Any]]

← Back to API reference