Benchmarking LLMs for Comprehensive EEG Understanding

BrainBench is a benchmark for comprehensive EEG understanding, evaluating how large language models and agentic systems analyze real-world EEG recordings and produce scientifically grounded conclusions.

📄 ArXiv Paper  |  🌐 Website  |  🤗 Hugging Face  |  🚀 Quickstart  |  🤖 Agent  |  📚 Citation

---

🔬 Overview

BrainBench evaluates EEG understanding from multiple complementary perspectives through four benchmark subsets:

• Foundational Analysis: Core EEG signal understanding and analysis.
• Sleep Assessment: Sleep-related EEG assessment and staging.
• Neurocognitive Assessment: EEG-based assessment of cognitive functions.
• Physiological Integration: Joint reasoning over EEG and physiological information.

📏 Benchmark at a glance

Item	Count
Subsets	4
Datasets	17
Tasks	172
Evaluation instances	~4,000

---

📁 Project structure

BrainBench/
├── brainbench/                  # core benchmark package
│   ├── codeact/                 # CodeAct agent execution modules
│   ├── agent.py                 # agent interface
│   ├── cases.py                 # benchmark case loading
│   ├── config.py                # runtime configuration
│   ├── evaluator.py             # evaluation pipeline
│   ├── llm.py                   # LLM request layer
│   ├── runners.py               # evaluation runners
│   └── scoring.py               # scoring and aggregation
├── benchmarks/                  # evaluation cases downloaded from Hugging Face
│   ├── foundational_analysis/
│   │   └── cases/               # evaluation JSON files
│   ├── sleep_assessment/
│   │   └── cases/               # evaluation JSON files
│   ├── neurocognitive_assessment/
│   │   └── cases/               # Neurocognitive Assessment cases
│   └── physiological_integration/
│       └── cases/               # Physiological Integration cases
├── docker/                      # Docker environments
│   └── codeact/                 # CodeAct Docker image definition
├── examples/                    # examples and offline smoke tests
│   └── synthetic_smoke/
├── assets/                      # README images and project artwork
├── main.py                      # command-line entry point
├── requirements.txt             # Python dependencies
├── .env.example                 # environment variable template
└── .gitignore                   # ignored local files

The benchmarks/ directory is intentionally shipped without evaluation JSON files. Download the corresponding case files from Hugging Face and place them under benchmarks/<subset>/cases/ before running an evaluation.

---

🚀 Quickstart

Run these commands from the repository root and use the identifier of the subset you want to operate on.

1. Install

git clone https://github.com/xiaobaben/BrainBench.git
cd BrainBench
python3.9 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

2. Configure

cp .env.example .env

Fill in the model credentials and choose the CodeAct execution mode in .env:

BRAINBENCH_API_KEY=YOUR_API_KEY
BRAINBENCH_BASE_URL=https://YOUR_PROVIDER_BASE_URL/v1
BRAINBENCH_MODEL=YOUR_MODEL_NAME

3. Smoke test

python main.py smoke

4. Download benchmark cases

The fixed case JSON files are published in the BrainBench Hugging Face dataset. They contain the benchmark inputs, parsing instructions, ground truth, and metrics; raw EEG/PSG recordings are not included.

Install the Hugging Face CLI and download all benchmark cases:

python -m pip install --upgrade huggingface_hub
hf download xbb083/BrainBench \
  --repo-type dataset \
  --local-dir ./benchmarks

To download only the Foundational Analysis cases:

hf download xbb083/BrainBench \
  --repo-type dataset \
  --include "foundational_analysis/**" \
  --local-dir ./benchmarks

For a different individual subset, replace foundational_analysis with sleep_assessment, neurocognitive_assessment, or physiological_integration. The downloaded files are placed under benchmarks/<subset>/cases/ and should remain unchanged.

5. Prepare data

First, obtain the required raw datasets from the official sources listed in the Dataset Access Guide. For Foundational Analysis, place the five dataset folders directly under one user-selected <data-root>:

<data-root>/
├── isruc/
├── bcic2020-3/
├── MentalArithmetic/
├── mumtaz/
└── seedv/

After the raw data is organized, prepare Foundational Analysis with:

python main.py prepare foundational_analysis \
  --data-root /path/to/data-root

To prepare a different subset, replace foundational_analysis with sleep_assessment, neurocognitive_assessment, or physiological_integration, and pass the corresponding subset's <data-root>. Prepared data is written to the output directory configured for the selected subset; Foundational Analysis, for example, is written to data/core/.

6. Build the CodeAct Docker image

CodeAct executes model-generated analysis code. With BRAINBENCH_CODEACT_MODE=docker, the code runs inside an isolated container built from the project image; this is the recommended mode for safer execution and reproducible dependencies. With BRAINBENCH_CODEACT_MODE=local, the code runs directly on the host machine without container isolation, so the host must provide the required packages and has a weaker safety boundary.

Build the image:

docker build -t brainbench-codeact:latest docker/codeact

To use Docker mode, set the following variables in .env:

BRAINBENCH_CODEACT_MODE=docker
BRAINBENCH_DOCKER_IMAGE=brainbench-codeact:latest

Use BRAINBENCH_CODEACT_MODE=local only when Docker is unavailable or direct host execution is intended.

7. Run the benchmark

python main.py run foundational_analysis --agent codeact

This command runs the Foundational Analysis subset with the built-in CodeAct agent. To run another subset, replace foundational_analysis with sleep_assessment, neurocognitive_assessment, or physiological_integration.

🤖 Custom Agent Integration

Edit run_custom_agent() in main.py. Pass the complete query to your Agent and return its natural-language response:

def run_custom_agent(query: str) -> AgentRunResult:
    """Send the complete query to the user's Agent and return its result."""

    class TargetAgent:
        @staticmethod
        def run(context: str) -> str:
            # Replace with your own Agent implementation.
            return context

    your_agent = TargetAgent()
    response = your_agent.run(query)

    return AgentRunResult(response=response, tokens=0)

Run the evaluation with:

python main.py run foundational_analysis --agent custom

---

📋 Audit and logs

Results are written to runs/<subset>.json by default. To choose a different path, pass --output-path:

python main.py run foundational_analysis \
  --agent codeact \
  --output-path runs/foundational_codeact.json

The result JSON is updated after every completed instance and contains:

• experiment: subset, model roles, execution mode, start and finish times, and Docker configuration when used.
• instances: case identifiers, source JSON, Agent response, parser output, scores, and metric details.
• instances[].sandbox: CodeAct policy, iteration trace, API attempts and retries, termination status, and runtime audit data when available.
• instances[].artifact_manifest: generated file name, size, SHA-256 digest, and scoring status.
• instances[].token_usage, timing_sec, and errors: per-instance resource usage, execution timing, and failure details.
• summary: planned and completed instances, aggregate score, failure categories, metric summaries, token usage, and wall-clock timing.

---