LLM Embedded Domain Knowledge Probe — Do LLMs actually understand embedded firmware, or do they just pattern-match?
EmbedEval measures whether LLMs possess the implicit domain knowledge to write safe embedded C code. It covers Zephyr RTOS, ESP-IDF, STM32 HAL, FreeRTOS, Linux kernel drivers, and Yocto recipes across 267 test cases (219 public + 48 private held-out).
Unlike HumanEval or SWE-bench which test general coding, EmbedEval tests knowledge that only embedded engineers would have: interrupt safety, cache coherency, DMA alignment, power management, and real-time constraints — without telling the LLM what to check.
Live leaderboard · Methodology · Roadmap · Contribute
Result write-ups are published on EdgeLog, the maintainer's embedded and edge engineering blog. Start with what the firmware benchmark actually measured.
Most benchmarks tell the LLM exactly what to do. EmbedEval tells the LLM what to build but not how to make it safe:
Prompt: "Implement DMA transfer from src to dst buffer.
Use a callback to signal completion."
What an embedded engineer knows (not in prompt):
- Buffer must be cache-line aligned (__aligned(32))
- Cache flush before DMA start
- Cache invalidate after DMA complete
- Completion flag must be volatile (shared with ISR)
- Flag checked AFTER synchronization, not before
Measured impact: Explicit prompts ("use volatile") pass at ~95%. Implicit prompts (derive from domain knowledge) pass at ~60%. This 35%p gap means current benchmarks overestimate LLM capability in embedded domains.
n=3 aggregate pass@1 means:
| Model | pass@1 (n=3 mean) | 95% CI | Stability | Weakest Category | Strongest | Cases |
|---|---|---|---|---|---|---|
| Sonnet 5 | 67.0% | [63.7%, 70.2%] | 82.1% | dma (31%), isr-concurrency (41%), threading (49%) | boot, adc, pwm (100%) | 263 (2026-07-19) |
| Sonnet 4.6 | 68.0% | [64.4%, 71.3%] | 87.1% | isr-concurrency (23%), dma (31%), threading (33%) | adc, device-tree, pwm (100%) | 233 (2026-04-12) |
| Haiku 4.5 | 56.9% | [53.2%, 60.6%] | 73.0% | dma (8%), isr-concurrency (38%), memory-opt (33%) | boot, device-tree, pwm (100%) | 233 (2026-04-12) |
Sonnet 5 vs Sonnet 4.6 — no measurable improvement. On the 232 cases common to both runs (isolating model change from the case-set growth), majority-vote pass@1 is 68.1% → 67.2% = −0.9%p: statistically tied, inside 4.6's own 66.1–70.4% run range. Sonnet 5 is the most stable model measured (stdev 0.29%p) but not more capable on embedded firmware; the weakest categories (dma, isr-concurrency, threading) are unchanged.
Sonnet vs Haiku: 11.1%p overall (CIs don't overlap — statistically significant).
See detailed comparison: docs/BENCHMARK-COMPARISON-2026-04-05.md (§10 = Sonnet 5) · delta report: docs/BENCHMARK-DELTA-sonnet5-vs-sonnet46.md
See analysis & conclusions: docs/LLM-EMBEDDED-CONSIDERATIONS.md
All 23 categories (Sonnet 4.6 vs Haiku, 233 cases, n=3 run, 2026-04-12). Sonnet 5's per-category profile tracks 4.6 closely — same weakest three (dma, isr-concurrency, threading):
Category Sonnet Haiku Gap What it tests
---------------- ------- ------- ------- --------------------------
adc 100% 50% +50%p ADC read, sampling patterns
ble 82% 45% +37%p BLE stack API, connection mgmt
boot 90% 100% -10%p Boot sequence + Kconfig
device-tree 100% 100% 0 DT syntax, node references
dma 31% 8% +23%p Cache alignment, DMA lifecycle
gpio-basic 67% 83% -16%p GPIO config, device_ready() checks
isr-concurrency 23% 38% -15%p ISR safety, volatile, barriers
kconfig 90% 60% +30%p Build config generation
linux-driver 70% 70% 0 Kernel module, syscall interfaces
memory-opt 67% 33% +34%p Memory domains, slab allocators
networking 75% 75% 0 Socket lifecycle, error paths
ota 67% 58% +9%p OTA update lifecycle
power-mgmt 75% 67% +8%p Sleep modes, power domains
pwm 100% 100% 0 PWM duty cycle, device binding (n=1)
security 50% 70% -20%p Crypto API, key management
sensor-driver 75% 67% +8%p Sensor API patterns
spi-i2c 79% 64% +15%p Bus protocol, transfer sequences
storage 54% 31% +23%p Flash lifecycle, NVS patterns
threading 33% 33% 0 Mutex ordering, thread safety
timer 83% 50% +33%p Timer callback safety
uart 33% 67% -34%p UART config, async TX/RX
watchdog 90% 60% +30%p WDT feed timing, reset handling
yocto 80% 70% +10%p Yocto recipe authoring
# Prerequisites: Python 3.12+, uv
pip install uv # if not installed
# Clone and install
git clone https://github.com/Ecro/embedeval.git
cd embedeval
uv sync
# Run benchmark (L0 + L3 static checks, no Docker needed)
uv run embedeval run --model claude-code://sonnet --cases cases/
# View results
cat results/LEADERBOARD.md| Mode | Example | Requirement |
|---|---|---|
| Claude Code (subscription) | --model claude-code://sonnet |
Claude Code CLI installed |
| LiteLLM (API key) | --model anthropic/claude-sonnet-4-20250514 |
ANTHROPIC_API_KEY env var |
| Mock (testing) | --model mock |
None |
# Filter by category or difficulty
uv run embedeval run --model claude-code://sonnet --cases cases/ -c isr-concurrency
uv run embedeval run --model claude-code://sonnet --cases cases/ -d hard
# Multiple attempts for pass@k
uv run embedeval run --model claude-code://sonnet --cases cases/ --attempts 5
# With compiler feedback (self-correction measurement)
uv run embedeval run --model claude-code://sonnet --cases cases/ --feedback-rounds 3
# Multi-turn agent mode
uv run embedeval agent claude-code://sonnet --cases cases/ --max-turns 5
# Bug fix scenario (LLM diagnoses + fixes seeded bugs)
uv run embedeval run --model claude-code://sonnet --cases cases/ --scenario bugfix
# Temporal filtering (contamination prevention)
uv run embedeval run --model claude-code://sonnet --cases cases/ --after-date 2026-01-01
# Include private held-out cases (separate repo)
uv run embedeval run --model claude-code://sonnet \
--cases cases/ --private-cases ../embedeval-private/cases/ --include-private
# Only retest cases changed since last run
uv run embedeval run --model claude-code://sonnet --cases cases/ --retest-only
# Validate all reference solutions pass
uv run embedeval validate --cases cases/
# List cases with metadata
uv run embedeval list --cases cases/
# Prompt sensitivity analysis
uv run embedeval sensitivity claude-code://sonnet --sample 30 --variants 3
# Generate safety guide from results
uv run embedeval guide --results results/Each case is evaluated through five progressive layers. Failure at any layer halts evaluation.
Generated Code
│
▼
┌─────────────┐
│ L0 Static │ checks/static.py — includes, CONFIG symbols, ISR signatures
└──────┬──────┘
▼
┌─────────────┐
│ L1 Compile │ west build / idf.py / arm-gcc (Docker or local, skippable)
└──────┬──────┘
▼
┌─────────────┐
│ L2 Runtime │ native_sim execution with 10s timeout + output validation
└──────┬──────┘
▼
┌─────────────┐
│ L3 Heuristic│ checks/behavior.py — domain-specific pattern analysis
└──────┬──────┘
▼
┌─────────────┐
│ L4 Mutation │ checks/negatives.py — meta-verification (30 cases)
└─────────────┘
| Layer | Method | What it catches | Docker needed |
|---|---|---|---|
| L0 | Static pattern matching | Missing headers, wrong CONFIG, bad ISR signatures | No |
| L1 | SDK compilation | Syntax errors, undefined symbols, type mismatches | Yes (or ZEPHYR_BASE) |
| L2 | QEMU/native_sim execution | Segfaults, deadlocks, wrong output | Yes |
| L3 | Domain heuristic checks | Missing volatile, wrong lock order, no error cleanup | No |
| L4 | Mutation testing | Validates that L0/L3 checks themselves are sound | No |
Default mode (no Docker): L0 + L3 provide strong discriminative power. L1/L2 auto-skip when EMBEDEVAL_ENABLE_BUILD is unset.
See METHODOLOGY.md for detailed architecture diagrams and implementation details.
| Platform | Cases | Build System | Evaluation |
|---|---|---|---|
| Zephyr RTOS (native_sim) | 145 | west build |
Full (L0-L4) |
| Zephyr RTOS (qemu_arm) | 8 | west build |
L0-L3 |
| ESP-IDF | 5 | idf.py build |
L0-L1, L3 |
| STM32 HAL + FreeRTOS | 5 | arm-none-eabi-gcc |
L0-L1, L3 |
| Linux kernel | 8 | kbuild |
L0, L3 |
| Yocto/Embedded Linux | 8 | bitbake |
L0, L3 |
Peripheral & Communication: gpio-basic, uart, adc, pwm, spi-i2c, dma, ble, networking
Concurrency & Timing: isr-concurrency, threading, timer, watchdog
System Configuration: kconfig, device-tree, boot, ota, power-mgmt
Safety & Resources: security, storage, sensor-driver, memory-opt
Platform-Specific: yocto, linux-driver
| Metric | Description |
|---|---|
| pass@1 | First-attempt accuracy (primary metric) |
| pass@k | Unbiased estimator from Chen et al. (2021): 1 - C(n-c,k) / C(n,k) |
| 95% CI | Wilson score confidence interval on pass@1 |
| Embed Gap | EmbedEval pass@1 minus HumanEval pass@1 (negative = harder than general coding) |
Results are sliced by model, category (23), difficulty tier (easy/medium/hard), evaluation tier (sanity/core/challenge), and reasoning type (api_recall, rule_application, cross_domain, system_reasoning).
A single benchmark run generates:
| File | Content |
|---|---|
results/LEADERBOARD.md |
Model comparison, category heatmap, layer pass rates |
results/<model>-results.json |
Full machine-readable report (model name is slug-sanitized) |
results/runs/<date>_<model>/report.md |
Per-case failure analysis with patterns |
results/SAFE_GUIDE.md |
Risk-tier guidance for embedded engineers (auto-generated by run) |
results/TEST_RESULTS.md |
Per-case test status tracker (incremental run history) |
| Mode | What it measures | CLI |
|---|---|---|
| Single-shot | Raw first-attempt accuracy | embedeval run |
| Multi-attempt | pass@k across N samples | embedeval run --attempts 5 |
| Feedback | Self-correction on L0/L1 errors | embedeval run --feedback-rounds 3 |
| Agent | Multi-turn iterative refinement | embedeval agent --max-turns 5 |
| Bug fix | Diagnose + fix seeded mutations | embedeval run --scenario bugfix |
| Context Quality | Effect of team's CLAUDE.md / system prompt | embedeval run --context-pack ./CLAUDE.md + embedeval context-compare |
| Harmful-case triage | Distinguish real regressions from check brittleness | embedeval harmful-inspect --bare X --expert Y |
Context Quality Mode quantifies how much your team's implicit-context
files actually help the LLM and how much room is left to improve them.
context-compare reports per-case effect (H/Hm/F/P) and token cost;
harmful-inspect sub-classifies harmful cases by failure layer so you
know whether to edit the pack (L1+ = likely real) or the static checks
(L0 = likely brittleness). See docs/CONTEXT-QUALITY-MODE.md
for the workflow and metric interpretation.
cases/isr-concurrency-003/
├── metadata.yaml # id, category, difficulty, platform, reasoning_types, ...
├── prompt.md # Task prompt (functional requirements, no safety hints)
├── reference/main.c # Verified correct solution
├── src/main.c # LLM-generated code goes here during evaluation
├── context/ # Additional context files (optional)
├── checks/
│ ├── static.py # L0: required includes, struct layout, ISR signature
│ ├── behavior.py # L3: volatile qualifiers, lock ordering, ISR safety
│ ├── expected_output.txt # L2: expected program output pattern (optional, 119/185)
│ └── negatives.py # L4: mutation tests to validate checks (optional, 30/185)
├── CMakeLists.txt # Zephyr build config
└── prj.conf # Zephyr Kconfig
Design principles:
- Self-contained — each case is a standalone Zephyr/ESP-IDF/STM32 project
- Deterministic — all checks are regex/pattern-based, no LLM-as-judge
- Implicit knowledge — prompts describe what to build, not how to make it safe
- Reference verified — every case has a reference solution that passes all layers
- 48 private cases in a separate repository — never exposed to LLM training data
- Temporal cutoff —
--after-datefilter for training data freshness analysis - Content-hash tracking —
--retest-onlydetects modified cases for efficient re-evaluation
embedeval/
├── src/embedeval/ # Core library (21 modules)
│ ├── cli.py # Typer CLI entry point
│ ├── runner.py # Case discovery, filtering, benchmark orchestration
│ ├── llm_client.py # LiteLLM + claude-code:// + mock providers
│ ├── evaluator.py # 5-layer evaluation pipeline
│ ├── scorer.py # pass@k (unbiased) + Wilson 95% CI
│ ├── reporter.py # JSON, Markdown, failure analysis, safe guide
│ ├── models.py # Pydantic models (EvalResult, BenchmarkReport, ...)
│ ├── check_utils.py # Scope-aware check utilities
│ ├── agent.py # Multi-turn agent evaluation
│ ├── bugfix.py # Bug fix scenario (mutation-based)
│ ├── sensitivity.py # Prompt sensitivity analysis
│ ├── difficulty.py # IRT difficulty calibration
│ ├── ablation.py # Layer contribution ablation study
│ ├── failure_taxonomy.py # Automated failure classification (8 patterns)
│ ├── safety_guide.py # Risk-tier safety guide generation
│ └── test_tracker.py # Incremental retest tracking
├── cases/ # 219 public test cases
├── tests/ # 1277 pytest tests
├── docs/
│ ├── METHODOLOGY.md # Full benchmark methodology + architecture diagrams
│ ├── CONTRIBUTING.md # How to add new test cases
│ ├── LLM-EMBEDDED-CONSIDERATIONS.md # Research insights + practical guidance
│ ├── LLM-EMBEDDED-FAILURE-FACTORS.md # 42-factor failure taxonomy (6 categories)
│ ├── LLM-EMBEDDED-DEVELOPMENT-GUIDE.md # End-to-end workflow + knowledge base
│ └── BENCHMARK-COMPARISON-2026-04-05.md # Haiku vs Sonnet detailed comparison
├── scripts/
│ ├── sync_docs.py # Auto-sync README/METHODOLOGY counts (mandatory at wrapup)
│ └── ... # Other analysis and verification scripts
├── external_benchmarks.yaml # HumanEval/SWE-bench reference scores
├── Dockerfile # Zephyr SDK build environment
├── Dockerfile.esp # ESP-IDF build environment
├── Dockerfile.stm32 # STM32 HAL build environment
└── .github/workflows/ # CI + benchmark dispatch + case validation
uv run pytest # Run all tests
uv run ruff check src/ tests/ # Lint
uv run ruff format src/ tests/ # Format
uv run mypy src/ # Type check
uv run embedeval validate --cases cases/ # Validate reference solutions
uv run embedeval validate-metadata --cases cases/ # Validate metadata consistency
uv run embedeval categories --cases cases/ # Show case counts per category
uv run python scripts/sync_docs.py # Sync README/docs counts (run before commit)Documented in LLM-EMBEDDED-CONSIDERATIONS.md:
- Implicit vs Explicit Gap — 35%p pass rate drop when removing safety hints from prompts
- 4-Level Implicit Knowledge Model — C language → RTOS patterns → Hardware constraints → System safety
- Failure Distribution — Sonnet: 31% L2 + 31% L3. Haiku: 43% L0 + 24% L1. Bigger models fail later (safety, not syntax).
- General vs Embedded — 56% of failures are general SW problems (error paths), 44% are embedded-specific (HW constraints)
- Model Size Sensitivity — Category-level gaps reach up to 50%p (e.g., adc); overall n=3 Sonnet–Haiku gap is 11.1%p (CIs non-overlapping)
- 6 LLM Failure Patterns — happy path bias, semantic mismatch, resource imbalance, order violation, cross-platform hallucination, missing safety guards
- "3AM Paranoia" Layer — 8 categories of field knowledge (timer overflow, flash wear, sensor plausibility, radio corruption) that LLMs structurally cannot learn
- Platform bias — 81% Zephyr, with ESP-IDF and STM32 at 5 cases each
- L3 precision — Static heuristic checks are regex-based; true semantic verification needs L1/L2
- Single-file scope — Cases test single-file code generation, not multi-file project scaffolding
- Difficulty calibration — Assigned labels may not match empirical difficulty (IRT calibration module in
src/embedeval/difficulty.py)
See METHODOLOGY.md for our complete self-assessment.
| Dimension | HumanEval | SWE-bench | EmbedAgent (ICSE'26) | EmbedEval |
|---|---|---|---|---|
| Domain | General Python | Python SWE | Arduino/ESP32/RPi | Embedded (Zephyr/ESP-IDF/STM32/FreeRTOS/Linux/Yocto) |
| Cases | 164 | 2,294 | 126 | 233 |
| Platforms | 1 | 1 | 3 | 6 |
| Verification | assert | pytest | Wokwi sim | 5-layer pipeline |
| Contamination | None | PR-based | HW combos | Separate private repo + temporal |
| Scoring | pass@k | % resolved | pass@1 | pass@k + 95% CI + Embed Gap |
| Unique | — | — | Circuit design | Implicit Knowledge Gap |
vs EmbedAgent (ICSE'26). Both target embedded LLM evaluation, but the focus is different. EmbedAgent measures cross-platform programming on hobbyist boards (Arduino / ESP32 / Raspberry Pi Pico) with Wokwi circuit simulation and bundles Programmer / Architect / Integrator role tasks. EmbedEval measures production embedded firmware on Zephyr RTOS, ESP-IDF, STM32 HAL, FreeRTOS, Linux kernel drivers, and Yocto, deliberately withholds safety hints from prompts (the "implicit knowledge gap"), and verifies through five layers including a mutation-testing meta-layer. They are complementary: EmbedAgent for breadth across hobbyist hardware, EmbedEval for depth on production RTOS/driver patterns.
See ROADMAP.md for the v0.2/v0.3 plan. Open an issue to influence direction. Briefly: v0.2 broadens model coverage and adds FreeRTOS / Linux driver cases; v0.3 adds multi-file scaffolding and cross-platform migration; v1.0 freezes the schema.
See CONTRIBUTING.md for case authoring. In brief:
- New case: create
cases/<category>-<NNN>/withmetadata.yaml,prompt.md,reference/main.c, pluschecks/static.pyandchecks/behavior.py. Verify withuv run embedeval validate --cases cases/ -c <category>. - New model: run
uv run embedeval run --model <litellm-id> --cases cases/for n=3, then PRresults/runs/<date>_<model>/along with an updatedLEADERBOARD.md. - Methodology critique: open an issue using the methodology-question template.
- Interop contract: see HILOOP-HANDOFF.md — producer-side schemas, stability tiers, and breaking-change protocol for every artifact consumed by downstream tools (Hiloop transpile, evidence injection).
Issue templates: model evaluation request, case contribution, methodology discussion (under .github/ISSUE_TEMPLATE/).
If EmbedEval is useful for your work, please cite:
@misc{embedeval2026,
title = {EmbedEval: A Benchmark for LLM-Generated Embedded Firmware},
author = {{EmbedEval Contributors}},
year = {2026},
url = {https://github.com/Ecro/embedeval},
note = {Open benchmark with 267 cases across Zephyr, ESP-IDF, STM32 HAL,
FreeRTOS, Linux kernel drivers, and Yocto. Measures the implicit
knowledge gap in LLM-generated embedded code.}
}Apache 2.0 — see LICENSE.