Skip to content

feat(power-electronics): add reproducible v1 backend - #279

Draft
SoheylM wants to merge 8 commits into
mainfrom
feat/power-electronics-v1-provenance
Draft

SoheylM wants to merge 8 commits into
mainfrom
feat/power-electronics-v1-provenance

Conversation

@SoheylM

@SoheylM SoheylM commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

This draft introduces an explicit PowerElectronics v1 path without changing the published v0 dataset or the package-level v0 default.

The intended v1 benchmark:

  • reuses the exact v0 input designs and train/validation/test membership;
  • does not claim to reproduce the v0 labels;
  • runs ngspice 44.2 in one frozen Linux x86_64 Apptainer image;
  • minimizes abs(dc_gain - 0.25) and Vpp / abs(mean_voltage);
  • retains signed mean voltage and signed gain, peak-to-peak voltage, validity/status fields, and exact source/code/simulator/container provenance.

The generator pins IDEALLab/power_electronics_v0 at revision 5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5, refuses dirty code, uncontainerized execution, noncanonical backend fingerprints, and existing output files by default, and writes each shard plus its manifest atomically. It contains no upload or overwrite path.

The exact original v0 runtime remains unknown. The archived notebook is evidence for Windows/x86_64 and ngspice 36, while a small ngspice 44.2 reproduction on an ARM64 Mac is much closer to the published labels than the matching Linux x86_64 reproduction. Because the operating system, ngspice version, compiler, and build also differ, that comparison is evidence consistent with ARM but not proof of the original architecture. The documentation retains both facts rather than asserting either architecture as certain.

Frozen backend

  • image SHA-256: 40816f203b7e1c68ae37f4d9353bd302486d77988b98733c021f1ff71f48ae02
  • ngspice binary SHA-256: 11a4334ee90509f5edfdceef541711a34a1943d26a14cf0928ac8d5947b72374
  • ngspice version/platform: 44.2, Linux x86_64
  • source dataset revision: 5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5
  • pilot EngiBench commit: 64e6107d44349e67e031cad36633c1af26817784

The SIF, its dependency freeze, source bundle, pilot outputs, manifests, scripts, and logs are preserved on backed-up Euler group work storage rather than purgeable scratch.

Euler pilot results

The final pilot selected 100 evenly spaced source indices from each split and was run twice from isolated directories:

Split Result Signed negative means JSONL SHA-256
train 100 ok 52 24e4eed443da4234bb576a722baa70ddd33acd339790f664926a2b9a16820909
val 99 ok, 1 invalid_measurements (source_index=223) 44 bb677633e4548a1e9ea655f1b6b646ad14225603b79f19933149293af62239ab
test 100 ok 47 679c43afbf033a77add527613c6981ba57cc37a69c1642fb1d5b7fcc1a546482

Both runs produced byte-identical JSONL hashes for all three splits. The same validation row was invalid in both runs, so the non-finite simulator measurement is preserved deterministically rather than dropped or replaced. A strict audit verified all selected indices, row counts, formulas, checksums, source revision, clean code commit, canonical image/binary fingerprints, statuses, and JSON finiteness.

Full Euler generation

After the repeatable pilot, the same frozen commit and image generated eight deterministic shards per split in Euler array job 14690255. All 24 tasks completed with exit code 0:0. Each task used an isolated work/runtime directory and an offline copy of the pinned v0 cache; no Hugging Face token or upload path was present.

The independent full-run audit passed with all 13,824 source indices present exactly once and all formulas, manifests, output hashes, source revisions, code commit, and backend fingerprints verified:

Split Rows Statuses Signed negative means
train 9,676 9,674 ok; 2 invalid_measurements (indices 4,309 and 5,960) 3,992
val 2,765 2,764 ok; 1 invalid_measurements (index 223) 1,168
test 1,383 1,383 ok 579

The SHA-256 of the audit report is 6a489f82b2a99cadf0c1d3e4e8b53202bc2528ff2ac22a8653d06a24e153bf46. The full shards, manifests, audit report, Slurm/audit scripts, and logs are preserved with the SIF on backed-up Euler group work storage.

Published dataset

The audited dataset is published at IDEALLab/power_electronics_v1:

  • immutable release tag: v1.0.0
  • publication commit: eefad7d727ea1e5bef5e1b7088dea20a9b0cd67f
  • 13,824 rows in the original split membership; no row was removed, clipped, or imputed
  • the published SHA-256 manifest validates all 56 staged content files; the manifest itself has SHA-256 68f3c8f9afa380490692f1a4417cb0773b6f0b1aed313ab138cc429b750291cf
  • an authenticated read-back at the publication commit and an unauthenticated clean-cache read-back through v1.0.0 both reproduced the exact split counts, row order, statuses, invalid indices, and per-split semantic hashes

The three invalid simulator measurements remain as rows with status/validity fields and null raw/objective values. Consumers should retain those rows and mask their objective loss. The original IDEALLab/power_electronics_v0 repository and labels were not modified.

Related to #271 and #276. This draft intentionally does not close either issue automatically; checkpoint/leaderboard integration remains out of scope.

Type of change

  • Documentation only change (no code changed)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have run the pre-commit checks with pre-commit run --all-files
  • I have run ruff check . and ruff format --check .
  • I have run mypy engibench
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • The 37 focused PowerElectronics/ngspice tests pass locally
  • The documentation builds; its four warnings are pre-existing and outside this change
  • The full cross-problem suite passes locally. The sandbox run reached 183 passed and 12 skipped; 16 dataset-dependent tests could not access/write the external Hugging Face cache. CI is left to validate those tests in its normal environment.
  • GitHub CI passes on Linux Python 3.10-3.13, macOS Python 3.13, and Windows Python 3.13, along with build, docs, doctests, lint, mypy, and pre-commit.

Reviewer Checklist

  • The content of this PR brings value to the community. It is not too specific to a particular use case.
  • The tests and checks pass (linting, formatting, type checking).
  • The documentation is updated.
  • The code is understandable and commented.
  • There is no merge conflict.
  • Existing v0 results remain unchanged, and the changed semantics are isolated behind problem version v1.
  • The Euler pilot and frozen backend provide an adequate basis for the full dataset run.

@SoheylM
SoheylM requested a review from liangXD523 September 23, 2026 07:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant