Define simulator provenance and backend policy for PowerElectronics

[SoheylM]

Problem

PowerElectronics datasets, surrogate models, optimization runs, and final
ngspice verification can currently use different numerical backends without
recording or checking that difference.

This matters because #269 found materially different transient results across
ngspice versions and between x86_64 and ARM64. A model could therefore be
trained on labels produced by one backend and evaluated against another. The
result would look like model error even when part of the discrepancy comes from
the simulator.

What we know about the current dataset

The current dataset is
IDEALLab/power_electronics_v0,
at revision 5c4adb2ec5cfc71794988b1297a7ff8ffe59daa5.

The Hugging Face dataset card records the schema and split sizes, but not the
simulator version, executable/build, operating system, CPU architecture,
EngiBench revision, netlist checksum, or output transformation.

Repository history does provide useful provenance:

• The dataset-generation notebook at a80feee ran
  ngSpice64/bin/ngspice.exe from a Windows path.
• The bundled distribution contained ngspice-36-manual.pdf, so the published
  labels appear to have been generated with ngspice 36 on Windows/x86_64.
• The notebook applied np.abs() to both gain and ripple before writing the
  dataset. The current simulation path returns the signed values parsed from
  ngspice directly.
• The three documented groups of 4,608 samples total 13,824 rows, matching the
  current Hugging Face train/validation/test total.

This means the published dataset is x86_64-derived, but it does not use the
same backend currently pinned by #269 (ngspice 44.2 x86_64). It also contains
an output transformation that is not represented in the dataset metadata or
the current simulation API.

As a small check, I re-simulated the first three published test designs on an
Apple M3 using both native Homebrew ARM64 ngspice 44.2 and the validated
x86_64 ngspice 44.2 build from #269 through Rosetta:

Row	Published values	x86_64 44.2 raw result	ARM64 44.2 raw result
0	(0.0576707, 0.1652683)	(0.01467894, 1.186994)	(0.05773014, 0.1617389)
1	(0.612778, 0.157394)	(-0.149966, -0.332231)	(-0.610211, -0.158559)
2	(0.09850643, 0.8131018)	(0.05978021, 0.1174428)	(0.09831222, 0.8146075)

The published-label comparison also includes a version and operating-system
difference because those labels used ngspice 36 on Windows. The x86_64/ARM64
comparison holds the ngspice version and EngiBench code constant, but the two
binaries have different architecture/build fingerprints. The results confirm
that neither the dataset nor two binaries with the same ngspice version can be
treated as interchangeable. The check also exposes the undocumented
absolute-value transformation.

Proposed work

1. Audit and document power_electronics_v0

• Confirm the historical simulator version, architecture, operating system,
  netlist, solver/build configuration, and output transformation.
• Re-run a representative sample with the historical Windows/x86_64 ngspice 36
  executable if possible.
• Add the recovered provenance and the exact dataset revision to the Hugging
  Face card and EngiBench documentation.
• Explicitly document whether DcGain and Voltage_Ripple are signed or
  absolute quantities.

2. Define a canonical backend policy

Choose one simulator fingerprint for official PowerElectronics dataset
generation, optimization, baselines, and final verification. A fingerprint
should include at least:

• ngspice version and executable checksum;
• CPU architecture;
• operating system/build configuration and linear solver;
• EngiBench commit and netlist checksum;
• output post-processing semantics.

Given #269 and the existing CI/reference setup, ngspice 44.2 x86_64 is a
reasonable candidate. If that becomes canonical, the ngspice 36 dataset should
remain an immutable v0, and a regenerated dataset should be published as a
new version rather than silently replacing it.

3. Prevent mixed-backend evaluation

• Record the dataset ID, dataset revision, and simulator fingerprint in trained
  surrogate checkpoints and evaluation outputs.
• Make optimization and simulation-based evaluation use the dataset's canonical
  backend by default.
• Warn or fail clearly when a user attempts to verify a model with a different
  backend. An explicit override can remain available for comparison studies.
• Add a small conformance set whose expected values are tied to the canonical
  backend and run it in CI.

4. Decide how to handle ARM64

Two policies seem defensible:

1. Publish one canonical x86_64 dataset and require the validated x86_64
   ngspice build for official optimization and verification, including through
   Rosetta on Apple Silicon.
2. Publish separate, clearly named dataset variants such as
   ngspice-44.2-x86_64 and ngspice-44.2-arm64, each with its own provenance
   and reference values.

If both variants are hosted, they should never be combined under one dataset
configuration or treated as interchangeable. Training on one and verifying on
the other should be an explicit cross-backend experiment, not the default
workflow.

Acceptance criteria

• The provenance and output transformation of power_electronics_v0 are
  documented.
• EngiBench has a documented canonical backend policy for PowerElectronics.
• Dataset versions/configurations identify their simulator fingerprint.
• Surrogate checkpoints and evaluation results record the dataset revision and
  simulator fingerprint.
• Official training, optimization, and verification cannot silently mix
  incompatible backends.

Related: #249, #269 and
ngspice bug #622.

[linear]

SOH-138

[markfuge]

This issue will be targeted to the documentation on the webpage. This can be:

• a notice at the top of the PowerElectronics page noting the architecture differences.
• a warning message when loading PowerElectronics that there may be differences between the dataset and the current running architecture.
• Relabel HF dataset noting it is ARM specific.

[SoheylM]

I’d like to tackle #271 and #276 together, starting with a small pilot on Euler.

My proposal is to keep the existing v0 dataset unchanged and generate a new dataset using ngspice 44.2 on Linux x86_64, with a fixed container environment. We would reuse the same input designs and train/validation/test splits so we can compare the results directly. We should also document what we know about v0’s original setup and mention that we suspect it was ARM-based, although we can no longer confirm the original environment.

There is currently a conflict between the documentation and the code: the documentation describes minimizing gain error and voltage ripple, while the code minimizes the gain itself and maximizes ripple. To resolve this, I suggest:

• Gain error: abs(gain - 0.25), minimized.
• Relative voltage ripple: Vpp / abs(mean_voltage), minimized.

Changing these definitions is a separate decision from changing the simulator environment. If we agree on both, a new power-electronics problem version would make that distinction clear. We would also keep the original simulation measurements, including negative values, alongside the calculated objectives, simulation status, and exact code and simulator versions.

I’d keep the EngiBench PR focused on the necessary power-electronics changes, tests and documentation.

After the pilot, we would incorporate the relevant changes from #272 and freeze the simulation code before the full run. Any broader checkpoint or leaderboard changes could be tracked separately.

Does this scope make sense, particularly the proposed objective definitions?