Generator contract, shared evaluation layer, and HF-hosted leaderboard

[mkeeler43]

Generator contract, shared evaluation layer, and HF-hosted leaderboard

Replaces 13 near-duplicate evaluate_*.py scripts with one adapter per model
and a shared evaluation package, and makes HuggingFace the single home for
checkpoints, metrics, and the leaderboard.

Large, but split into four commits meant to be reviewed one at a time.
Commit 1 is a pure file move you can skim. The design fits in commit 2.

---

Why

Every model had a training script and an evaluation script. The evaluation
scripts were ~79% identical — evaluate_cgan_cnn_2d.py and
evaluate_gan_cnn_2d.py differed by 26 lines out of 123, and those 26 lines
were only ever four things: which network class to import, how to build it from
run config, whether to pass conditions, and a model-id string.

So adding a metric meant editing 13 files, and adding a model meant copying one.
A leaderboard can't be built on that.

Training scripts are deliberately untouched in structure. Their variation is
real — 340 to 1894 lines of genuinely different algorithms — so unifying them
would relocate complexity rather than concentrate it, and would cost the
single-file readability the repo is built around.

---

Review guide

1 · 646617d — Move model packages under engiopt/generators/ (68 files, skim)

Mirrors engibench/problems/<name>/. Pure move: file contents are unchanged
apart from the import paths the move forces (engiopt.<model> →
engiopt.generators.<model>). Git detects all 47 renames, so git log --follow
still works.

Nothing to review here beyond "yes, that's a move."

2 · 35f1041 — What defines a "generator" (5 files, read this one)

The whole design is here. Today a model is defined implicitly, by whatever its
evaluation script happened to do. This commit makes it explicit: a generator is
six facts and two actions.

The six facts a model states about itself. These are plain values, not code:

Fact	Example	What it buys you
algo_id	"gan_2d"	The name it appears under everywhere — its folder, its leaderboard row, its HuggingFace repo. One name, so those can never drift apart.
conditional	False	Whether it listens to the design requirements it is given. Unconditional models are still given them, so a metric can measure that they ignored them.
design_kinds	("2d",)	Which problems it can serve. Pointing a 2D model at a 3D problem now fails immediately with a clear message, instead of producing garbage.
checkpoint_files	("generator.pth",)	Which files must be present to load it. Missing files fail at load, not halfway through an evaluation.
primary_state_key	"generator"	Where the weights sit inside the saved file.
output_clip	(1e-3, 1.0)	The range its designs must stay in, because several simulators are unstable at exactly zero.

The two actions a model must implement:

• build — given a downloaded checkpoint, reconstruct me. The model
  receives the training run's saved settings and local file paths, and returns
  a working network.
• _sample — given design requirements, produce designs. This is the only
  genuinely model-specific step, and it is where models really differ: one line
  for a GAN, a 1000-step loop for diffusion, one forward pass per pixel for
  PixelCNN++.

Everything else is done once, centrally: finding and downloading the
checkpoint, picking CPU/GPU, choosing which hyperparameter configuration to
load, seeding for reproducibility, timing how long sampling took, reshaping
outputs, and clamping them to the valid range.

The outcome: a new model is ~45 lines, of which ~10 are unique to it. Before
this, adding a model meant copying a 123-line evaluation script and editing
four scattered places in it. The contract mirrors engibench.core.Problem, so
the two halves of the stack are learned once.

engiopt/utils/all_generators.py finds models by walking the folder, exactly as
engibench/utils/all_problems.py finds problems — BUILTIN_GENERATORS sits
alongside BUILTIN_PROBLEMS.

Checkpoint storage also changes here. Each training run is filed under a
short hash of its own hyperparameters:

{problem}/cfg_{fingerprint}/seed_{n}/     one per configuration
{problem}/seed_{n}/                        the "standard" one

The outcome: a 50-configuration sweep produces 50 separately loadable models
instead of 50 runs overwriting one folder. The plain {problem}/seed_{n} path
is claimed only by a run that used the training script's default settings —
so no amount of hyperparameter searching can change what a bare model name
means. W&B is no longer a checkpoint source at all.

3 · a796025 — Shared evaluation layer (28 files, −1942 lines)

Deletes the 13 eval scripts. Adds:

• engiopt/evaluation/ — a metric registry declaring each metric's family,
  cost, and ranking direction; an EvaluationContext that computes the
  expensive optimizer pass once for every performance metric (iog, cog,
  fog, viol share one sweep instead of running four); an Evaluator; and
  the leaderboard, published to HF by merging rather than overwriting.
• engiopt/specs/<problem>/v1.json — frozen, hash-verified evaluation
  contracts, so rows are comparable across people and across time.
• python -m engiopt.evaluate replaces the deleted scripts.
• engiopt/generators/_template/ — copy-me starting point for new models.

Metrics are registered functions, not model methods — a metric compares a
generated set against a reference set under a problem, so it belongs to the
comparison:

@register_metric("mmd", family="distribution", cost="cheap", higher_is_better=False)
def mmd(ctx) -> float: ...

Declared cost means a cheap run provably cannot start a simulator.

4 · 2b9c58d — W&B decoupling + four correctness fixes (29 files, scrutinise this one)

These are behaviour changes, not refactors.

Training / W&B. 10 of 14 scripts only saved checkpoints when --track was
on, so training without W&B silently produced nothing on HuggingFace. VQGAN
additionally gated early stopping — real training logic — behind it, so under
default settings its transformer checkpoint was never written at all. Found by a
live run, not by tests.

iog was not an optimality gap. It stored the raw simulated objective while
cog/fog measured against the reference optimum. Verified with an oracle
generator that returns the reference design itself: it now scores iog ≈ 3e-07
where it previously reported the design's raw compliance.

Condition sampling couldn't serve half the problems. photonics2d declares
solver settings that aren't dataset columns; thermoelastic2d encodes boundary
conditions as 65×65 matrices. Scalar conditions now feed the model tensor while
the simulator still receives every condition — the physics is unchanged.

Multi-objective scores leaked. thermoelastic2d has a per-sample weight
spanning 0.0–1.0. Summing or averaging its three objectives meant a purely
structural sample was scored mostly on thermal compliance:

weight	sum	mean	weighted
1.0 (structural only)	42.31	14.10	0.30 ← correct
0.0 (thermal only)	42.31	14.10	42.00

Scalarization is now declared in the eval spec, and the evaluator refuses to
guess when it's absent.

Gaps ignored objective direction. photonics2d maximizes total_overlap,
so a design that beat the reference scored +0.3 and ranked last. Gaps are now
signed by ObjectiveDirection.

Also removes metrics.metrics() and simulate_failure_ratio() — both dead once
the evaluator landed. (This was the only remaining mypy error in the codebase.)

---

Verification

76 tests. Metric correctness is asserted by properties that must hold for any
correct implementation — mmd(X, X) == 0, collapsed generators score below
varied ones, a purely structural sample is unaffected by thermal compliance —
rather than by pinning historical numbers.

Verified live against HuggingFace with W&B disabled: trained two
hyperparameter configurations, confirmed they landed in separate packages,
loaded each back by fingerprint, evaluated, pushed the leaderboard incrementally,
and confirmed the first model's row was untouched. VQGAN was exercised
separately as the multi-file, multi-stage case. All test repos deleted after.

Ruff clean; mypy clean.

---

Migration notes for reviewers

• The lvae branch will need adapting. Its evaluate_*.py scripts call the
  removed metrics.metrics() and pass W&B artifact arguments that no longer
  exist. That was accepted deliberately rather than maintaining two evaluation
  paths.
• All checkpoints must be retrained, which was already the plan — the storage
  layout changed and there is no W&B read-fallback.
• surrogate_model/ is untouched and sits outside the contract; it's neither a
  generator nor evaluated.

Known limitations

• cog sums across steps and objectives while fog averages objectives — a
  pre-existing inconsistency, left alone rather than changed silently. Worth
  settling alongside the metric-suite work.
• dpp underflows toward zero on collapsed generators (a 1-epoch GAN scored
  8.9e-22), so it may lose ordering at the bottom of a real zoo. A log-domain
  form would fix it.
• Leaderboard submissions are internal for now. The Generator contract is
  the submission interface, so opening it later is a policy change, not a
  rewrite.

---

Update — review round 1 addressed (commits 3a2568e, 810529f)

All 13 of @SoheylM's comments are fixed, each with a threaded reply. Summary comment
below has the full breakdown; the two things worth knowing before re-reading the diff:

1. Evaluation now needs EngiBench from source, and CI pins a commit.
The two specs that would not resolve for Soheyl were not a case of datasets drifting
upstream — they were frozen against EngiBench main, whose photonics2d and
thermoelastic2d read the v1 datasets, while release 0.2.0 still points those two at
v0. On 0.2.0 they sample different data entirely. Specs now record the problem
definition they were frozen against, so that case reports itself instead of surfacing as
a hash mismatch. CI installs EngiBench at a pinned commit; the README says so.

2. Four changes are in the diff that nobody asked for. Flagged so they are not a
surprise: the EngiBench pin above; two extra bug fixes inside the feasibility work
(check_constraints raised on thermoelastic2d, and float64 designs were judged
infeasible against a float32 design space on dtype alone); a pytest job in CI, since the
repo ran only ruff and mypy and the requested spec test had nowhere to live; and the
deletion of two dead helpers this PR had itself added.

Also removed the sweep YAMLs and the per-algorithm notebook that should never have been
committed.

Testing: 136 tests, up from 111. CI green on all four checks.

Known limitation — image conditions

thermoelastic2d models are conditioned on 3 scalars while the simulator receives all 7
conditions, because its four 65x65 boundary masks cannot travel in a dense condition
tensor. Scoring is correct — simulate and optimize get the full conditions — but a
model cannot currently see where the part is held, loaded, or cooled unless it reads
ConditionBatch.dataset itself, as vqgan does for its own preprocessing. First-class
support is being prepared as a follow-up PR; it is purely additive to this contract
(new optional fields, no adapter changes), so it does not block this one.

---

Update — review round 2: making the board survive being public

All seven remaining comments are fixed, with a threaded reply on each. Two of
them turned out to be the tip of a larger problem, so this round also closes the
gap that would have stopped this leaderboard working as a public one.

The board was a self-report system

A row recorded a checkpoint hash but never a repo, and every default
pointed at IDEALLab. An outside contributor could neither publish weights nor
produce a row anyone could trace back to them, so every number was an assertion
nobody could falsify.

Rows now carry checkpoint_repo / checkpoint_path / checkpoint_revision /
checkpoint_hash — an address, not just a fingerprint — and
python -m engiopt.verify follows it: fetch the package at the recorded
revision, confirm it still hashes to what was scored, rebuild through the
registered adapter, score it again. Rows land verified=false and unranked
until a runner reproduces them.

Verification publishes its own numbers, not a pass/fail on the submitted
ones. Sampling from one seed on different hardware genuinely produces different
designs — CUDA and CPU draw different values from the same generator state — so
any tolerance loose enough for honest variation is loose enough for real
fudging. Re-scoring sidesteps that; the submitted value becomes a claim reported
as corroborated or not.

The runner is not privileged. Because the address is public, python -m engiopt.verify with no credentials and no --publish is an audit anyone can
run, which is what keeps the runner honest too.

The protocol is public, so three metrics are copyable by construction

The spec names the scored conditions, and the dataset supplies the optimal
design for each. mmd, iog and fog are therefore defined as closeness to
designs anyone can look up. Measured on the real beams2d spec, against the
trained models in this repo:

	mmd	viol	copy_rate	cond_sens
lookup table	0.000	0.00	1.00	0.0	flagged memorized
cgan_cnn_2d	0.561	0.92	0.00	0.171
gan_cnn_2d	1.049	1.00	0.00	0.0	unconditional, correctly unflagged
vqgan	0.007	0.46	0.02	0.386

A lookup table beats every trained model on both headline metrics. That is not
an implementation flaw and it cannot be fixed by hiding which rows are scored,
since the whole dataset is public. So the board measures retrieval instead:
novelty / copy_rate catch it, and cond_sens catches a model that ignores
conditions it declares it uses.

Both are diagnostic, with no ranking direction, and that is the load-bearing
choice. Ranking on novelty would seat pure noise in first place — zero means
retrieval, but large means only "unlike the data", which a broken model also
achieves. Closing one gaming vector by opening another is not progress. Flagged
rows are published and left out of the ordering: deleting a submission is
moderation, declining to rank one is a statement about what its number measures,
and only the second scales.

cond_sens is not a new feature so much as a claim this PR had already made —
registry.py declared a conditions metric family and core.py referred three
times to "conditional-adherence metrics", with no such metric existing. The
alternative was deleting the claims.

Other ways a score could drift from the thing scored

• required_seeds. An entry must cover seeds 1, 2, 3 to be ranked. A count
  would not do: it still permits running twenty and publishing the best three,
  which turns a median into a maximum while every individual row stays honest.
• ROW_KEY gains checkpoint_repo. Two contributors who both train
  cgan_cnn_2d on default hyperparameters and seed 1 produced the same key and
  different weights, so either one's push would silently replace the other's
  verified row.
• A silent skip. --generators gan_cnn_2d vqgan --config-fingerprints gan_cnn_2d:6293adb3 scored gan_cnn_2d and never mentioned vqgan — the empty
  fingerprint list meant the loop body never ran, producing no row, no load
  attempt and no error. Found by running it, not by a test.

Verification

255 tests, up from 201; CI green on all four checks with no skips.

Exercised against the live Hub rather than fakes: a lookup table scored on the
real beams2d spec (copy_rate=1.00); four published checkpoints loaded and
scored through the contract, including VQGAN's multi-file case; the documented
diffusion_2d_cond --seeds 1 2 3 run produced three rows collapsing into one
entry with n_seeds=3; all four committed specs still reproduce their frozen
digests; and 143 published packages were checked to confirm the new
incomplete-package refusal rejects none of them.

New docs: LEADERBOARD.md — the trust model, the flags, and
the design of the v2 spec that would actually close the copying hole rather
than price it (scoring on off-dataset conditions, which needs an optimizer run
per condition and so is not built here).

Still open, deliberately

Migration scope. Ten of the fourteen registered generators have no published
checkpoint. --list-generators --check-availability now reports that per
generator instead of implying all fourteen are usable — but reporting is not
choosing between migrating them, keeping a fallback, and narrowing the registry.
That is a project decision, so #73 should stay open.

🤖 Generated with Claude Code

[SoheylM]

The generator interface and shared evaluator make sense to me, and removing the duplicated evaluation scripts is a good change. I am requesting changes because several parts of the documented evaluation flow do not work end to end yet. I have left the concrete code issues inline.

There are four cross-cutting points as well:

• Checkpoint migration: Issue #73 says to remove the migration playbook after the W&B migration is complete, but the issue is still open and this PR removes both the playbook and W&B loading. On 2026-08-03, IDEALLab/engiopt-cgan-cnn-2d contained only .gitattributes and README.md; IDEALLab/engiopt-vqgan and IDEALLab/engiopt-gan-cnn-2d did not exist. The README's cGAN evaluation command cannot currently load beams2d/seed_1. Please publish and verify the required HF packages before removing the old path, or keep the migration support until that is done.
• Documentation: The README still describes separate training and evaluation scripts, says the power-electronics optimizer accepts legacy W&B artifacts, and links to the deleted migration playbook. run_pe_optimization.py still uses W&B references in its example and argument descriptions, while Generator.resolve_checkpoint and the evaluation CLI help still list wandb as a model source. These should match the final migration decision.
• Lint CI: With Ruff 0.15.11, ruff check . reports 20 I001 errors on this commit. The GitHub job is green because it runs ruff check --fix in the temporary checkout without checking whether files changed. Please commit the import fixes and run plain ruff check . in CI. Pinning Ruff would also make the check stable.
• Workshop integration: The open DCC workshop branch still imports engiopt.cgan_cnn_2d... in notebook_helpers.py, which is used by the participant notebooks. Please update that branch before these changes reach participants, or keep a temporary import alias if the old path must remain supported.

Verification on the PR head: 76 tests passed with 3 warnings; two of four committed specs failed to load; the thermoelastic adapter shape error and mixed-ranking result were reproduced; the documented cGAN evaluation loaded no model and exited with status 0; git diff --check passed.

[mkeeler43]

Thanks for this — the review caught one genuine correctness bug (the condition count) and
several things that would have quietly corrupted the leaderboard once more than one person
was writing to it. All 13 are addressed, and each has a threaded reply above.

Fixes for your comments

Correctness

• Condition schema unified and recorded per checkpoint; the thermoelastic2d failure is fixed
  and mismatches now raise instead of reshaping wrongly (Add CGAN Model #3)
• Spec digest covers condition names and values, including array-valued ones ([Proposal] Metrics computation #2)
• Specs pin a dataset revision and the problem definition; all four re-frozen and
  resolving, including the two that failed for you (Add GAN 1D implementation #1)
• viol defined for every problem via check_constraints plus a spec-declared volume
  budget (made algos and metrics compatible with eval_model.py files #11)

Leaderboard integrity

• Ranking keeps configurations and spec versions apart (Diffusion 1d #4)
• Download failures no longer read as an empty board (Bad Import engibench.utils.all_problems #5)
• Publishing is conditional on the revision it read, with re-merge on conflict (Fix import #6)
• Rows carry checkpoint revision, weight hash, and code version (Fix problem build #7)

Usability

• --local-model-dir makes the local source actually reachable (Tag artifacts with seed #8)
• Non-zero exit when nothing evaluates (pyproject.toml Specifies engilearn Instead of engiopt #9)
• Device-synced sample timing (Remove occurences of engilearn #10)
• Per-package checkpoint metadata (Feature/surrogate optimization #12)
• Sweep configs removed rather than repaired — they should never have been committed (Add vscode to gitignore #13)

Changes beyond your comments, and why

Four things are in this branch that you did not ask for. Flagging them so nothing in the
diff is a surprise:

1. EngiBench must now come from git, and CI pins a commit. Not a preference — no PyPI
   release has the photonics2d/thermoelastic2d v1 problems, so on 0.2.0 those two specs
   cannot resolve at all. This is the root cause of what you reported in Add GAN 1D implementation #1. See that reply;
   it is the change most worth a second opinion.
2. Two extra bug fixes inside made algos and metrics compatible with eval_model.py files #11. Implementing feasibility surfaced that
   check_constraints raised on thermoelastic2d (HuggingFace returns its boundary
   matrices as lists, which EngiBench's bound checks cannot compare), and that float64
   designs were judged infeasible against a float32 design space on dtype alone — which
   marked that problem's own dataset optima as violations. Both are fixed and tested here
   because viol is wrong without them. The list/array crash may deserve a fix in EngiBench
   too.
3. A pytest job in CI. The repo ran only ruff and mypy, so the spec test you asked for
   in Add GAN 1D implementation #1 had nowhere to live. 136 tests now run on every push (111 before).
4. Two dead helpers deleted — _discover_reference_files (a near-duplicate of live code,
   no callers) and problem_id_of (public, undocumented, unused). Both were added by this
   PR, so this is removing our own dead weight rather than unrelated tidying.

Known limitation, not addressed here

thermoelastic2d models are conditioned on 3 scalars while the simulator receives all 7
conditions, because the four 65x65 boundary matrices cannot travel in a dense condition
tensor. Scoring is correct — simulate and optimize get the full conditions — but a
model cannot currently see where the part is bolted or heated unless it reads
ConditionBatch.dataset itself, as vqgan does for its own preprocessing. Giving image
conditions first-class support in the contract is worth a follow-up; it is a modelling
capability rather than a bug, and it predates this PR.

[SoheylM]

Thank you for the detailed follow-up. Several original issues are fixed, including local loading, exit status, timing, checkpoint metadata, and grouping configurations into separate entries. The remaining inline findings still affect reproducibility or leaderboard correctness, so I am keeping Changes Requested.

The migration and repository cleanup also remain unfinished. Issue #73 is open, the public HF repositories still do not provide the documented checkpoint packages, and the README links to the deleted migration playbook while retaining obsolete W&B source instructions. ruff check . reports 20 import-order errors; CI stays green because it runs with --fix. The DCC workshop branch also still imports the old pre-move module path.

Verification on 810529f: the local suite reports 134 passed and 2 skipped with PyPI EngiBench; all four spec-resolution tests pass when the actual pinned EngiBench source is imported; git diff --check passes; the worktree is clean.

[mkeeler43]

All nine addressed; eight adopted, one half-declined with reasoning (B).

The one that mattered most is A: you were right that CI was green because it skipped
the two specs it exists to verify. My EngiBench pin was a no-op — the git build reports the
same version string as the PyPI build, so pip kept the wrong copy. I had reported this PR
as "CI green" on that basis, which was wrong. It is now a --force-reinstall with an
explicit assertion that the pin took effect, and a spec mismatch fails rather than skips.

Four of these were defects the previous round introduced (C, F, G, and A itself). Two of
the fixes made the code smaller rather than larger:

• viol is now cheap — decoupled from the optimizer pass, so it survives an optimizer
  that rejects an invalid design, and costs a constraint check rather than a full run.
• VQGAN's bespoke preprocessing is deleted, not patched. The contract now replays
  whatever preprocessing a checkpoint recorded, so any two-stage model gets the pathway
  without writing code for it. This also fixed a silent correctness bug: VQGAN was
  refitting its condition normalization on the 50 evaluation rows and discarding the
  training statistics, so the network saw a scale it was never trained on.

Where I would push back — B, second half only. Recording the EngiBench revision is
right and is done. Enforcing it at resolution would refuse to evaluate for any contributor
whose EngiBench differs by a commit, and putting it in ranking identity would partition the
board per build, so two models could not be compared unless scored on byte-identical
EngiBench. Traceability is what the concern justifies, and the revision is recorded in the
spec with code_version already on every row. Happy to add engibench_version to the
provenance columns as well if you want it visible per row — that is the useful half
without the cost. Say the word and I will.

153 tests, up from 136.

[SoheylM]

I rechecked the latest head. The condition handling, VQGAN preprocessing, feasibility, ranking, Hub publishing, and CI spec verification now look correct. The pinned suite passes all 155 tests without skips.

One documentation cleanup from the previous review summary remains. This PR removes the per-model evaluation scripts, removes W&B as a checkpoint source, and deletes docs/checkpoint_migration_playbook.md, but the README still says there are two scripts per algorithm, still lists legacy W&B checkpoint references, and still links to the deleted playbook. run_pe_optimization.py and one Generator.resolve_checkpoint docstring retain the removed W&B source as well. Please update those references. W&B experiment tracking and dashboard documentation can remain; only the checkpoint-loading claims are obsolete.

The documented default HF repository also still has no checkpoint package. Before presenting the evaluation command as runnable, please either publish that package, retain the migration path until issue #73 is complete, or state clearly that pretrained checkpoints are not available yet.

The remaining inline comments are limited to checkpoint identity, replaying training-fitted normalizers, hashless --skip-existing behavior, and runtime EngiBench provenance. I am not requesting further VQGAN work, a visual leaderboard, repository-wide Ruff cleanup, or workshop-branch changes in this PR.

[mkeeler43]

Thanks — all four of the latest round are addressed, each in its own thread. Summary here,
plus the cross-cutting items, plus three findings from verifying against the finished
checkpoint pool that you should know before re-reviewing.

The four inline items

	Item	Outcome
1	metrics.json inside the checkpoint hash	Fixed. Also found a second instance in _discover_package_files
2	Five adapters refitting normalizers at load	Fixed for all five; bounds recorded and replayed
3	Hashless rows treated as a match	Took your version; legacy-row test updated
4	Runtime engibench_version, and the git -C bug	Both done; the sha logic now uses PEP 610 direct_url.json first

Cross-cutting

Checkpoint migration / issue #73. The precondition you set is now met, and I checked
rather than assuming. All four architectures load from HF and score end to end:

beams2d  vqgan        mmd=0.0103  viol=0.36    (98 of 102 runs complete)
beams2d  cgan_cnn_2d  mmd=0.0277  viol=0.38

IDEALLab/engiopt-cgan-cnn-2d and -gan-cnn-2d hold 138 packages each,
-diffusion-2d-cond 139, -vqgan 102, -constrained-plvae-2d 54. So the W&B path is no
longer load-bearing for anything, and I think #73 can close with this PR rather than block
it. If you would rather keep it open until the leaderboard is actually published, that is
reasonable too — but the checkpoints themselves are there and verified.

Lint CI. Confirmed your diagnosis exactly: ruff check . reported 20 I001 on the
old head, and the job was green only because it ran --fix inside the runner. --fix is
gone and the 20 violations are committed. ruff check . is now clean at the un-fixed
setting CI uses.

Documentation. README no longer advertises W&B artifact refs or links the deleted
migration playbook; run_pe_optimization.py's argument docs and usage example are updated;
resolve_checkpoint no longer lists wandb as a source. The README also gained a
leaderboard section, which was previously undocumented — --push-to, --skip-existing,
--attach-metrics, and how an outside contributor submits a model.

DCC workshop branch. Here I would push back on the remedy rather than the observation.
The import break is real, but that branch is being overhauled separately and its notebooks
are changing substantially for IDETC, so a temporary import alias would be dead weight
added to main to serve a branch that will not use it by the time participants see it.
I would rather fix the import as part of that overhaul, and I am happy for that to be a
stated dependency of the workshop rather than of this PR.

Three findings from verifying against the pool

1. Nothing in a sweep writes the canonical path — the documented command is broken.
{problem}/seed_{seed} is what --seeds 1 resolves, and only a default-hyperparameter run
writes it. Every sweep arm sets at least one flag, so cgan_cnn_2d and gan_cnn_2d have
138 packages each and zero canonical packages. The README command fails for both.
(vqgan and diffusion_2d_cond do have them, but only because their center configuration
coincidentally equals the script defaults — luck, not design.)

Two changes: the resolution error now lists what the repo actually holds instead of a bare
"not found", and python -m engiopt.promote_checkpoint promotes an existing arm to the
canonical path rather than spending a few hundred GPU-hours retraining a model we have.

2. A correction to something I said earlier. On 2026-08-06 I reported that VQGAN's
packages held nothing but cvqgan.pth and attributed it to walltime exhaustion. That was
wrong: I sampled the Hub mid-sweep, and VQGAN uploads after each of its three stages, so a
run still training is indistinguishable from one that died. The pool finished at the
original epoch settings, 98 of 102 complete. I had staged an epoch-budget change on the
back of that diagnosis and have dropped it.

The 4 that genuinely did not finish are still the reason the completeness marker is worth
having: they hold vqgan.pth and no transformer.pth, and without package_complete they
look exactly like finished packages until you try to load one.

3. mypy in CI is passing vacuously. Not fixed here, flagging it because it is the
same shape as the two you already caught. The job installs only mypy numpy pytest, so
with torch/gymnasium/diffusers absent the ignore_missing_imports = true overrides turn
every library type into Any. Reproduced both ways:

• CI's exact environment: Success: no issues found in 74 source files
• Same commit with the real dependency stack: 61 errors in 10 files

All 61 predate this PR (63 on the previous head, so this branch reduces it by 2). Fixing it
means either installing the stack in that job or committing to burning down 61 pre-existing
errors, so I have deliberately left it rather than expand this PR further. Worth its own
issue.

Also worth flagging, deliberately not changed

dpp is closer to its floor than it looks. The determinant is a product of n eigenvalues
below 1, so it decays exponentially: on the two real checkpoints above it returned 1.3e-29
and 8.4e-15, and on Gaussian designs at sigma=10 it hits exactly 0.0 from around
n=200, at which point every model ties while the column still looks like a score. Two
training scripts already work around this — cgan_cnn_3d and cgan_vae both log
np.log(max(final_dpp, tiny)).

I had a log_dpp metric written and dropped it from this PR: at the n=50 the specs draw,
dpp is nonzero and ordinally valid, so nothing here is actually broken by it, and adding a
metric no spec references would have shipped dead code into an already large PR. Suggest a
spec v2 adopting it as a separate change — happy to open that if you agree.

Verification

CI on e6231bc: 201 passed, 0 skipped — counts rather than the badge, since the skip
was the failure mode last round; the four spec-resolution tests run for real. Locally:
ruff check . and ruff format --check clean without --fix, pre-commit including pyright
green, worktree clean. Every claim above about the Hub was checked against the live repos
rather than inferred.

[SoheylM]

Thanks for the substantial follow-up. The earlier concerns around fitted preprocessing, null checkpoint hashes, EngiBench provenance, Ruff CI, and obsolete W&B documentation have been addressed. I reran the complete test and quality suite against the pinned EngiBench revision: all 201 tests pass, and Ruff, formatting, pre-commit, and pyright are clean. I also tested the documented evaluation path and checkpoint-package behavior directly.

The remaining inline comments concern the current checkpoint publishing and loading contract, reproducibility metadata, spec creation, and documentation. One cross-cutting migration point also remains: the shared registry exposes 14 generators, but the default HF repositories currently exist for only cgan_cnn_2d, diffusion_2d_cond, gan_cnn_2d, and vqgan. Because this PR removes W&B checkpoint loading repository-wide, please either migrate the remaining registered generators, retain a fallback for the unmigrated ones, or explicitly narrow what the shared evaluator advertises as available. Issue #73 should remain open until the chosen scope is complete.

The README sentence saying that each algorithm usually has separate training and evaluation scripts should also be removed, since evaluation is now shared. I am treating the missing device override on Apple silicon as a follow-up rather than a blocker for this PR.

[mkeeler43]

Round 2 — all seven comments answered, plus the change that lets this board be public

Threaded reply on each of the seven; short version below, with the two that
opened into something larger called out.

Comment	Fix
package_complete only checked while building the message	Checked before file discovery; the old branch became unreachable and was deleted rather than left as dead defensive code
upload_folder retains stale files	Deletes when the upload is the whole package, and deliberately not for a staged one — a blanket delete would have destroyed VQGAN's stage 1
Preprocessing excluded from content_hash	Folded in via identity_metadata, as a denylist so a future output-affecting field cannot be silently missed
freeze_spec cannot set volume_condition / volfrac_tol	Every contract field now settable; you asked for tests and I had missed them — added
code_version has the parent-repository bug	Shared source_checkout_commit(...) helper, used by both sides
README command still fails	Reproduced first; your 825831f6 confirmed and both commands now run verbatim
CONTRIBUTING conditions.dataset guidance + viol	Rewritten; the viol half was only half-done until af6b018

Two of these were symptoms of a bigger gap. A row recorded a checkpoint
hash but never a repo, and every default pointed at IDEALLab — so an
outside contributor could neither publish weights nor produce a row anyone could
trace. Every number was an assertion nobody could falsify. Rows now carry a full
address and python -m engiopt.verify re-runs from it; nothing is ranked until a
runner reproduces it. Separately, the protocol being public means a lookup table
scores mmd=0.000 and viol=0.00 on the real beams2d spec — better than every
trained model here — so the board now measures retrieval (novelty/copy_rate)
and condition-blindness (cond_sens) and declines to rank what trips them.

Full write-up in the PR description, and the trust model in
LEADERBOARD.md.

On the earlier threads

~21 threads from 3 and 6 Aug are still open in the UI. I re-verified each
against current code rather than assuming your summary covered them, and they
are all fixed — spot-checking the ones I trusted least: sweep YAMLs deleted, the
VQGAN adapter no longer refits on the evaluation sample, hash_package_contents
covers cvqgan.pth by listing the directory, local_model_dir reaches both
from_pretrained and the CLI, the device is synchronised before the sample timer
stops, and feasibility is computed independently of optimization. The two you
flagged as hidden by skips now genuinely run: 255 collected, 255 passed, zero
skipped. Happy to bulk-resolve those, or leave them for you.

Still open, and I don't think it's mine to close

Migration scope. Ten of fourteen registered generators have no published
checkpoint. --list-generators --check-availability now reports that per
generator instead of implying all fourteen are usable, but reporting is not
choosing between migrating them, keeping a fallback, and narrowing the registry.
#73 should stay open.

I also did not publish a canonical cgan_cnn_2d package. That is a
checkpoint-publishing decision that overlaps the above, so it seemed yours.

[SoheylM]

The public submission and verification boundary is not complete yet.

The documentation now describes a public leaderboard where outside contributors publish checkpoint-backed rows and an “official runner” later downloads and re-scores them. However, this PR does not define or deploy that runner: ideallab-ci is currently only a value stored in verified_by. There is no workflow, schedule, queue, or specified execution environment. We should not assume any particular future infrastructure.

More importantly, the submitted artifact is not yet safe for unattended execution. Verification reaches generator_cls.build(...), after which adapters deserialize participant-controlled .pth files using torch.load; VQGAN explicitly uses weights_only=False. The revision and content hash establish which bytes were evaluated, but they do not make those bytes safe to load.

I think this PR should establish a submission contract that a future official runner can consume safely, even if deploying that runner remains follow-up work:

• Require safetensors for new external submissions and reject pickle-based .pth/.pt files on the public path.
• Make EngiOpt export safetensors automatically so this does not become a manual participant step.
• Keep reconstruction settings and preprocessing state in validated JSON.
• Accept only registered EngiOpt adapters. New executable model code must still enter through a reviewed code PR.
• Before loading weights, validate the configuration and safetensors header against adapter-specific expectations: tensor names, shapes, supported dtypes, file size, tensor count and total element count.
• Pin the checkpoint repository revision and content hash, as this PR already does.
• Provide a submission mechanism that does not require IDEALLab write access. The current --push-to IDEALLab/engiopt-leaderboard performs a direct upload, despite LEADERBOARD.md saying outside contributors need no IDEALLab access. A pending manifest through a Hugging Face Community PR (create_pr=True) would be one straightforward option.
• Keep participant results verified=false; only the eventual official runner should publish verified scores.

Existing IDEALLab .pth checkpoints can temporarily use a separate, explicit allowlisted legacy path, with a follow-up issue to migrate them. A user-controlled repository or URL must not be able to select that path.

Isolation and runtime controls belong to the eventual runner: no credentials, restricted network access, and CPU/GPU, memory and time limits. Those controls do not have to be deployed in this PR, but they should be tracked before unattended verification is enabled.

Deploying and operating the official runner may remain follow-up work, but the safe submission contract cannot. Every external submission accepted after this PR should already contain safetensors weights, validated JSON configuration and preprocessing metadata, a registered adapter identifier, and an immutable repository revision and content hash. Participant-computed metrics may remain visible as verified=false until the official runner is available, but the submitted package must be directly consumable by that runner without conversion, migration, or resubmission.

[SoheylM]

README cleanup: The sentence “Usually, we provide two scripts per algorithm: one to train the model, and one to evaluate it” describes the architecture removed by this PR. Each generator still has its own training script and adapter, but evaluation is now handled centrally by python -m engiopt.evaluate. Could we replace it with: “Each generator provides its own training script and adapter; evaluation is handled through the shared python -m engiopt.evaluate command.”

[SoheylM]

The migration playbook is being removed before #73 meets its removal criteria.

Issue #73 says docs/checkpoint_migration_playbook.md should be deleted after the historical W&B checkpoint migration is complete. This PR deletes the file while leaving #73 open, and the PR description explicitly says migration scope remains unresolved.

I do not think this PR needs to migrate every historical checkpoint. However, could we either:

1. Restore and update the playbook for the new package layout, leaving Remove checkpoint migration playbook after W&B checkpoint migration is complete #73 open until migration is complete; or
2. Record an explicit decision not to migrate the remaining checkpoints, update the supported-generator documentation accordingly, and close or replace Remove checkpoint migration playbook after W&B checkpoint migration is complete #73.

Deleting the migration instructions while the migration decision remains open leaves no documented route for completing the work.

When documenting availability, please also distinguish missing checkpoints from incompatible generators. For beams2d, only seven registered generators support the problem kind; at the time of this review four have published checkpoints and three compatible families are missing. Counting all fourteen makes unrelated 1D/3D generators look like failed migrations.

[mkeeler43]

Round 3 — two comments fixed, one split into #78

Pushed as 963fc02 and b087001.

Comment	Outcome
README still describes two scripts per algorithm	Fixed — replaced with your wording
Playbook deleted while #73 is open; availability counts unrelated 1D/3D generators	Fixed — decision recorded, listing made kind-aware, #73 closed
Safe submission contract (safetensors, header validation, Community PR, isolation)	Split: the free half is in, the subsystem is #78

The availability report was wrong in the way you said, and slightly worse

You called it as 7 of 14 fitting beams2d, so 3 missing rather than 10. Confirmed, and the
old message compounded it: having counted seven inapplicable models as unpublished, it then
reassured the reader that "they can be trained and evaluated" — false for exactly those
seven, since evaluator.py:161 refuses the pairing outright.

generators_for already answered this and the listing path never called it. Now:

14 generators registered, 7 of them fit beams2d:

  cgan_1d              1d/dict    conditional      does not fit beams2d
  cgan_cnn_2d          2d         conditional      46 package(s)
  cgan_2d              2d         conditional      no published checkpoints
  ...
3 of the 7 generators that fit beams2d have no published checkpoint under IDEALLab:
cgan_2d, gan_2d, pixel_cnn_pp_2d.

Seven fewer Hub requests too, since those repos cannot hold beams2d packages.

#73 — taking your option 2

Closed, with the decision written into docs/checkpoint_layout.md: W&B-era checkpoints are
not being migrated. A migrated checkpoint would arrive without a config fingerprint or a
content hash, so it could be stored but never canonically addressed or verified — a package
no row can be ranked against. Retraining costs less than a provenance story nobody can check.

The submission contract — agreed on the mechanism, split on the scope

Your analysis is right and I have stopped implying otherwise in the docs. A .pth is pickled
Python, unpickling executes what the file carries, and the revision and hash establish which
bytes ran, not that running them was safe.

What is in this PR, because it is free and this branch is what makes a remote package the
only route to a checkpoint: all sixteen load sites now pass weights_only=True. Thirteen
never set it; VQGAN opted into weights_only=False at three sites and did not need to —
its checkpoints hold nothing but state dicts, tensors and scalars. Verified against the Hub
rather than assumed: all nine weight files across the four published packages load unchanged
under the restriction. A test pins it across core.py and every registered adapter.

LEADERBOARD.md gained a Loading someone else's weights section saying what that does
not cover — resource exhaustion, tensors that are not the model, participant-controlled
config feeding build(), and the runner's own blast radius.

What is in #78, and why not here. The argument for landing the format now is that
packages accepted after this PR would otherwise need converting later. That is a migration
cost, and it is currently zero: there are no external submissions, and there cannot be until
a no-IDEALLab-access route exists. The set that would need converting is empty and stays
empty until the board opens. Against that, items 1–5 and 7–8 are a subsystem — safetensors
export across 14 training scripts, per-adapter header validation, a schema layer, a
submission flow, an allowlist — on a PR at 120 files and three rounds. And a submission
format built before the runner that consumes it is a guess at what the runner needs. They
are one piece of work; #78 is that piece, gated on opening the board.

I also fixed the contradiction underneath your Community-PR point rather than building the
flow: LEADERBOARD.md claimed "Nothing here needs write access to IDEALLab" while
--push-to does a direct upload_file to the board. True of the checkpoints, false of the
row. The doc now says which is which, and names the self-service route as tracked in #78.

Verification

Run in a clean worktree, because my working tree has untracked generator directories that
fail test_every_generator_directory_has_an_adapter locally; all of them have zero tracked
files, so CI never sees them.

• 260 passed, 0 failed, 0 skipped — up from 255, five new tests. Counts, not the badge.
• ruff check . and ruff format --check clean, verbatim as CI runs them, no --fix.
• pre-commit including pyright green on both commits.
• mypy with the real dependency stack: 64 errors before, 64 after, none introduced. Still
  pre-existing and still passing vacuously in CI, as flagged last round.
• --list-generators --check-availability run against the live Hub; output above.
• The weights_only claim checked by downloading and loading real packages.

One limit worth stating: that check covered one package per published generator on beams2d,
not all 189. I read the save side of every training script — all dicts of scalars and
state_dict()s — but did not load every package.

Still open, deliberately

Migration scope. Unchanged in substance, but the reporting is now honest about it:
3 generators fit beams2d and lack weights, not 10, and the non-migration decision is
written down rather than implied. Narrowing the registry further is still a project call.

[SoheylM]

Thanks, Matthew. I completed another review of b087001. The previous correctness, checkpoint integrity, availability reporting, and CI concerns now look addressed. I also reran the full suite locally: 260 tests pass, and Ruff and the formatting check are clean.

Splitting the safe external-submission contract into #78 is reasonable now that this PR clearly keeps submission internal and #78 is explicitly required before accepting external checkpoints.

I have left two final documentation comments: one correcting who can currently publish, and one clarifying the reason for not migrating historical W&B checkpoints. Once those are addressed, I think we can resolve the remaining review threads and approve the PR.

[mkeeler43]

Both documentation comments are addressed in dfeb373.

32670ba is a consistency pass over every document this branch ships, so the second commit touches more files than the two comments alone would. Each item was checked against the code before changing the text:

• CONTRIBUTING_A_MODEL.md's build() example loaded a checkpoint without weights_only=True, contradicting what every adapter now does and what tests/ pins; and it used checkpoint_identity without importing it.
• The PR template still asked for evaluate_<algo>.py and for restoring models from W&B artifacts. It now asks for the Generator adapter and the HuggingFace package.
• LEADERBOARD.md said "the mechanism this PR adds", and described the self-service route without naming Safe submission contract before the leaderboard accepts external checkpoints #78. Both fixed.
• cgan_cnn_3d/README.md listed --lr_enc; that model has no encoder.
• README: "thirteen times" (the registry has fourteen), two sentences that read as migration notes, and the EngiBenc typo.
• American spelling in the new docstrings and test comments; one dash style per document; engiopt/.DS_Store untracked and ignored.

The README's live claims were re-verified against the Hub today: engiopt-cgan-cnn-2d holds 46 beams2d packages and no canonical path; diffusion_2d_cond and vqgan do have canonical packages.

Verification, in a clean worktree: 256 passed / 4 deselected (-m "not network"; nothing network-facing changed), ruff format --check and pre-commit including pyright clean, git diff --check clean. No code logic changed in either commit.

[SoheylM]

Thanks, Matthew. I reviewed the latest head (32670ba). Both final documentation comments are addressed, and the additional consistency updates match the current implementation.

The earlier correctness, checkpoint integrity, availability reporting, migration scope, and CI concerns are now resolved. The safe external-submission contract is clearly gated in #78 before external checkpoints can be accepted.

I reran the complete local suite: 260 tests pass, and Ruff, formatting, pre-commit, and pyright are clean. All GitHub checks are also green. I have no further changes to request and am approving the PR.