Make the metrics suite complete and usable from one object

[mkeeler43]

What this is

After #75 the evaluation layer on main registered eight metrics and no way to score several models at once without building an EvaluationContext by hand. The full suite lived on feat/dataset-sensitivity-fields as ~42 separately maintained functions with prefixed names (pca_mmd, lv_vendi, iog_median, ...). This PR brings the suite to main on a contract simple enough to explain in one sentence, and adds the object you use it through.

Nothing here touches the LVAE, the baselines, or the pool tooling. Those are separate PRs.

How you use it

from engiopt.evaluation import Board, METRICS

METRICS.explain()                                     # every metric: the question it answers, direction, cost

board = Board(problem, reference=REF, train=TRAIN)    # saved designs
board.evaluate(models, space="pca", aggregation="median")
board.explain()                                       # who each column picks, and what real designs score
board.rank("mmd")

Board.load("beams2d", ["cgan_cnn_2d", "vqgan"], expensive=True)   # published checkpoints by name

example_metrics_suite.ipynb at the repo root walks all of this on a toy problem in ten code cells. It runs in seconds with no downloads.

The contract

A metric is a function of an EvaluationContext plus five declarations: name, family, cost, direction, and (rarely) pixel_only. The first line of its docstring is the question the column answers, and that line is what explain() and --list-metrics print. higher_is_better=None means a diagnostic: read it beside the other columns, never rank on it.

Two things that used to be baked into metric names are now arguments to Board.evaluate:

• space — where the question is asked. pixel and pca are built in; a learned latent space registers one projection function with register_space. Metrics that need the actual designs (a constraint check, a copy corpus) declare pixel_only and are skipped elsewhere. This replaces the pca_*/lv_*/lvoff_*/pixel_* prefixes.
• aggregation — how one-value-per-design metrics collapse to a number. mean or median, declared on the spec and recorded in every row. This replaces the *_median columns.

The metrics

Twenty, down from forty-two. Renamed so the name says what is measured:

Family	Metrics
distribution	mmd, coverage
diversity	vendi, dpp
conditions	cond_sens, per_condition_distance (was paired_distance), volume_error (was cond_err)
memorization	train_distance, copy_rate (together, was novelty)
feasibility	viol
performance	iog, cog, fog, calls_to_settle, gap_after_calls, reaches_reference_rate, first_call_gain
cost	generation_seconds, n_parameters, train_minutes

dpp keeps its name but is now the n-th root of the kernel determinant, bounded in (0, 1]. The raw determinant that v1 rows hold reads around 1e-20 on every real board and cannot be compared with this column. calls_to_parity is gone: parity is measured against the reference optimum, so a copier reaches it in zero calls.

The board also scores one random half of the reference designs against the other half and shows that as a row. That is what real, correct designs score on each column on this problem, and it is the only comparison point that is measured rather than assumed.

Specs

All four problems get a v2.json with the full metric list and the aggregation policy; v2 is now the default. v1.json stays committed so rows published under it can be read. A v1 spec still loads, but evaluating under it raises Unknown metric 'novelty' at metric selection, because that metric no longer exists. Nothing on the IDEALLab/engiopt-leaderboard dataset references v1 (it has no rows).

Also in here

• OptimizationResults.trajectories (each design's gap at every optimizer call), EvaluationContext.reduce, model_params, train_minutes, train_designs.
• register_metric silently registered into the global registry when handed an empty custom one, because MetricRegistry is a Mapping and an empty mapping is falsy. Fixed.
• "brief" is replaced by "conditions" throughout, including three docstrings from Generator contract, shared evaluation layer, and HF-hosted leaderboard #75.
• LEADERBOARD.md, README.md and CONTRIBUTING_A_MODEL.md describe the suite that exists now.

What I checked

• pytest tests/ -m "not network": 298 passed, 0 skipped, 1 failed. The failure is test_every_generator_directory_has_an_adapter, which trips on six leftover __pycache__-only directories in my working copy from branch switching. They are not in git and cannot exist in CI.
• python -m engiopt.evaluate --list-metrics runs as documented and prints the table above.
• The notebook executes end to end with no errors or warnings.
• Every pre-commit hook (pyright, codespell, ruff) passed on each commit.
• New tests: tests/test_metrics_suite.py (19), tests/test_board.py (8), tests/test_metric_registry.py (4). The trajectory metrics are checked against hand-built paths with known answers.

What I did not check

• Nothing was run against a real EngiBench problem or the Hub. Board.load calls Evaluator.for_problem and Generator.from_pretrained with the signatures they have on main, but has not been exercised end to end. The notebook is a toy problem by design.

Follow-ups

• Bootstrap confidence intervals beside each value (evaluate(bootstrap=...)). Planned, small, not in this PR.
• Baselines (knn_retrieval, deconv_regression, the planted constructions): separate PR.
• space="lv": registered by the LVAE PR once the Lipschitz fixes land.

🤖 Generated with Claude Code