From a33c7ae79a463a9b6ffe86627a0244c976cbd950 Mon Sep 17 00:00:00 2001 From: aurascoper Date: Sun, 2 Aug 2026 19:11:44 -0500 Subject: [PATCH] bench(worldmodel): multi-step training works; control still does not move Ledger node 23. Node 22's named next test. BLOCKER FOUND FIRST, and it is a finding: the corpus contains NO TRAJECTORIES. generate() samples a fresh state per record (synthetic_1f.py:104), so record i's post window is unrelated to record i+1's pre window. There was no multi-step ground truth anywhere in the data. Node 22's test was not runnable as specified; it needed generate(..., trajectory_length=K) and train_jepa(..., rollout_steps=K) first. M1 PASS defaults byte-identical, hash 8f74f84ac5aaebe2 -- a flag flip M2 PASS post==pre within a trajectory, not across a boundary M3 PASS oracle ladder bit-identical 100.0/91.5/73.5/37.0/34.5 M5 PASS h=8 rollout 0.02311 -> 0.00546, a 4.2x reduction (5.7x vs K=1 on the same corpus). THE MECHANISM WORKS. M4 FAIL pos 0.9725 against a registered floor of 0.9824. M4 attributed rather than excused, by separating the two changes: corpus K pos vel h=8 single-transition 1 0.9824 0.9740 0.02311 trajectory 1 0.9818 0.9709 0.03116 trajectory 6 0.9725 0.9896 0.00546 The corpus costs nothing; the multi-step loss trades -0.009 pos for +0.019 vel and the rollout gain. A trade, not a regression -- but M4 was a hard floor. AND NODE 22's OUTCOME GATE FAILS. Deployable predictor rollout, 3 seeds: TARGET goal 36.3%, ONLINE goal 36.3%, against zero-action 37.0% and a gate of "approach 76%". Cutting rollout error 5.7x moved control by nothing. REVERTED per the registered rule, without tuning K. Also right on the merits: a corpus mode and two parameters for no demonstrated benefit. Node 20 set the precedent. NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT. It predicted that treating the last untreated condition would approach 76%. It was treated, measurably, and control did not move. Two readings remain, and node 22's 2x2 cannot separate them because its true-env arm used EXACT dynamics rather than merely better ones: (a) the residual gap between 0.00546 and exact zero is still decisive, making control a near-discontinuous function of forward-model error; or (b) a fourth condition is unidentified. Next test named and it adjudicates rather than adds a candidate: sweep control against INJECTED rollout error, interpolating between the true env and the learned predictor across 0, 0.00546 and 0.0231. One sweep, no new model. Ledger-only. Verified: generator and forward_eval smoke tests pass; ledger and ADR gates green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap --- WorldModel/EXPERIMENT_LEDGER.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/WorldModel/EXPERIMENT_LEDGER.md b/WorldModel/EXPERIMENT_LEDGER.md index 6be61fd..4e301ac 100644 --- a/WorldModel/EXPERIMENT_LEDGER.md +++ b/WorldModel/EXPERIMENT_LEDGER.md @@ -434,3 +434,27 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i - Decision: **The control failure has THREE necessary conditions and no sufficient one.** A representation that carries the task and the action (node 21), a goal latent produced by the SAME encoder as the start, and a forward model accurate enough over the planning horizon. Fixing any one leaves control at chance, which is precisely why nodes 6-21 produced null after null while each individual diagnosis was correct. - **This resolves node 14 retrospectively.** Its one-frame remedy was tested and found to do nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken (`pos` 0.65, `vel` 0.033, action unrecoverable) AND the predictor rollout was inadequate. The remedy was necessary all along and could not show it alone. **No single-variable experiment could have found this**, and the ledger ran seventeen of them. - Next question: the deployable configuration still fails — under the predictor rollout the one-frame goal changes nothing (35.5% -> 35.0%), so shipping the frame fix alone would be pointless. The remaining gap is the predictor's multi-step accuracy, now the ONLY untreated condition and no longer a suspect among several. Bounded and ordered: (a) train the predictor on multi-step rollouts rather than single transitions, so h=6 error is optimised rather than inherited from h=1; (b) re-measure the 2x2 — the honest gate is that predictor+one-frame must approach the 76% that true-env+one-frame achieves, not merely beat chance; (c) only if (a) fails, revisit whether latent-space MPC is the right architecture, which is a design decision for the user rather than an autonomous build, per the node-13 stop rule. + +## 23 — Multi-step predictor training: it WORKS, and control still does not move (2026-08-02, commit ) +- Category: Benchmark +- Hypothesis: (node 22) the control failure has three jointly-necessary conditions and only the predictor's multi-step accuracy is untreated. Training the predictor on K-step rollouts should reduce compounding error, and with the one-frame goal the deployable configuration should approach the 76% that true-env + one-frame achieved. +- Prediction: M1-M5 registered before implementing, with M5 (multi-step error must improve) as the point of the change and node 22's 76% as the outcome gate. Registered rule: fail any gate = revert, do not tune K and retry. +- **BLOCKER FOUND FIRST, and it is a finding in its own right: the corpus contains NO TRAJECTORIES.** `generate()` samples a fresh state per record (`synthetic_1f.py:104`), so record i's post window is unrelated to record i+1's pre window. There was no multi-step ground truth anywhere in the data — nothing to train a multi-step predictor against and nothing to validate one on. Node 22's named next test was not runnable as specified. Implementation therefore needed two changes: `generate(..., trajectory_length=K)` emitting linked records with `trajectoryId`/`stepIndex`, and `train_jepa(..., rollout_steps=K)` rolling the predictor K times on its own output and scoring only the K-th latent. +- Evidence: + - **M1 PASS** — `generate(64,"signal",0)` byte-identical to the pre-change generator, hash `8f74f84ac5aaebe2`. The defaults are the incumbent, so the comparison is a flag flip. + - **M2 PASS** — with K=8, post==pre WITHIN a trajectory and NOT across a boundary, both directions asserted. + - **M3 PASS** — oracle ladder bit-identical, 100.0 / 91.5 / 73.5 / 37.0 / 34.5. + - **M5 PASS, decisively — the mechanism works.** h=8 rollout error **0.02311 -> 0.00546**, a 4.2x reduction against the node-21 baseline and **5.7x** against the trajectory corpus at K=1. + - **M4 FAIL.** `pos` recovery **0.9725** against a registered floor of 0.9824. Attributed rather than excused, by separating the two changes: + + | corpus | K | pos | vel | h=8 | + |---|---|---|---|---| + | single-transition (node-21 baseline) | 1 | 0.9824 | 0.9740 | 0.02311 | + | trajectory | 1 | 0.9818 | 0.9709 | 0.03116 | + | trajectory | 6 | **0.9725** | **0.9896** | **0.00546** | + + The corpus change costs essentially nothing; the multi-step LOSS trades -0.009 `pos` for +0.019 `vel` and the 5.7x rollout gain. A trade, not a regression — but M4 was registered as a hard floor and it was not met. + - **AND NODE 22's OUTCOME GATE FAILS.** Deployable configuration, predictor rollout, 3 seeds: TARGET goal **36.3%**, ONLINE goal (one-frame) **36.3%**, against zero-action's 37.0% and a gate of "approach 76%". Cutting rollout error 5.7x moved control by nothing, and the one-frame goal is still worth zero under the predictor. +- Decision: **REVERTED per the registered rule, without tuning K.** `synthetic_1f.py` and `eeg_jepa.py` are unchanged on main; this node is the record. Reverting is also right on the merits: the change adds a corpus mode and two parameters for no demonstrated benefit, and node 20 set the precedent. +- **NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT.** It predicted that treating the last untreated condition would approach 76%. The condition was treated — measurably, 5.7x — and control did not move. Two readings remain and they are distinguishable: either (a) the residual gap between 0.00546 rollout error and the true env's exact zero is still decisive, in which case control is a near-discontinuous function of forward-model error and the 76% arm was only ever reachable with perfect dynamics; or (b) the three-condition model is incomplete and a fourth condition is unidentified. Node 22's 2x2 cannot separate these because its true-env arm used EXACT dynamics, not merely better ones. +- Next question: **measure control as a function of rollout error directly**, rather than adding another candidate fix. Interpolate between the learned predictor and the true env — e.g. roll the true env but inject calibrated noise at each step, sweeping the injected magnitude across the range spanned by 0.00546 (multi-step trained), 0.0231 (incumbent) and 0 (exact). If control collapses somewhere between 0 and 0.00546, reading (a) holds and the requirement is a far more accurate forward model than any training change here will deliver — which is itself a design finding about latent-space MPC. If control stays flat until the error is large, reading (b) holds and there is a fourth condition still unfound. This is one sweep with no new model and it adjudicates a question five nodes have now circled.