Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions WorldModel/EXPERIMENT_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,3 +434,27 @@ Two reviewers verified the delivered synthetic A/Bs are sound (symlog threaded i
- Decision: **The control failure has THREE necessary conditions and no sufficient one.** A representation that carries the task and the action (node 21), a goal latent produced by the SAME encoder as the start, and a forward model accurate enough over the planning horizon. Fixing any one leaves control at chance, which is precisely why nodes 6-21 produced null after null while each individual diagnosis was correct.
- **This resolves node 14 retrospectively.** Its one-frame remedy was tested and found to do nothing — correctly measured, and the right conclusion given what was then known. It did nothing because the representation was broken (`pos` 0.65, `vel` 0.033, action unrecoverable) AND the predictor rollout was inadequate. The remedy was necessary all along and could not show it alone. **No single-variable experiment could have found this**, and the ledger ran seventeen of them.
- Next question: the deployable configuration still fails — under the predictor rollout the one-frame goal changes nothing (35.5% -> 35.0%), so shipping the frame fix alone would be pointless. The remaining gap is the predictor's multi-step accuracy, now the ONLY untreated condition and no longer a suspect among several. Bounded and ordered: (a) train the predictor on multi-step rollouts rather than single transitions, so h=6 error is optimised rather than inherited from h=1; (b) re-measure the 2x2 — the honest gate is that predictor+one-frame must approach the 76% that true-env+one-frame achieves, not merely beat chance; (c) only if (a) fails, revisit whether latent-space MPC is the right architecture, which is a design decision for the user rather than an autonomous build, per the node-13 stop rule.

## 23 — Multi-step predictor training: it WORKS, and control still does not move (2026-08-02, commit <pending>)
- Category: Benchmark
- Hypothesis: (node 22) the control failure has three jointly-necessary conditions and only the predictor's multi-step accuracy is untreated. Training the predictor on K-step rollouts should reduce compounding error, and with the one-frame goal the deployable configuration should approach the 76% that true-env + one-frame achieved.
- Prediction: M1-M5 registered before implementing, with M5 (multi-step error must improve) as the point of the change and node 22's 76% as the outcome gate. Registered rule: fail any gate = revert, do not tune K and retry.
- **BLOCKER FOUND FIRST, and it is a finding in its own right: the corpus contains NO TRAJECTORIES.** `generate()` samples a fresh state per record (`synthetic_1f.py:104`), so record i's post window is unrelated to record i+1's pre window. There was no multi-step ground truth anywhere in the data — nothing to train a multi-step predictor against and nothing to validate one on. Node 22's named next test was not runnable as specified. Implementation therefore needed two changes: `generate(..., trajectory_length=K)` emitting linked records with `trajectoryId`/`stepIndex`, and `train_jepa(..., rollout_steps=K)` rolling the predictor K times on its own output and scoring only the K-th latent.
- Evidence:
- **M1 PASS** — `generate(64,"signal",0)` byte-identical to the pre-change generator, hash `8f74f84ac5aaebe2`. The defaults are the incumbent, so the comparison is a flag flip.
- **M2 PASS** — with K=8, post==pre WITHIN a trajectory and NOT across a boundary, both directions asserted.
- **M3 PASS** — oracle ladder bit-identical, 100.0 / 91.5 / 73.5 / 37.0 / 34.5.
- **M5 PASS, decisively — the mechanism works.** h=8 rollout error **0.02311 -> 0.00546**, a 4.2x reduction against the node-21 baseline and **5.7x** against the trajectory corpus at K=1.
- **M4 FAIL.** `pos` recovery **0.9725** against a registered floor of 0.9824. Attributed rather than excused, by separating the two changes:

| corpus | K | pos | vel | h=8 |
|---|---|---|---|---|
| single-transition (node-21 baseline) | 1 | 0.9824 | 0.9740 | 0.02311 |
| trajectory | 1 | 0.9818 | 0.9709 | 0.03116 |
| trajectory | 6 | **0.9725** | **0.9896** | **0.00546** |

The corpus change costs essentially nothing; the multi-step LOSS trades -0.009 `pos` for +0.019 `vel` and the 5.7x rollout gain. A trade, not a regression — but M4 was registered as a hard floor and it was not met.
- **AND NODE 22's OUTCOME GATE FAILS.** Deployable configuration, predictor rollout, 3 seeds: TARGET goal **36.3%**, ONLINE goal (one-frame) **36.3%**, against zero-action's 37.0% and a gate of "approach 76%". Cutting rollout error 5.7x moved control by nothing, and the one-frame goal is still worth zero under the predictor.
- Decision: **REVERTED per the registered rule, without tuning K.** `synthetic_1f.py` and `eeg_jepa.py` are unchanged on main; this node is the record. Reverting is also right on the merits: the change adds a corpus mode and two parameters for no demonstrated benefit, and node 20 set the precedent.
- **NODE 22's THREE-CONDITION MODEL IS NOW IN DOUBT.** It predicted that treating the last untreated condition would approach 76%. The condition was treated — measurably, 5.7x — and control did not move. Two readings remain and they are distinguishable: either (a) the residual gap between 0.00546 rollout error and the true env's exact zero is still decisive, in which case control is a near-discontinuous function of forward-model error and the 76% arm was only ever reachable with perfect dynamics; or (b) the three-condition model is incomplete and a fourth condition is unidentified. Node 22's 2x2 cannot separate these because its true-env arm used EXACT dynamics, not merely better ones.
- Next question: **measure control as a function of rollout error directly**, rather than adding another candidate fix. Interpolate between the learned predictor and the true env — e.g. roll the true env but inject calibrated noise at each step, sweeping the injected magnitude across the range spanned by 0.00546 (multi-step trained), 0.0231 (incumbent) and 0 (exact). If control collapses somewhere between 0 and 0.00546, reading (a) holds and the requirement is a far more accurate forward model than any training change here will deliver — which is itself a design finding about latent-space MPC. If control stays flat until the error is large, reading (b) holds and there is a fourth condition still unfound. This is one sweep with no new model and it adjudicates a question five nodes have now circled.
Loading