Publish Sonnet/Terra PlanBench results and assessment

.github/workflows/ci.yml

@@ -39,6 +39,8 @@ jobs:
         run: python benchmarks/site/tests/test_publication.py
       - name: Check paired benchmark statistics and missingness
         run: python benchmarks/planbench/test_score.py
+      - name: Check benchmark role routing and retry accounting
+        run: python benchmarks/planbench/test_runner.py
 
   test:
     strategy:

benchmarks/planbench/README.md

@@ -1,5 +1,25 @@
 # Bounded PlanBench measurement
 
+## Sonnet/Terra replication
+
+The new profile uses Sonnet as author/self-critic/reviser and Terra as external
+critic, with Claude280/Codex56 caps. `run.py` now accepts author/reviewer/grant
+selection; frozen manifests control effort, role concurrency and retry limits.
+Claude's combined response allowance is passed per call rather than shared
+mutable state. Original Astra/Fable behavior remains the default profile.
+
+The recorded Sonnet/Terra run had a readiness-only medium-effort amendment;
+`reprofile_readiness.py` preserves six spent calls and allocates only the
+remaining330. Do not reinitialize a settled run or repeat its high-effort
+readiness failure. The completed medium stage produced200 scored outputs:
+48/50 original and50/50 for plain revision, self-review and Terra review.
+Use its manifest and profile amendment as the authoritative run settings.
+
+The generic scorer supports both model pairs. `publish_report.py` accepts
+separate data/page/study IDs and an optional reviewed assessment tied to the
+source report hash. The site retains the Astra/Fable result alongside the
+Sonnet/Terra result instead of pooling or overwriting them.
+
 The September 13, 2026 attempt used the official PlanBench Blocksworld Hard
 corpus, pinned at `fc638a1aff7df3fe7a1a1d289fa2c04cc24dc284`, and its bundled
 VAL executable and PDDL extractor. The scored sample is 50 fixed tasks from

benchmarks/planbench/assessments/sonnet-terra-20260914.json

@@ -0,0 +1,11 @@
+{
+  "source_report_sha256": "937f437e1f56d4fdd417ec96bb3168afe6db9d5e9b7373f0025a66bd16a0ccc0",
+  "assessment": {
+    "question": "Does Terra review improve Sonnet plans beyond plain revision or independent Sonnet self-review on the same fixed PlanBench subset?",
+    "finding": "Sonnet originals scored 48/50 (96%). Plain revision, Sonnet self-review, and Terra review followed by Sonnet revision each scored 50/50 (100%). The two original plans executed legally but failed to satisfy the complete goal; all three revision approaches fixed them. Terra review added 0 points over either revision control and 4 points over the original draft.",
+    "test_quality": "All 200 scored plans were produced and evaluated after generation froze. The same pinned 50 tasks, original drafts, role prompts, upstream PDDL extractor and official VAL were used across arms. The two failed originals were goal failures, not parsing errors. Model routing, budget allocation, retry accounting, and exact exported plan hashes were checked; no validator feedback reached participants.",
+    "limitation": "The 96% original score is still ceiling-limited, and both controls reached 100%, leaving no room for a positive D-C effect. Fifty tasks and one response per arm cannot establish general equivalence. The [0,0] paired bootstrap interval resamples ties, not population certainty. Before any scored task, a readiness amendment changed both models from high to medium effort and gave Claude 8,192 combined reasoning/response tokens; six initial calls remain charged and excluded. Comparisons with the earlier high-effort Astra run are not model-only. Validity does not measure optimality, human rework, or software delivery.",
+    "decision": "Use plain Sonnet revision for this benchmark: it achieved the same 100% score at an estimated $0.0552 per task versus $0.0655 with Terra review. Terra review cost about 19% more than plain revision. It was about 20% cheaper than Sonnet self-review ($0.0821), so it was the more economical reviewer here, but it did not outperform the cheaper plain-revision workflow. The predeclared score-improvement threshold was not met.",
+    "next_action": "Do not spend another full run on this saturated subset. If continuing to measure accuracy gains, specify a harder recognized benchmark or task stratum before running, keep matched revision controls, and retain cost and coverage reporting. Treat the observed reviewer cost advantage separately from claims of better planning quality or productivity."
+  }
+}

benchmarks/planbench/publish_report.py

@@ -4,15 +4,24 @@
 from statistics import mean,median
 from datetime import datetime,timezone
 
-def stage(root,site):
+def stage(root,site,data_name='planbench-results.json',page_name='planbench.html',study_id='planbench',assessment_path=None):
     root=Path(root);site=Path(site);public=site/'public'
     result=json.loads((root/'report.json').read_text(encoding='utf-8'))
+    profile_path=root/'PROFILE-AMENDMENT.json'
+    if profile_path.is_file() and not result.get('profile_amendment'):
+        profile=json.loads(profile_path.read_text(encoding='utf-8'))
+        frozen=json.loads((root/'manifest.json').read_text(encoding='utf-8'))
+        assert hashlib.sha256(profile_path.read_bytes()).hexdigest()==frozen['profile_amendment_sha256']
+        result['profile_amendment']=profile
+        result['assessment']['limitation']+=' A readiness-only amendment changed both models from high to medium effort and gave Claude 8,192 combined reasoning/response tokens before any scored task. The six initial smoke calls remain charged and excluded. This is not a model-only comparison against Astra at high effort.'
     assert result['completion'] in ('complete','partial','readiness-failed')
-    assert result['spend']['calls']<=336 and result['spend']['families']['codex']<=280 and result['spend']['families']['claude']<=56
+    caps=result['spend'].get('family_caps',{'codex':280,'claude':56})
+    assert result['spend']['calls']<=336 and all(result['spend']['families'][f]<=caps[f] for f in ('codex','claude'))
+    labels=result.get('model_labels',{'author':'Astra','reviewer':'Fable'});author=labels['author'];reviewer=labels['reviewer']
     spec=importlib.util.spec_from_file_location('gate',site/'validate-publication.py');gate=importlib.util.module_from_spec(spec);spec.loader.exec_module(gate)
     archive=public/'archive/2026-09-13-planbench-readiness';archive.mkdir(parents=True,exist_ok=True)
     manifest=json.loads((site/'archive-manifest.json').read_text(encoding='utf-8'))
-    for name in ('planbench-results.json','test-evaluations.json'):
+    for name in (('planbench-results.json','test-evaluations.json') if study_id=='planbench' else ()):
         dest=archive/name
         if not dest.exists():
             original=json.loads((public/name).read_text(encoding='utf-8'))
@@ -32,6 +41,8 @@ def stage(root,site):
     receipts=[result['timing']['controller_receipt']]
     previous=root/'previous-evidence/controller.exit.json'
     if previous.is_file():receipts.append(json.loads(previous.read_text(encoding='utf-8-sig')))
+    prior_readiness=root/'prior-readiness/controller.exit.json'
+    if prior_readiness.is_file():receipts.append(json.loads(prior_readiness.read_text(encoding='utf-8-sig')))
     source_name=(result.get('continuation') or {}).get('source_run')
     if source_name and Path(source_name).name==source_name:
         initial=root.parent/source_name/'original-evidence/controller.exit.json'
@@ -49,16 +60,24 @@ def stage(root,site):
                   median_phase_seconds=median(t['phase_seconds'] for t in values) if all(t['phase_seconds'] is not None for t in values) else None)
     if all(result['scores'][a]['score']==100 for a in 'ABC') and result['scores']['D']['invalid']==0:
         d=result['scores']['D']
-        result['assessment']['finding']=f'Original Astra, plain revision and self-review each produced 50/50 valid plans (100%). Fable review delivered {d["evaluated"]}/50 plans: {d["valid"]} valid, {d["invalid"]} invalid and {d["missing"]} missing. Cross-model review gained zero points on the jointly evaluated tasks. With the missing outcomes unresolved, its full-cohort score can only be {d["score_bounds"][0]}–{d["score_bounds"][1]}.'
-        result['assessment']['decision']+=' For this benchmark, prefer the original Astra workflow: it already reached every goal. Additional review produced no measured validity gain and introduced delivery failures and extra compute.'
+        result['assessment']['finding']=f'Original {author}, plain revision and self-review each produced 50/50 valid plans (100%). {reviewer} review delivered {d["evaluated"]}/50 plans: {d["valid"]} valid, {d["invalid"]} invalid and {d["missing"]} missing. Cross-model review gained zero points on jointly evaluated tasks. Its fixed-cohort score is bounded by {d["score_bounds"][0]}–{d["score_bounds"][1]}.'
+        result['assessment']['decision']+=f' For this benchmark, prefer the original {author} workflow: it already reached every goal. Additional review produced no measured validity gain and required extra compute.'
         result['assessment']['next_action']='Do not repeat the same matrix on this saturated subset. Keep the simpler workflow for tasks at this level. Any further study should use a separately specified benchmark with more headroom, preserve a fresh scored cohort, and measure application outcomes before claiming productivity benefits.'
     if result['contrasts']['D-C']['n'] and result['contrasts']['D-C']['repairs']==result['contrasts']['D-C']['regressions']==0:
         result['assessment']['limitation']+=' The [0,0] empirical bootstrap interval resamples only ties; it does not prove population-level equivalence. VAL validity also does not assess shortest-plan optimality.'
-    result['publication_note']='Initial readiness snapshot is archived. This current export reflects the bounded continued stage; missing outcomes remain distinct from incorrect plans.'
-    (public/'planbench-results.json').write_text(json.dumps(result,indent=2,ensure_ascii=False,allow_nan=False)+'\n',encoding='utf-8',newline='\n')
+    if assessment_path:
+        reviewed=json.loads(Path(assessment_path).read_text(encoding='utf-8'))
+        assert reviewed['source_report_sha256']==result['provenance']['source_report_sha256'],'Assessment belongs to different evidence'
+        assert set(reviewed['assessment'])==set(gate.FIELDS)
+        result['assessment'].update(reviewed['assessment'])
+        result['provenance']['reviewed_assessment_sha256']=hashlib.sha256(Path(assessment_path).read_bytes()).hexdigest()
+    result['publication_note']='This is a separate bounded model-pair study. Earlier published studies remain available; missing outcomes remain distinct from incorrect plans.'
+    (public/data_name).write_text(json.dumps(result,indent=2,ensure_ascii=False,allow_nan=False)+'\n',encoding='utf-8',newline='\n')
     registry=json.loads((public/'test-evaluations.json').read_text(encoding='utf-8'))
-    e=next(x for x in registry['studies'] if x['id']=='planbench')
-    e.update(title='PlanBench Hard: scored workflows and practical impact',status=result['completion'],coverage=f'{sum(s["evaluated"] for s in result["scores"].values())}/200 scored plans; smoke tasks excluded',data_sha256=gate.digest(public/'planbench-results.json'))
+    e=next((x for x in registry['studies'] if x['id']==study_id),None)
+    if e is None:
+        e=dict(id=study_id,data=data_name,page=page_name);registry['studies'].insert(0,e)
+    e.update(title=f'PlanBench Hard: {author}/{reviewer} workflows and practical impact',status=result['completion'],coverage=f'{sum(s["evaluated"] for s in result["scores"].values())}/200 scored plans; smoke tasks excluded',data_sha256=gate.digest(public/data_name))
     for key in gate.FIELDS:e[key]=result['assessment'][key]
     if any(s['evaluated'] for s in result['scores'].values()):
         planning=next(x for x in registry['studies'] if x['id']=='planning')
@@ -68,4 +87,4 @@ def stage(root,site):
     print('Staged outcome and assessment; review then render and run the publication gate.')
 
 if __name__=='__main__':
-    p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--site',type=Path,required=True);a=p.parse_args();stage(a.root,a.site)
+    p=argparse.ArgumentParser();p.add_argument('--root',type=Path,required=True);p.add_argument('--site',type=Path,required=True);p.add_argument('--data',default='planbench-results.json');p.add_argument('--page',default='planbench.html');p.add_argument('--study-id',default='planbench');p.add_argument('--assessment',type=Path);a=p.parse_args();stage(a.root,a.site,a.data,a.page,a.study_id,a.assessment)

benchmarks/planbench/reprofile_readiness.py

@@ -0,0 +1,52 @@
+"""Amend an unstarted scored stage after readiness-only runtime failures."""
+import argparse,json,shutil,sqlite3,time
+from pathlib import Path
+from support import sha,write_once,now
+import run
+from campaign import Campaign
+
+def prepare(source,root):
+    source=Path(source);root=Path(root);load=lambda p:json.loads(p.read_text(encoding='utf-8-sig'))
+    old=load(source/'manifest.json');status=load(source/'status.json')
+    assert status['controller']=='finished' and status['calls']==6 and status['candidate_plans']==0
+    assert not root.exists() and time.time()<old['dispatch_cutoff']
+    original_sha=sha(source/'campaign.sqlite');root.mkdir(parents=True)
+    shutil.copyfile(source/'campaign.sqlite',root/'campaign.sqlite');shutil.copyfile(source/'readiness.json',root/'readiness.json')
+    shutil.copytree(source/'attempts',root/'attempts')
+    for name,digest in old['benchmark_hashes'].items():
+        assert sha(source/'upstream'/name)==digest
+        dest=root/'upstream'/name;dest.parent.mkdir(parents=True,exist_ok=True);shutil.copyfile(source/'upstream'/name,dest)
+    (root/'runtime').mkdir()
+    for file in Path(__file__).resolve().parent.glob('*.py'):shutil.copyfile(file,root/'runtime'/file.name)
+    (root/'prior-readiness').mkdir()
+    for name in ('manifest.json','status.json','generation-freeze.json','controller.exit.json'):
+        shutil.copyfile(source/name,root/'prior-readiness'/name)
+    db=sqlite3.connect(root/'campaign.sqlite');db.row_factory=sqlite3.Row
+    calls=[dict(x) for x in db.execute('SELECT * FROM calls ORDER BY id')]
+    jobs=[dict(x) for x in db.execute('SELECT * FROM jobs ORDER BY id')]
+    assert len(calls)==6
+    with db:
+        db.execute('UPDATE stages SET cap=?,family_caps=? WHERE id=?',(6,json.dumps({'claude':5,'codex':1,'glm':0,'kimi':0}),old['stage']))
+    assert calls==[dict(x) for x in db.execute('SELECT * FROM calls ORDER BY id')]
+    assert jobs==[dict(x) for x in db.execute('SELECT * FROM jobs ORDER BY id')]
+    db.close()
+    amendment=dict(at=now(),reason='High-effort Sonnet smoke draft timed out and critique exhausted the small combined output allowance. No scored task began and no scores were used to select settings.',
+      change='Both models use medium effort; Claude combined reasoning/response allowance8192, visible prompt targets unchanged4096/1024. Fresh smoke outputs are necessary because effort changed. Prior high-effort smoke is excluded and charged.',
+      authorization='User requested the Sonnet/Terra test within the established bounded workflow; this is a readiness-only implementation adjustment, not a score-based change or a new budget.',
+      prior_calls=6,original_db_sha256=original_sha,original_manifest_sha256=sha(source/'manifest.json'),source_run=source.name,
+      old_allocation=6,new_allocation=330,new_family_allocation={'claude':275,'codex':55,'glm':0,'kimi':0},
+      cap=336,family_caps=old['family_caps'],deadline=old['deadline_epoch'],comparability='Medium effort and combined response allowance differ from the earlier Astra/Fable high-effort experiment. Within the new trial, settings are fixed across all matching roles.')
+    write_once(root/'PROFILE-AMENDMENT.json',amendment)
+    m=dict(old);m.update(stage=old['stage']+'-medium',effort='medium',combined_output_limit=8192,retry_limits={'claude':15,'codex':3},stage_cap=330,
+      profile_amendment_sha256=sha(root/'PROFILE-AMENDMENT.json'),source_hashes={p.name:sha(p) for p in (root/'runtime').glob('*.py')},
+      output_caps={'sonnet':'8192-token combined reasoning/response allowance; visible targets4096 plans and1024 critiques.','codex':'4096/1024 visible instruction targets; Codex CLI cap not enforced.'})
+    write_once(root/'manifest.json',m)
+    run.configure(m);c=Campaign(root,m['grant'])
+    c.allocate(m['stage'],330,amendment['new_family_allocation'],sha(root/'manifest.json'),run.definitions(m['smoke_tasks']+m['execution_order'],m['smoke_tasks']),m['dispatch_cutoff'])
+    assert c.count()==6 and c.limits()[0]==336;c.close()
+    assert sha(source/'campaign.sqlite')==original_sha
+    (root/'STATUS.md').write_text('Medium-effort readiness amendment prepared. Six high-effort calls retained and charged; no study scores observed.336 total cap, unchanged deadline. Fresh smoke tests use medium and8192 combined response tokens. Launch once.\n',encoding='utf-8')
+    print(json.dumps(amendment,indent=2))
+
+if __name__=='__main__':
+    p=argparse.ArgumentParser();p.add_argument('--source',type=Path,required=True);p.add_argument('--root',type=Path,required=True);a=p.parse_args();prepare(a.source,a.root)