Publish PlanBench outcome and require test assessments

.github/workflows/ci.yml

@@ -35,6 +35,8 @@ jobs:
         run: python benchmarks/site/tests/test_observatory.py
       - name: Check score inclusion and repository denominators
         run: node --test benchmarks/site/tests/test-observatory-data.cjs
+      - name: Check publication assessments and stale-evidence rejection
+        run: python benchmarks/site/tests/test_publication.py
 
   test:
     strategy:

.github/workflows/pages.yml

@@ -33,6 +33,9 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Require current evidence assessments before publication
+        run: python3 benchmarks/site/validate-publication.py
+
       - name: Stage the published page as the site root
         shell: bash
         run: |

benchmarks/planbench/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+.sessions/

benchmarks/planbench/README.md

@@ -0,0 +1,48 @@
+# Bounded PlanBench measurement
+
+The September 13, 2026 attempt used the official PlanBench Blocksworld Hard
+corpus, pinned at `fc638a1aff7df3fe7a1a1d289fa2c04cc24dc284`, and its bundled
+VAL executable and PDDL extractor. The scored sample is 50 fixed tasks from
+110. Four arms compare Astra original, plain revision, self-review, and Fable
+review followed by Astra revision.
+
+The attempt stopped at readiness after one Fable safeguard refusal. No scored
+task ran. Seven available excluded smoke plans validated, but that is not a
+benchmark accuracy result. Public outcome: `../site/public/planbench-results.json`.
+The run's local immutable manifest, attempts, SQLite accounting, generation
+freeze, readiness fixtures and validator logs are under
+`runs/planbench-hard-20260913` in the parent working repository.
+
+`run.py init --root RUN --started EPOCH` creates a new frozen run from a
+pre-fetched official repository at RUN/upstream. It does not authorize a new
+budget by itself; supply an explicitly authorized plan before using this
+task-specific controller. `run.py check --root RUN` validates the three
+offline fixtures. `run.py run --root RUN --live` runs one locked controller.
+`score.py --root RUN` settles only frozen terminal generation. `run.py status`
+is read-only. Never reinitialize or restart a settled attempt unchanged.
+
+This implementation is specific to the recorded stage. Source hashes are
+frozen inside the run before dispatch; copying edited source over it is not
+a valid resume. The preserved launch3 planning transport supplies the exact
+model/catalog isolation repair, process-tree ownership and subscriptions.
+Changes for this stage replace the system prompt and set Fable's output
+allowance to 1024 tokens. Astra's CLI output targets are instructions, not an
+enforced token limit. Both calls time out at 120 seconds. Model fallback is
+not allowed. Transport is isolated from solutions and scoring files.
+
+The preserved campaign ledger reserves each call transactionally, retains
+failed calls and limits jobs to two attempts. The controller additionally
+enforces family retry reserves, family concurrency and the dispatch cutoff.
+The old custom-study ledgers are not imported, reset or modified.
+
+Validation performed once: official valid/invalid/malformed fixtures; an
+offline lifecycle with a charged transient retry and a duplicate-free resume;
+the two excluded live smoke tasks. No scored benchmark was run after the
+provider refusal. Publication includes an explicit evidence assessment.
+
+Sources: https://github.com/karthikv792/LLMs-Planning and
+https://github.com/KCL-Planning/VAL . Upstream PDDL extraction is loaded as
+the exact AST function from the pinned source, avoiding unrelated LLM imports.
+The VAL wrapper recognizes its normal invalid-plan exit code 1 as a failed
+plan when its diagnostic identifies that outcome; validator infrastructure
+failures stay missing.

benchmarks/planbench/campaign.py

@@ -0,0 +1,202 @@
+"""Single cumulative authorization and attempt ledger for planning stages."""
+from contextlib import ExitStack
+import hashlib
+import json
+from pathlib import Path
+import sqlite3
+import time
+from support import writer_lock, write_once, now
+
+CAP = 200
+FAMILY_CAPS = {'claude': 80, 'codex': 80, 'glm': 20, 'kimi': 20}
+FAMILY = {'sonnet': 'claude', 'fable': 'claude', 'codex': 'codex',
+          'astra': 'codex', 'glm': 'glm', 'kimi': 'kimi'}
+
+
+class BudgetError(RuntimeError):
+    pass
+
+
+class Campaign:
+    def __init__(self, root, grant='pilot'):
+        self.root = Path(root).resolve()
+        self.grant_id = grant
+        self.db = sqlite3.connect(self.root / 'campaign.sqlite', timeout=30)
+        self.db.row_factory = sqlite3.Row
+        self.db.executescript('''
+            CREATE TABLE IF NOT EXISTS history (name TEXT PRIMARY KEY, count INTEGER, digest TEXT);
+            CREATE TABLE IF NOT EXISTS stages (id TEXT PRIMARY KEY, cap INTEGER, family_caps TEXT,
+                manifest_sha TEXT, deadline REAL);
+            CREATE TABLE IF NOT EXISTS jobs (stage TEXT, id TEXT, definition TEXT,
+                state TEXT DEFAULT 'pending', result TEXT, error TEXT, not_before REAL DEFAULT 0,
+                PRIMARY KEY(stage,id));
+            CREATE TABLE IF NOT EXISTS calls (id INTEGER PRIMARY KEY, stage TEXT, job TEXT,
+                seat TEXT, family TEXT, attempt_index INTEGER, started TEXT, finished TEXT,
+                state TEXT, prompt_sha TEXT, UNIQUE(stage,job,attempt_index));
+            CREATE TABLE IF NOT EXISTS grants (id TEXT PRIMARY KEY, cap INTEGER,
+                family_caps TEXT, authorization TEXT);
+        ''')
+        for table in ('calls', 'stages'):
+            if 'grant_id' not in {r[1] for r in self.db.execute('PRAGMA table_info(' + table + ')')}:
+                self.db.execute("ALTER TABLE " + table + " ADD COLUMN grant_id TEXT NOT NULL DEFAULT 'pilot'")
+        self.db.execute('INSERT OR IGNORE INTO grants VALUES(?,?,?,?)',
+                        ('pilot', CAP, json.dumps(FAMILY_CAPS), 'Original 200-call planning pilot authorization'))
+        self.db.commit()
+
+    def close(self):
+        self.db.close()
+
+    def authorize(self, cap, families, authorization):
+        """Record a NEW user-approved grant; never enlarge an existing grant."""
+        if cap <= 0 or any(v < 0 for v in families.values()) or sum(families.values()) != cap or set(families) != set(FAMILY_CAPS) or not authorization.strip():
+            raise BudgetError('Invalid authorization')
+        with self.db:
+            old = self.db.execute('SELECT * FROM grants WHERE id=?', (self.grant_id,)).fetchone()
+            if old:
+                if old['cap'] != cap or json.loads(old['family_caps']) != families:
+                    raise BudgetError('Existing authorization cannot be raised or replaced')
+                return
+            self.db.execute('INSERT INTO grants VALUES(?,?,?,?)',
+                            (self.grant_id, cap, json.dumps(families), authorization))
+
+    def limits(self):
+        grant = self.db.execute('SELECT * FROM grants WHERE id=?', (self.grant_id,)).fetchone()
+        if not grant:
+            raise BudgetError('No authorization recorded for this grant')
+        return grant['cap'], json.loads(grant['family_caps'])
+
+    def lifetime_count(self):
+        return self.db.execute('SELECT count(*) FROM calls').fetchone()[0]
+
+    def import_history(self):
+        """Close historical controllers before importing all spent reservations."""
+        if self.grant_id != 'pilot':
+            raise BudgetError('Historical pilot spend belongs to its original grant')
+        with ExitStack() as stack:
+            for name in ('pilot-001', 'pilot-002'):
+                stack.enter_context(writer_lock(self.root / name))
+            snapshots = []
+            for name in ('pilot-001', 'pilot-002'):
+                path = self.root / name / 'ledger.sqlite'
+                old = sqlite3.connect(path.as_uri() + '?mode=ro', uri=True)
+                old.row_factory = sqlite3.Row
+                if old.execute("SELECT count(*) FROM jobs WHERE state IN ('pending','running')").fetchone()[0]:
+                    old.close()
+                    raise RuntimeError('Historical controller must be settled: ' + name)
+                rows = [dict(r) for r in old.execute('SELECT * FROM attempts ORDER BY id')]
+                old.close()
+                digest = hashlib.sha256(path.read_bytes()).hexdigest()
+                marker = self.root / name / 'CLOSED.json'
+                if not marker.exists():
+                    write_once(marker, dict(at=now(), reason='Allocation transferred to central planning campaign', spent=len(rows)))
+                if json.loads(marker.read_text(encoding='utf-8'))['spent'] != len(rows):
+                    raise RuntimeError('Historical budget changed after closure')
+                snapshots.append((name, rows, digest))
+            self.db.execute('BEGIN IMMEDIATE')
+            try:
+                for name, rows, digest in snapshots:
+                    previous = self.db.execute('SELECT * FROM history WHERE name=?', (name,)).fetchone()
+                    if previous:
+                        if previous['count'] != len(rows) or previous['digest'] != digest:
+                            raise RuntimeError('Historical ledger changed; refusing to allocate more spend')
+                        continue
+                    for row in rows:
+                        self.db.execute('INSERT INTO calls(stage,job,seat,family,attempt_index,started,finished,state,prompt_sha) VALUES(?,?,?,?,?,?,?,?,?)',
+                                        ('history:' + name, str(row['id']), row['seat'], row['family'], 1,
+                                         row['started'], row.get('finished'), row['state'], row.get('prompt_sha')))
+                    self.db.execute('INSERT INTO history VALUES(?,?,?)', (name, len(rows), digest))
+                if self.db.execute('SELECT sum(count) FROM history').fetchone()[0] != 143:
+                    raise BudgetError('Expected 143 historical reservations; reconcile before proceeding')
+                self.db.commit()
+            except Exception:
+                self.db.rollback()
+                raise
+
+    def allocate(self, stage, cap, families, manifest_sha, jobs, deadline):
+        if sum(families.values()) != cap or cap <= 0 or any(v < 0 for v in families.values()):
+            raise BudgetError('Invalid allocation')
+        self.db.execute('BEGIN IMMEDIATE')
+        try:
+            old = self.db.execute('SELECT * FROM stages WHERE id=?', (stage,)).fetchone()
+            if old:
+                if old['grant_id'] != self.grant_id:
+                    raise BudgetError('Stage belongs to another authorization')
+                if (old['cap'], json.loads(old['family_caps']), old['manifest_sha']) != (cap, families, manifest_sha):
+                    raise BudgetError('Cannot replace an existing frozen allocation')
+                self.db.commit()
+                return
+            historical = dict(self.db.execute("SELECT family,count(*) FROM calls WHERE grant_id=? AND stage LIKE 'history:%' GROUP BY family", (self.grant_id,)))
+            if self.grant_id == 'pilot' and sum(historical.values()) != 143:
+                raise BudgetError('History must be imported before allocating')
+            grant_cap, grant_families = self.limits()
+            allocations = self.db.execute('SELECT cap,family_caps FROM stages WHERE grant_id=?', (self.grant_id,)).fetchall()
+            if sum(historical.values()) + sum(x['cap'] for x in allocations) + cap > grant_cap:
+                raise BudgetError('New allocation exceeds the campaign cap')
+            for family, limit in grant_families.items():
+                total = historical.get(family, 0) + sum(json.loads(x['family_caps']).get(family, 0) for x in allocations) + families.get(family, 0)
+                if total > limit:
+                    raise BudgetError('Allocation exceeds family cap: ' + family)
+            self.db.execute('INSERT INTO stages(id,cap,family_caps,manifest_sha,deadline,grant_id) VALUES(?,?,?,?,?,?)',
+                            (stage, cap, json.dumps(families), manifest_sha, deadline, self.grant_id))
+            for job in jobs:
+                self.db.execute('INSERT INTO jobs(stage,id,definition) VALUES(?,?,?)', (stage, job['id'], json.dumps(job)))
+            self.db.commit()
+        except Exception:
+            self.db.rollback()
+            raise
+
+    def jobs(self, stage):
+        return self.db.execute('SELECT * FROM jobs WHERE stage=? ORDER BY id', (stage,)).fetchall()
+
+    def count(self, stage=None, family=None):
+        terms, args = ['grant_id=?'], [self.grant_id]
+        if stage is not None:
+            terms.append('stage=?'); args.append(stage)
+        if family is not None:
+            terms.append('family=?'); args.append(family)
+        query = 'SELECT count(*) FROM calls' + (' WHERE ' + ' AND '.join(terms) if terms else '')
+        return self.db.execute(query, args).fetchone()[0]
+
+    def attempts(self, stage, job):
+        return self.db.execute('SELECT * FROM calls WHERE stage=? AND job=? ORDER BY attempt_index', (stage, job)).fetchall()
+
+    def reserve(self, stage, ident, prompt_sha):
+        self.db.execute('BEGIN IMMEDIATE')
+        try:
+            allocation = self.db.execute('SELECT * FROM stages WHERE id=?', (stage,)).fetchone()
+            job = self.db.execute('SELECT * FROM jobs WHERE stage=? AND id=?', (stage, ident)).fetchone()
+            if not allocation or allocation['grant_id'] != self.grant_id or not job or job['state'] != 'pending':
+                raise BudgetError('Job is not pending in an allocated stage')
+            if time.time() >= allocation['deadline']:
+                raise BudgetError('Absolute stage deadline reached')
+            definition = json.loads(job['definition'])
+            seat = definition['seat']; family = FAMILY[seat]
+            caps = json.loads(allocation['family_caps'])
+            grant_cap, grant_families = self.limits()
+            if (self.count() >= grant_cap or self.count(family=family) >= grant_families[family]
+                    or self.count(stage) >= allocation['cap'] or self.count(stage, family) >= caps.get(family, 0)):
+                raise BudgetError('Cumulative dispatch limit reached')
+            attempts = self.attempts(stage, ident)
+            if len(attempts) >= 2:
+                raise BudgetError('One retry per job maximum')
+            if attempts and attempts[0]['prompt_sha'] != prompt_sha:
+                raise RuntimeError('Retry would change the frozen prompt')
+            cursor = self.db.execute('INSERT INTO calls(stage,job,seat,family,attempt_index,started,state,prompt_sha,grant_id) VALUES(?,?,?,?,?,?,?,?,?)',
+                                     (stage, ident, seat, family, len(attempts) + 1, now(), 'reserved', prompt_sha, self.grant_id))
+            self.db.execute("UPDATE jobs SET state='running',error=NULL WHERE stage=? AND id=?", (stage, ident))
+            self.db.commit()
+            return cursor.lastrowid
+        except Exception:
+            self.db.rollback()
+            raise
+
+    def finish(self, stage, ident, call, state, result=None, error=None, retry_at=0):
+        with self.db:
+            self.db.execute('UPDATE calls SET state=?,finished=? WHERE id=?',
+                            ('failed' if state == 'pending' else state, now(), call))
+            self.db.execute('UPDATE jobs SET state=?,result=?,error=?,not_before=? WHERE stage=? AND id=?',
+                            (state, json.dumps(result) if result is not None else None, error, retry_at, stage, ident))
+
+    def block(self, stage, ident, reason):
+        with self.db:
+            self.db.execute("UPDATE jobs SET state='blocked',error=? WHERE stage=? AND id=? AND state='pending'", (reason, stage, ident))

benchmarks/planbench/evaluator.py

@@ -0,0 +1,44 @@
+"""Official zero-shot PDDL extraction and bundled VAL, without model scoring."""
+import ast, json, subprocess
+from pathlib import Path
+from support import sha
+
+def upstream_function(upstream,name,relative):
+    tree=ast.parse((Path(upstream)/relative).read_text(encoding='utf-8'))
+    function=next(n for n in tree.body if isinstance(n,ast.FunctionDef) and n.name==name)
+    namespace={}
+    exec(compile(ast.Module(body=[function],type_ignores=[]),relative,'exec'),namespace)
+    return namespace[name]
+
+def linux(path):
+    p=Path(path).resolve().as_posix()
+    return '/mnt/'+p[0].lower()+p[2:]
+
+def evaluate(upstream,problem,text,dest):
+    upstream=Path(upstream);dest=Path(dest);dest.mkdir(parents=True,exist_ok=True)
+    extract=upstream_function(upstream,'save_gpt3_response','llm_planning_analysis/utils/llm_utils.py')
+    plan=extract(text,str(dest/'plan.pddl'))
+    if not plan.strip():
+        return dict(valid=False,category='invalid_serialization',extracted_plan_sha=sha(dest/'plan.pddl'))
+    command=['wsl','-d','Ubuntu','--',linux(upstream/'planner_tools/VAL/validate'),
+      linux(upstream/'llm_planning_analysis/instances/blocksworld_hard/generated_domain.pddl'),linux(problem),linux(dest/'plan.pddl')]
+    try:
+        proc=subprocess.run(command,capture_output=True,text=True,encoding='utf-8',errors='replace',timeout=15)
+    except (OSError,subprocess.TimeoutExpired) as e:
+        return dict(valid=None,category='validator_infrastructure',error=type(e).__name__)
+    output=proc.stdout+'\n'+proc.stderr
+    (dest/'validator.txt').write_text(output,encoding='utf-8')
+    if proc.returncode not in (0,1) or 'Problem in domain' in output or not any(s in output for s in ('Plan valid','Plan failed','Failed plans','Bad plan','Plan invalid')):
+        return dict(valid=None,category='validator_infrastructure',returncode=proc.returncode,output=output)
+    return dict(valid='Plan valid' in proc.stdout,category='valid' if 'Plan valid' in proc.stdout else 'invalid_plan',returncode=proc.returncode,extracted_plan_sha=sha(dest/'plan.pddl'))
+
+def check(root):
+    root=Path(root);fixture=root/'validator-fixture';fixture.mkdir(exist_ok=True)
+    problem=fixture/'problem.pddl'
+    problem.write_text('(define (problem fixture) (:domain blocksworld-4ops) (:objects a b) (:init (handempty) (ontable a) (ontable b) (clear a) (clear b)) (:goal (on a b)))',encoding='utf-8')
+    results={name:evaluate(root/'upstream',problem,text,fixture/name) for name,text in
+      [('valid','(pick-up a)\n(stack a b)'),('invalid','(stack a b)'),('malformed','this is not a plan')]}
+    assert results['valid']['valid'] is True,results
+    assert results['invalid']['valid'] is False,results
+    assert results['malformed']['valid'] is False,results
+    return results