Skip to content

feat: evidence governance — deterministic bench, E0 protocol gate, machine-checked RESULTS.md - #1

Open
AricRedemption wants to merge 14 commits into
mainfrom
feat/evidence-governance
Open

AricRedemption wants to merge 14 commits into
mainfrom
feat/evidence-governance

Conversation

@AricRedemption

Copy link
Copy Markdown
Collaborator

概要

借鉴 M1 仓库(MT-LNN)的证据链治理制度,为本仓建立"单一事实源 + 机器门禁",并顺手修掉两起被门禁暴露的真实事故。

两笔提交

1. fix(bench) — 确定性 + E0 协议门

  • 事故修复bench --fake 曾连续两次运行结果矛盾(recall 一说 8/8 一说 7/8)。根因是词法编码器用内建 hash(),盐随进程随机。已收敛为共享模块 awareliquid/hashing.py::stable_hash,三连跑逐字节一致(留档 benchmarks/results/bench_adapter_fake_20260906_r{1,2,3}.log)。
  • E0 协议门(移植自 M1):benchmarks/experiment_protocol.pypublishable() 要求 ≥3 seeds、双臂非双峰(对量化分数加了绝对间隙下限,避免正常一致性数据被误杀)、配对符号检验 p<0.05;脏臂判负而非崩溃。
  • 回归锁:双 PYTHONHASHSEED 子进程对照 + bench 输出与留档日志逐字节钉扎;autouse RNG 钉扎 fixture。

2. docs(evidence) — 单一事实源 + 机器门禁

  • docs/RESULTS.md:Proven / Suspended / Do-not-claim 三表 + 对账日志;每个数字指回仓库产物(含"91.7% 无本仓产物支撑""answer.csv 非可提交状态"等负面声明)。
  • docs/PREREGISTRATION.md:判定规则先于结果落盘、判负也归档、不做清单。
  • scripts/check_results_refs.py(+9 个测试):校验 RESULTS.md 引用路径存在、run manifest 可解析、每个 .sha256 sidecar 与磁盘字节一致、歧义裸文件名报 FAIL。
  • 修正 66 处 sidecar 失配:生成管线哈希的是 canonical compact JSON 而落盘是序列化文件——逐一验证对象等价后重写为磁盘字节哈希(0 个 JSON 本体被触碰);5 个 bootstrap 脚本的 sidecar 调用点全部改为磁盘字节哈希,杜绝复发。
  • CI 证据门禁:pytest + 引用校验,最小权限 + 30min 超时。

验证证据

  • 197 passed(离线全量,留档 benchmarks/results/pytest_20260907.log);校验器 PASS(17 引用 / 228 manifest 路径 / 120 sidecar)
  • 双盐活体验证:输出互相一致且 == 留档日志
  • 协议门 5 场景事故回放全过;README 数字与日志逐项对账一致
  • 经一轮独立只读评审:1 BLOCKER + 6 SHOULD-FIX + 10 NIT 全部闭环

注意

  • answer.csv 为既有占位文件(token 全 0),不在本 PR 范围内,保持 untracked。
  • CI 首跑需要 Actions 权限;PR 合并前建议看一眼首跑结果。

- Replace salted built-in hash() with shared awareliquid/hashing.py::stable_hash
  in the FakeEncoder (tests/conftest.py) and the bench LexicalEncoder — root
  cause of the 2026-09-06 cross-process bench drift incident (two runs reported
  recall 8/8 vs 7/8); three byte-identical runs archived under benchmarks/results/
- Port the E0 protocol gate from the M1 repo as benchmarks/experiment_protocol.py:
  publishable() requires >=3 seeds, non-bimodal arms (absolute gap floor so
  quantized metrics are not misflagged), paired sign test, Fisher exact; dirty
  arms fail closed instead of crashing
- Pin RNG in an autouse pytest fixture; add determinism regression tests that
  run bench under two different PYTHONHASHSEED salts and pin its stdout
  byte-exact to the archived log
…vidence gate

- Add docs/RESULTS.md as the canonical evidence base (Proven / Suspended /
  Do-not-claim tables + reconciliation log; every number cites a repo artifact)
  and docs/PREREGISTRATION.md (judge-before-run discipline + do-not list)
- Add scripts/check_results_refs.py (+ tests): verifies every path referenced
  from RESULTS.md exists, run manifests resolve inside the repo, and every
  .sha256 sidecar matches the disk bytes; ambiguous bare names fail instead of
  silently matching
- Correct 66 JSON sidecars to disk-byte hashes after verifying object
  equivalence (pipeline had hashed canonical compact JSON while persisting
  pretty-printed files); bootstrap scripts now sidecar disk bytes — canonical
  object hashes remain only in the documented *_sha256 fields
- Add CI evidence gate (pytest + reference check) and reconcile the README
  validation table to the 8-question deterministic run
AricRedemption added a commit that referenced this pull request Sep 16, 2026
… halt collapses

- G1 PASS: fixed-arm rerun science metrics bit-identical (only wall_s timer
  differs by 0.1s, instrumentation field, noted).
- G2 FAIL: indist acc fixed 0.494 / adaptive 0.508 (< 0.55, chance; loss stuck
  at ~ln2 after 1200 steps) — pooled-linear readout learns no parity signal
  at this capacity/schedule.
- G3 FAIL: adaptive mean steps 1.010 < 1.3 — halting collapses; downstream of
  G2 (no task signal rewards extra compute yet).
- G4 PASS: fixed 22.6s / adaptive 90.9s per arm -> 2 arms x 5 seeds ~9.5 min,
  multi-seed stays local per compute policy.
- Disposition per preregistered G2 clause: task/capacity underpowered; next
  round recalibrates (steps 6-10k, deeper trunk, length curriculum, or
  iteration-friendly task) and re-screens. Attempt-1 pooling bug kept as
  *_attempt1_poolingbug.log for the record.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant