feat: evidence governance — deterministic bench, E0 protocol gate, machine-checked RESULTS.md - #1
Open
AricRedemption wants to merge 14 commits into
Open
AricRedemption wants to merge 14 commits into
AricRedemption wants to merge 14 commits into
Conversation
- Replace salted built-in hash() with shared awareliquid/hashing.py::stable_hash in the FakeEncoder (tests/conftest.py) and the bench LexicalEncoder — root cause of the 2026-09-06 cross-process bench drift incident (two runs reported recall 8/8 vs 7/8); three byte-identical runs archived under benchmarks/results/ - Port the E0 protocol gate from the M1 repo as benchmarks/experiment_protocol.py: publishable() requires >=3 seeds, non-bimodal arms (absolute gap floor so quantized metrics are not misflagged), paired sign test, Fisher exact; dirty arms fail closed instead of crashing - Pin RNG in an autouse pytest fixture; add determinism regression tests that run bench under two different PYTHONHASHSEED salts and pin its stdout byte-exact to the archived log
…vidence gate - Add docs/RESULTS.md as the canonical evidence base (Proven / Suspended / Do-not-claim tables + reconciliation log; every number cites a repo artifact) and docs/PREREGISTRATION.md (judge-before-run discipline + do-not list) - Add scripts/check_results_refs.py (+ tests): verifies every path referenced from RESULTS.md exists, run manifests resolve inside the repo, and every .sha256 sidecar matches the disk bytes; ambiguous bare names fail instead of silently matching - Correct 66 JSON sidecars to disk-byte hashes after verifying object equivalence (pipeline had hashed canonical compact JSON while persisting pretty-printed files); bootstrap scripts now sidecar disk bytes — canonical object hashes remain only in the documented *_sha256 fields - Add CI evidence gate (pytest + reference check) and reconcile the README validation table to the 8-question deterministic run
AricRedemption
added a commit
that referenced
this pull request
Sep 16, 2026
… halt collapses - G1 PASS: fixed-arm rerun science metrics bit-identical (only wall_s timer differs by 0.1s, instrumentation field, noted). - G2 FAIL: indist acc fixed 0.494 / adaptive 0.508 (< 0.55, chance; loss stuck at ~ln2 after 1200 steps) — pooled-linear readout learns no parity signal at this capacity/schedule. - G3 FAIL: adaptive mean steps 1.010 < 1.3 — halting collapses; downstream of G2 (no task signal rewards extra compute yet). - G4 PASS: fixed 22.6s / adaptive 90.9s per arm -> 2 arms x 5 seeds ~9.5 min, multi-seed stays local per compute policy. - Disposition per preregistered G2 clause: task/capacity underpowered; next round recalibrates (steps 6-10k, deeper trunk, length curriculum, or iteration-friendly task) and re-screens. Attempt-1 pooling bug kept as *_attempt1_poolingbug.log for the record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
借鉴 M1 仓库(MT-LNN)的证据链治理制度,为本仓建立"单一事实源 + 机器门禁",并顺手修掉两起被门禁暴露的真实事故。
两笔提交
1. fix(bench) — 确定性 + E0 协议门
bench --fake曾连续两次运行结果矛盾(recall 一说 8/8 一说 7/8)。根因是词法编码器用内建hash(),盐随进程随机。已收敛为共享模块awareliquid/hashing.py::stable_hash,三连跑逐字节一致(留档benchmarks/results/bench_adapter_fake_20260906_r{1,2,3}.log)。benchmarks/experiment_protocol.py—publishable()要求 ≥3 seeds、双臂非双峰(对量化分数加了绝对间隙下限,避免正常一致性数据被误杀)、配对符号检验 p<0.05;脏臂判负而非崩溃。PYTHONHASHSEED子进程对照 + bench 输出与留档日志逐字节钉扎;autouse RNG 钉扎 fixture。2. docs(evidence) — 单一事实源 + 机器门禁
docs/RESULTS.md:Proven / Suspended / Do-not-claim 三表 + 对账日志;每个数字指回仓库产物(含"91.7% 无本仓产物支撑""answer.csv 非可提交状态"等负面声明)。docs/PREREGISTRATION.md:判定规则先于结果落盘、判负也归档、不做清单。scripts/check_results_refs.py(+9 个测试):校验 RESULTS.md 引用路径存在、run manifest 可解析、每个 .sha256 sidecar 与磁盘字节一致、歧义裸文件名报 FAIL。验证证据
benchmarks/results/pytest_20260907.log);校验器 PASS(17 引用 / 228 manifest 路径 / 120 sidecar)注意
answer.csv为既有占位文件(token 全 0),不在本 PR 范围内,保持 untracked。