notes: add an index, a lifecycle policy, and a retrieval benchmark

[andrewboudreau]

TL;DR for a reader in a hurry

notes/ had grown to 111 markdown files and 4.2MB with no index, no policy, and no way
in. 90% of it was unreachable in one hop from any front door, so an agent answering a
question read a megabyte to find a paragraph.

This adds the index and the rules, and cuts the cost of answering a question by 67%.
It does not make notes/ smaller — bytes go up ~45KB, because the index, registry,
policy and rules digest are new files. That trade is the honest summary of this PR: much
faster to use, not smaller. If the goal was fewer bytes, this does not achieve it, and
the last section explains why that turns out to be hard rather than merely undone.

Measured

	main	this branch
notes/ files	158	155
markdown files	113	108
notes/ bytes	4,713,896	4,761,180
cost to answer 10 canonical questions	1,100,325 B	397,053 B (-64%)
worst-case hops to an answer	5	1
questions resolving at all	n/a	10/10

Diagnosis that motivated it, all measured: notes/ grew append-only (+92,348/-6,534 lines
lifetime, a 14:1 ratio); 76 markdown files were added in August alone; exactly 7 of 111
files were reachable from a front door; AGENTS.md cited exactly one. There is almost no
textual duplication (0 file pairs share 3+ paragraphs) -- the redundancy is conceptual,
and the dominant decay mode is "the method stays true, the numbers go stale".

What this adds

• notes/README.md -- generated index, the one notes/ never had.

• notes/data/notes-registry.tsv -- tier/bucket/status/summary per file, the index's source.

• notes/CONVENTIONS.md -- lifecycle policy. §3 is the load-bearing rule: a retirement
  deletes the file and rewrites every citation in the same commit, or does neither.

• notes/codegen-rules.md -- ~20KB digest of the codegen rules meant to be loaded whole.

• tools/notes_index.py -- --write / --check, following cpp_tu_state.py's idiom.

• tools/notes_bench.py -- 10 canonical questions with answer regexes, so "did a fact
  survive this edit" is a command rather than an opinion.

• Root CLAUDE.md -- there wasn't one.

• tools/check_dead_references.py extended to scan config/**/*.json, *.jsonl and
  tools/**/*.js. ~75% of citations of notes/ paths live outside .md and were
  previously ungated. 24 genuine dead references found and fixed; 0 banked into
  the baseline, which shrinks by 12 lines.

  This branch originally also scanned src//src_tu//include/. main landed a
  better implementation of that while this PR was open (7d83b57a2), with a real
  lexer and -- the part that matters -- a separate grouped ratchet so the prose
  --update cannot bank hundreds of source-comment entries. This branch now defers to
  it entirely and drops its own NOTES_ONLY_HEADS restriction, which had kept the gate
  quiet at the cost of leaving the tree's ~950 dead src/ refs in code comments
  permanently ungated. See ece7c4303 -- two tests were deleted with it, because they
  asserted the restriction that is now gone.

Two harness defects this surfaced

Both were found by composing the branch against 23 commits of main rather than testing it
alone, and both were gates crying wolf on work that changed no fact -- the failure mode
that gets a gate switched off:

• notes_bench failed on prose re-wrapping. main moved the D0/D1/D2 emission-order
  fact in writer.md from L296 to L523, where it still reads verbatim; the reflow just
  split the literal phrase across a newline. Now matches whitespace-flattened text.
  Negative-controlled: always -> sometimes still fails the row.
• notes_index --check demanded a hand-written registry row per file, so every
  promotion PR dropping a class-facts JSON turned it red -- main added five in 23 commits.
  Those self-register now from their own class/overlay/rtti_name fields. Curation is
  still enforced elsewhere: a stray .md still fails --check.

Why the file count barely moved

It is at the floor the policy allows. After the first pass zero files have zero inbound
citers, so the retire rule is exhausted; further reduction needs MERGE of cited files.
The four strongest candidates were examined and all four declined, each verified
independently:

• collision-query-classes.md <-> handoff-sphereclsn-detectclsn.md: collision-system.md
  pins 9 exact line-number citations into them. Line numbers are invisible to the
  dead-reference gate, so any content move breaks them silently.
• ghidra-dsd-landscape.md <-> overlay-residency.md: the latter is bench-anchored and
  §9-cited by number.
• arm9-endgame.md: 28 src/ files cite it as the asm-primitive policy anchor.

Going further means accepting silent breakage. That is a decision for a reviewer, not one
to make quietly inside a cleanup PR.

Deliberately untouched

writer.md, builder.md, matching-style.md, cpp-tu-current-state.md are byte-identical
to main -- verified after rebase. matching-style.md got a surgical dead-reference fix on
main that a rewrite would have reverted; writer.md is dense operational rules and grew 51%
on main during this work; builder.md was being edited by another session; cpp-tu-current-state.md
is tool-owned output.

mwccarm-codegen.md is the one exception: 8 insertions, 0 deletions, at line 4768 of 4778.
153 files cite it by section and two by line number, so the pointer to codegen-rules.md
goes at the end where it cannot shift anything. Both cited anchors verified intact.

Notes for review

• The doctrine tier reads 370,209 bytes against a 153,600 target. That gap widened
  during this work and none of it is from this branch: writer.md alone went 43,936 ->
  75,202 bytes on main during this PR's life. Reported, not gated -- no check fails on it.
• tools/actor_names.py truncates symbols/actor_renames.tsv (386KB) to a header row on
  re-run. Found, flagged, not fixed -- out of scope for a notes PR.
• Not done: wiring cpp_tu_state.py --write-note into update-chaos-data.yml's refresh
  step. The note it maintains is stale on main.

Verification

Green on the rebased branch against origin/main:

check_dead_references.py   EXIT 0   no new dead refs, no broken markdown links
notes_index.py --check     EXIT 0   registry, README and 155 notes/ files agree
pytest test_check_dead...  EXIT 0   48 passed
notes_bench.py             EXIT 0   10/10 resolve, 397,053 path_bytes

🤖 Generated with Claude Code

https://claude.ai/code/session_013qeuLvkrjFxZriAZ93kFnF

[tangos-validator]

❌ PR validation — Failed

test merge conflicts with exact base 5555bb3

Full merge validation

Check	Result
Committed test merge	no — conflicts

Conflicts:

AGENTS.md
config/dead-reference-baseline.json
notes/enemy-leaf-provenance.md
notes/minigame-provenance.md
notes/mwccarm-codegen.md
notes/overlay-ambiguous-references.md
notes/sinit-ownership-pilot-ov002.md
notes/sinit-wave-a-ownership.md

_{The private worker commits a test merge, builds the stock ROM profile, compares every executable module, measures matched and source-built code, checks contributor lineage, and verifies affected relocations. The mod profile is opt-in and is not part of this merge gate.}

[andrewboudreau]

Bounded tooling/integration review at draft head ece7c4303cd9502dae7572d7a5ef01699cdbdd5b found a reproducible generated-index failure:

P2: the index does not survive a fresh Windows checkout. With core.autocrlf=true, python tools/notes_index.py --check exits 1 on the untouched candidate. The committed budget is 370,228 bytes; the fresh render expects 370,425. notes/README.md gains exactly 197 bytes from its 197 CRLF line endings. doctrine_bytes() counts raw filesystem bytes, while --write emits the self-counted README with LF, so regenerating locally does not establish a stable check after Git checks it out again. Make the indexed size calculation independent of checkout line endings, and verify a fresh checkout after regeneration.

The strict retrieval benchmark passes 10/10, with one-hop answers and 398,038 total path bytes in this checkout. These are benchmark proxy measurements, not measured agent time or a general 67% reduction in question-answering cost; qualify the PR wording and reconcile its differing percentages.

Composition against main c1c8846df622f3a49b6b107e0845e9929473d059 has four conflicts: AGENTS.md, enemy-leaf-provenance.md (modify/delete), minigame-provenance.md and mwccarm-codegen.md. Preserve the newly landed protocol and evidence when resolving them, then rerun the index/reference checks on that actual composition. In particular, the conventions' current-status command should use the active v2 protocol rather than classqueue.py list.

The draft remains unmerged. This is not a complete audit of every condensed research paragraph or a source-acceptance pass.

[andrewboudreau]

Recording what actually blocks this PR, since the branch is otherwise in good shape.

The conflict is resolved. notes/audit-0905-restack-0908 @ 0bb529421 merges cleanly into current main — git merge-tree --write-tree origin/main returns no conflict lines.

What blocks it is a false positive introduced by this branch's own checker change.

python tools/premerge_check.py 0bb529421d2bd31435accaf395e55bd6fae0fe02 --base origin/main

  gate                base     merge    verdict
  dead-references     pass     fail     REGRESSION

  FAIL: 1 prose/comment reference(s) name a path that does not exist:

    config/port_linkage.json
        names `port/link100`, which is not in the tree

config/port_linkage.json:34 reads "branch": "port/link100". That is a git branch, not a path — it exists at refs/heads/port/link100, currently 597e584fcdeee54f1d77c18e344c4e4e2799b3c4, which is the exact sha the same JSON object cites in its basis field. It is correct as written and there is nothing in the tree for it to name.

The regression appears because this branch widens the scan to config JSON, and port is in TOPDIRS, so a slash-separated branch name is shaped like a repo-rooted path. main's current checker does not scan that file, which is why the base side is green.

Both baselines are the wrong remedy here — banking it would put a waiver in the file whose own docstring says a baseline nobody needs "is a place for real breakage to hide." The fix belongs in the scanner: a value under a "branch" key (or a string that resolves as a ref) is not a path reference.

Two other signals from the same run, neither a failure:

• 2 baselined prose references now resolve — the prose baseline can shrink with --update.
• 15 baselined C/C++ comment references now resolve — refresh with --update-code from a clean, pinned main.

Adjacent but distinct: #2534 covers a separate defect in the same file, where PATH_RE truncates a path at a + and reports the non-existent prefix as dead. Different root cause; both would be natural to fix together.

I am not able to edit tools/check_dead_references.py in this session, so I have left the branch as it stands rather than work around the finding.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G1Gdj5fTjMsW2VjRLpXxYA

[andrewboudreau]

The dead-reference blocker is real, but it is a composition failure, not a head failure

Run on this PR's head alone, tools/check_dead_references.py is green:

check_dead_references: 10507 scan target(s), 9930 C/C++ file(s), 11937 repo-rooted
path reference(s), 1717 that do not resolve; 980 markdown link(s), 0 that do not
resolve relative to their own file
  no new dead references
  no broken markdown links
EXIT 0

pytest tools/test_check_dead_references.py is 48 passed, and tools/notes_bench.py
is 10/10. So the PR's own verification block is accurate as written.

It goes red the moment it meets today's main. This PR extends the prose scan
surface to config/**/*.json. On the merge base, config/port_linkage.json held
"branch": "port-mount-noseat-cluster", which the path regex does not match. On
main today that same field reads "branch": "port/link100", which the regex does
match, and port/link100 is a git branch, not a tracked path.

Isolated — this PR's head with only config/port_linkage.json replaced by main's
version, nothing else changed:

FAIL: 1 prose/comment reference(s) name a path that does not exist:

  config/port_linkage.json
      names `port/link100`, which is not in the tree

EXIT 1

That is the whole delta. Every other entry that appears when this PR's scanner is
run against main's content (notes/pret-idioms.md in tangos.json,
tools/refine_run.js, tools/sched_run.js, tools/archive/crack_pr104.js and
notes/levers.jsonl; notes/stale-tu-map-overcut-ov006.md and
notes/minigame-family-decisions.md in three config/tu_manifest.d manifests;
src/actors/daPropeller_Heyho_c.cpp in config/profile-campaign-baseline.json)
is already fixed by this branch — all of those files are in its diff.

Minimal fix

config/port_linkage.json is rewritten by the progress bot on every restamp — the
last twenty-odd commits touching it are all "Restamp LINKED off port/link100 …" —
so the branch name in it will keep changing. Banking port/link100 in the prose
baseline therefore re-breaks on the next rename, and leaves a waiver for a path
nobody wrote, which is the failure mode tools/check_dead_references.py's own
docstring warns about.

The durable minimal fix is to keep config/port_linkage.json out of the prose
scan surface the same way this PR already excludes the other bot-written ledgers:
it is generated state whose branch field is a ref name by construction, not
prose anyone can fix. One line alongside the existing ROOT_JSON_LEDGERS /
JSONL_LEDGERS sets does it. Excluding the branch key specifically would work
too and is narrower, but costs a special case in the scanner.

I have not edited the tool.

Note: this is not issue #2534

#2534 is the +-truncation defect in PATH_RE (paths like
src_tu/actors/Koopa+KoopaSmall.cpp matching as src_tu/actors/Koopa). I could
not get that one to fire from this branch, on the head or in the composition
above. Landing this PR would still close it if the regex fix ships with it, but
the thing actually blocking the gate today is the port/link100 value, which is a
different false positive with a different fix.

Two blockers ahead of that one

• This PR is still marked draft.
• It conflicts with main in 8 files, four of them modify/delete where main
  edited a note this branch retires: AGENTS.md,
  config/dead-reference-baseline.json, notes/minigame-provenance.md,
  notes/mwccarm-codegen.md, and the deletions of
  notes/enemy-leaf-provenance.md, notes/overlay-ambiguous-references.md,
  notes/sinit-ownership-pilot-ov002.md, notes/sinit-wave-a-ownership.md.
  The four modify/delete pairs are decisions, not merges — main added content to
  files this branch retires, and the retire rule in the new notes/CONVENTIONS.md
  §3 requires the citations to move in the same commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G1Gdj5fTjMsW2VjRLpXxYA