Fix #112: recover 328 replacement characters in the registries, and guard the data files

[emmanuelgjr]

Closes #112. Item 2 of the three follow-ups.

What was wrong

Fourteen files under data/frameworks/ carried 328 U+FFFD replacement characters, across 176 fields:

"description": "ML SBOM as software asset inventory <U+FFFD> all agent components (tools, MCP servers, models)"

These ship — the registries go out in the npm package, in docs/frameworks-registry.js, and in the OSCAL/STIX exports.

Recovered, not guessed

The registries were extracted from the mapping rows, so the same sentence usually survives intact in Markdown. For each corrupted string the text on either side is looked up in the mapping files and entry JSON, and the character that sits between them is read off the source. A character is accepted only when every match agrees.

Two refinements were needed, both visible in the run:

• the window narrows from 24 → 14 → 8 characters, because a registry description concatenates several rows and a wide window can straddle the join;
• when only one word follows the character, the raw slice is kept rather than trimmed to a word boundary — otherwise the window collapses and a recoverable character looks unrecoverable. That alone accounted for the last 4.

Result: 328 of 328 resolved, every one to U+2014 (—). Nothing was assumed, and no other character was found in any position.

Guard

The encoding guard from T-ACC06 only ever read Markdown — which is exactly why this sat unnoticed through every prior run. It now also reads data/frameworks/*.json, data/entries/*.json and data/*.json, checking for U+FFFD only: the arrow heuristic ("A ? B") is tuned for prose and diagrams and would misfire on JSON.

Negative-tested: a replacement character injected into soc2.json fails validation with the file and line number; removing it passes.

Verification

• node scripts/validate.js: 0 errors, 87 warnings, 327 passed (one new check).
• npm run test:scripts: 85 passed, 0 failed.
• Zero U+FFFD and zero control bytes remain anywhere under data/.
• docs/frameworks-registry.js regenerated; stats current.

🤖 Generated with Claude Code