Skip to content

Fix #35: parse control identifiers by framework grammar, not by column position - #110

Merged
emmanuelgjr merged 1 commit into
mainfrom
fix/swapped-control-ids
Sep 18, 2026
Merged

emmanuelgjr merged 1 commit into
mainfrom
fix/swapped-control-ids

Conversation

@emmanuelgjr

Copy link
Copy Markdown
Contributor

Closes the data half of #35. 611 prose-shaped control_id values → 0.

The bug

parseControlTable() asked looksLikeId(), which rejects any identifier containing a space — Req 6.2.4, Art. 9, §5.3, SR 3.3 — and then fell through to "column 0 is the id". Where a table wrote CC3.2 — Name | requirement prose, the row came out as:

{ "control_id": "Prompt injection documented as a threat in LLM application risk assessment — vectors, likelihood, impact assessed",
  "control_name": "CC3.2" }

A reader could not follow an id back to its framework, and the control-level join resolved by luck.

The fix

scripts/control-ids.js — a per-framework identifier grammar for the ten affected frameworks. Each row is scanned for its own framework's id shape, wherever it sits; the id becomes control_id, the remainder of that cell becomes the name, and the requirement moves to notes. Where a table has no name column, the name is taken from the framework registry's title for that id instead of repeating the id.

Canonical forms are your ruling on #35: §5.3, Art. 9, Req 6.2.4, NHI-5, bare ISO clauses (6.1), SP 800-218A keeping NIST's -PS.

  • (A) CIScontrol_id is the safeguard (16.1), with the control carried as parent (CIS-16); 138 rows.
  • (B) SP 800-53 rows inside the 800-82 files — kept, and labelled id_source: "NIST SP 800-53 control cited by the SP 800-82 overlay"; 30 rows.

EU AI Act keeps paragraph references (Art. 55(1)(b)), because the paragraph is what says which obligation was mapped.

Framework Before After
SOC 2 168 0
EU AI Act 123 0
OWASP NHI Top 10 105 0
CIS Controls v8.1 58 0
PCI DSS v4.0 40 0
NIST SP 800-218A 40 0
NIST SP 800-82 Rev 3 33 0
CWE/CVE 25 0
OWASP AI Testing Guide 17 0
ISO/IEC 42001:2023 3 0

Two parser bugs found on the way

  • A section can hold more than one table. LLM_NISTSP80082.md follows its SP 800-82 sections with an SP 800-53 table, and that second table's header row was being parsed as a mapping — 10 junk rows like control_id: "Title". Splitting on the |---| separators drops them and gives each table its own headers, which is what lets the SP 800-53 rows keep their titles (SI-10 → Information Input Validation) instead of naming themselves.
  • Two-column tables lost the requirement text once the id stopped being mis-filed there. It now becomes notes, so nothing is discarded: 1,149 of 1,337 repaired rows carry their requirement.

Guard

checkControlIdShapes() in validate.js fails the build if any of the 1,337 rows in these frameworks stops carrying a well-formed identifier. scripts/control-ids.test.mjs pins the grammar against 20 row shapes taken from the files as they are today, including the ones that must yield no name rather than promote a requirement into one.

Scope and verification

  • Mappings 3,781 → 3,771 — exactly the 10 header rows. 969 rows changed id. No row was removed, and no severity, relationship, confidence or rationale changed.
  • node scripts/validate.js: 0 errors, 87 warnings, 326 passed.
  • npm run test:scripts: 85 passed, 0 failed, twice (81 + 4 new).
  • npm run stats:check current; generator reruns clean.

Not in this PR

The registries in data/frameworks/ were seeded from these same broken rows and still hold prose ids (NHI 104 of 117, AI Testing Guide 58 of 58). Migrating them moves headline control counts substantially and needs a note on the README and the webapp, so it follows as its own PR.

🤖 Generated with Claude Code

`parseControlTable()` tried `looksLikeId()`, which rejects every identifier
containing a space — `Req 6.2.4`, `Art. 9`, `§5.3` — then fell through to
"column 0 is the id". On 611 rows that wrote the requirement prose into
`control_id` and the identifier into `control_name`, so an id could not be
followed back to its framework and the control-level join resolved by luck.

scripts/control-ids.js holds a per-framework identifier grammar for the ten
frameworks affected. A row is scanned for its framework's own id shape,
wherever that sits: the id becomes `control_id`, the rest of that cell the
name, and the requirement moves to `notes`. When the table has no name column
the name comes from the framework registry's title for that id rather than
repeating the id. Canonical forms follow the ruling on #35 — §5.3, Art. 9,
Req 6.2.4, NHI-5, bare ISO clauses, SP 800-218A keeping NIST's -PS — with CIS
identified by safeguard and the control carried as `parent` (A), and the SP
800-53 rows inside the 800-82 files kept and labelled `id_source` so they are
not read as 800-82 sections (B).

Two parser bugs surfaced on the way:
  - a section can hold more than one table, and the second table's header row
    was parsed as data. Splitting on the |---| separators drops those 10 junk
    rows and gives each table its own headers, which is what lets the SP
    800-53 rows keep their titles.
  - two-column tables lost the requirement text entirely once the id stopped
    being mis-filed; it now becomes `notes`.

Prose-shaped ids across the corpus: 611 -> 0. Mappings 3,781 -> 3,771, the
difference being those 10 header rows. 969 rows changed id; no row was
removed, and no severity, relationship or confidence changed.

checkControlIdShapes() in validate.js fails the build if any of the 1,337
rows in those frameworks stops carrying a well-formed identifier.

The registries were seeded from these broken rows and still hold prose ids;
migrating them changes headline control counts and the webapp, so it follows
separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@emmanuelgjr
emmanuelgjr merged commit 0df8c05 into main Sep 18, 2026
6 checks passed
@emmanuelgjr
emmanuelgjr deleted the fix/swapped-control-ids branch September 18, 2026 13:45
emmanuelgjr added a commit that referenced this pull request Sep 18, 2026
…bout timeline (#113)

The landing paragraph still claimed 3,781 control mappings; #110 removed ten
rows that were table header lines the parser had mistaken for mappings, so
the figure is 3,771.

The About timeline gains a row for the repair itself — identifiers now parsed
by each framework's grammar rather than by column position, the registries
migrated off the ids that produced (1,541 → 991 controls), and the MITRE
ATLAS re-transcription. The earlier rows are left as they are: they record
what shipped at the time, which is what a timeline is for.

Data-fed content only; no structural change.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
emmanuelgjr added a commit that referenced this pull request Sep 18, 2026
…m the published standards (#115)

* Fix #112: recover 328 replacement characters in the registries, and guard data files

Fourteen registry files carried 328 U+FFFD replacement characters, mostly
where an em dash belongs. They ship: the registries go out in the npm
package, in docs/frameworks-registry.js and in the OSCAL/STIX exports.

Every character is recovered from the source text rather than assumed. The
registries were extracted from the mapping rows, so for each corrupted string
the text either side of the character is looked up in the Markdown and the
entry JSON, and the character that sits between them is read off. A character
is taken only when every match agrees; the window narrows from 24 to 8
characters when a description concatenates several rows, and keeps the raw
slice when only one word follows, since trimming to a word boundary could
otherwise empty it.

All 328 resolved, every one to U+2014. None was guessed, and no other
character was found.

The guard from T-ACC06 only ever read Markdown, which is why this sat
unnoticed. It now also reads data/frameworks/*.json, data/entries/*.json and
data/*.json, checking for U+FFFD alone — the arrow heuristic is tuned for
prose and diagrams, not JSON. Negative-tested: a replacement character
injected into soc2.json fails validation, naming the file and line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Title the 19 registry entries that carried only their identifier, from the
published standards

After #110 and #111, 35 mapping rows still named themselves after their id —
"Req 11.3 | Req 11.3" — because neither the mapping table nor the registry
held a title for those 19 ids. The text now comes from the standards
themselves, transcribed rather than written.

PCI DSS, 17 requirements: the headings as printed in PCI SSC's own "PCI DSS
v4.0 SAQ D for Merchants", extracted with pdftotext from the published PDF so
the wording is the document's, not a paraphrase. Req 11.3 reads "External and
internal vulnerabilities are regularly identified, prioritized, and
addressed."

SOC 2, PI1.3 and P7.1: TSP section 100, 2017 Trust Services Criteria (with
Revised Points of Focus — 2022). The AICPA download needs a session, so the
wording was taken from two independent copies of that document and compared;
both agree character for character.

Only entries whose title was empty or equal to their own id were filled, so
no curated title is overwritten. Rows with no published title available were
left alone — there are none left.

Mapping rows whose control_name is merely their control_id: 35 -> 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant