test(sysml2): grade the parser on 310 official models instead of our own fixtures

[avrabe]

First item of v0.43.0 "stop grading ourselves" — REQ-SYSML2-CONFORMANCE-001
(proposed in #441). Everything else in that release depends on this one, because
until it lands no SysML v2 number this project quotes means anything.

The suite could not fail

conformance_tests.rs had eight tests. Seven asserted on the CST's text:

assert!(!result.syntax_node().text().is_empty());          // input is non-empty by construction
assert!(text.contains("part def Vehicle"));                 // lossless CST echoes the input back
assert!(u32::from(result.syntax_node().text().len()) > 50000);   // that is a file size

Not one of the eight called errors() or ok(). parse_annex_a_simple_vehicle
reported ok while the SysML v2 specification's own Annex A model failed on line 3.

A correction to my own audit. I found that grep sysml .github/workflows/*.yml returns nothing and inferred that none of this ran in
CI. The inference was wrong. cargo nextest run --workspace has been running
these tests on every PR — the Test job's log names conformance_tests seventeen
times. They ran. They just measured nothing, which is the worse finding: a green
required check reporting on a parser that rejects the standard's own examples.
No CI wiring is added here for that reason — the new tests are picked up by the
same workspace run.

What lands

The official corpus, vendored at a pin. 310 model files from
Systems-Modeling/SysML-v2-Release @29a3d2ac — EPL-2.0, LICENSE copied
unaltered, same arrangement as the vendored OSATE corpus. 56 validation, 100
training, 96 examples, 58 KerML examples.

download-official-suite.sh is deleted. It fetched master through the
GitHub contents API with || true on every download — so a network blip silently
produced a smaller corpus, and a smaller corpus of still-failing files reads as
a better parse rate. The error path yielded the flattering answer. Its target
directories were empty, which is how 43 self-written fixtures came to stand in for
a conformance suite. tools/vendor-sysml2-corpus.sh replaces it: pinned, verifies
the checkout matches the requested SHA, and refuses to exit 0 having copied nothing.

official_corpus.rs — the bulk gate. OFFICIAL_PARSING is exact, not a
floor: below fails as a regression, above fails until the constant is raised. Same
two-sided discipline as MAX_TOO_PERMISSIVE. OFFICIAL_TOTAL is asserted
separately so a vanished corpus cannot masquerade as a parser regression.

conformance_tests.rs — rewritten to declared verdicts. Asserting whatever
the parser currently does would have been just as inert; each named file carries
its expected verdict and the construct responsible, so closing a gap turns a
named test red rather than only moving a number. The one honest assertion in the
original suite — lossless round-trip — is kept, and is exactly why the other seven
could pass on files that never parsed.

The number, measured for the first time

.sysml     0 / 252
.kerml     1 /  58
TOTAL      1 / 310   (0.3%)

A rate is a property of the corpus as much as the parser, so the failures were
classified, not just counted. Five distinct first-error kinds across 309
failing files — not 309 problems:

first error	files	construct
expected name	161	quoted names — package 'Application Layer';
expected member declaration	97	visibility — private import Objects::*;
expected SEMICOLON	41	KerML class definitions
expected package, import, or definition	9	visibility-modified import at file scope
expected definition after abstract	1	abstract type A specializes …

Quoted names and visibility modifiers alone are 258 of the 309.
REQ-SYSML2-VISIBILITY-001 is scoped to those two; the KerML class and
abstract type kinds were not in its original scope and are a finding of this
vendoring — recheck that requirement before planning its release.

Non-vacuity, executed

Four mutations, each reddening a distinct assertion, then restored:

OFFICIAL_PARSING 1 -> 0      FAILED   (regression path)
OFFICIAL_PARSING 1 -> 2      FAILED   (below-floor path)
OFFICIAL_TOTAL 310 -> 309    FAILED   (corpus guard, 2 tests)
parse predicate -> true      FAILED   (predicate stuck on one answer)
restored                     4 passed

Both verdicts of parse().ok() are separately pinned, so the count cannot be
produced by a predicate that only ever says yes.

RUSTFLAGS="-D warnings" cargo test -p spar-sysml2 → 277 passed, 0 failed.
-D warnings initially rejected the crate over an unconstructed Parses
variant; it is allowed with a note rather than deleted, because it is the variant
every entry is headed for.

Not claimed

That a parsing file is correctly lowered — Parse::ok() is a syntax predicate.
That the parser is good. Only that its score is now measured by somebody else's
models, and can only go up.

Refs #441.

🤖 Generated with Claude Code

[github-actions]

Rivet verification gate

✅ 20/20 passed

	count
Passed	20
Failed	0
Skipped (no steps)	0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

_{Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!