Corpus expansion: cover the shapes the rules are judged on

[aksOps]

Question

What shapes must the detection corpus carry before any generic-rule change on this map can be called measured, and what do recall and precision read once it does?

Parent: #48

The corpus is 21 hand-written config files. It cannot see the shapes that produced the worst regression in the project's history, and it cannot measure the three deferred debts (#44, '=[^=]', base64) because none of their shapes are in it.

Known-missing shapes, in rough order of how badly they bit:

• C amalgamations and generated bindings - pub const PPPIOCSPASS: u32 = 2148561991;, let token = Literal::u8_unsuffixed(0);. These produced ~870 indefensible findings when the corpus could not see them.
• XML / web.config / .csproj - the shapes both this corpus and the gitleaks corpus cover worst, and the reason '=[^=]' in generic-markup-config-secret was left unmeasured.
• Kubernetes manifests - Secret data: fields are base64 by spec; needed to measure whether decoding is worth its false-positive cost.
• All-vocabulary passwords in credentialed URLs - AdminPassword, SuperSecretKey, the generic-credentialed-url: an all-vocabulary password is suppressed (AdminPassword, SuperSecretKey) #44 shape.
• Lockfiles, vendored trees, minified JS - high-entropy noise that must stay unreported.

Deliverable: expanded corpus with manifest rows, plus a recorded recall/precision baseline and a finding count on a large real tree (the 1.8 GB registry checkout used previously) so every later ticket has a before to compare against.

No rule changes in this ticket. Measurement only - a corpus that moves at the same time as a rule proves nothing.

[aksOps]

Resolved by grilling 2026-08-07 (map #48).

Sourcing: hybrid. The committed corpus stays synthetic - hand-written shapes, credentials assembled at runtime, push-protection-safe, license-clean. The sanctioned external measurement targets are the ~1.8 GB local cargo-registry checkout (precision) and a leaky-repo clone (recall + parity reuse). Both ungated and reproducible; SecretBench rejected for gated access and real-credential handling risk.

Precision enforcement: split. A committed noise corpus (C amalgamations, generated bindings, lockfiles, minified JS) carries zero-finding floors enforced in CI. The registry tree stays a scripted, manual, pre-release measurement with recorded numbers.

Floors: per-family. Each shape family carries its own recall floor, ratcheted as rules improve. Precision stays a global 1.0 on classified rows. A weak new family cannot erode a mature family's guarantee.

Families to add, from the ticket: C amalgamations + generated bindings (noise), XML/web.config/.csproj, Kubernetes manifests (including base64 data: values - these also serve the #52 gate), all-vocabulary credentialed-URL passwords (#44's shape), lockfiles/vendored/minified (noise). Measurement only in this ticket: no rule changes.

[aksOps]

Baseline measurements, recorded per this ticket's resolution. Branch feat/measured-detection (PR #54), siloscan v1.5.1 rules, release build, 2026-08-07.

Corpus (committed, CI-enforced)

family	recall	reported/positives	floor	known gap
core	0.9881	166/168	0.9880	-
keys	1.0000	4/4	1.0000	-
k8s	0.8333	15/18	0.8333	#53
xml	0.5926	16/27	0.5925	#51
urls	0.4783	11/23	0.4782	#44

Overall recall 212/240 = 0.8833. Precision 1.0000 (311/311 negatives clean), global floor unchanged. 60 files, 551 manifest rows.

Registry tree (external precision target, manual)

~/.cargo/registry/src - 1.8 GB, 84,063 files measured, 4,643 skipped as binary, 38.0M lines. Wall clock 241.3 s. 434 findings across 10 rules:

238 private-key, 79 generic-secret-assignment, 33 age-secret-key, 27 generic-authorization-header, 27 generic-credentialed-url, 13 github-pat, 9 aws-amazon-bedrock-api-key-short-lived, 4 generic-password-assignment, 3 square-access-token, 1 jwt.

Many are deliberately committed test fixtures inside crates, so 434 is an upper bound on true FPs - but it is the number every future rule change is compared against. Peak RSS 11.4 GB, consistent with the accepted ~6x memory gap.

leaky-repo (external recall target, manual)

31 findings across 16 files; 17 of 43 seeded secret files detected. Of the 26 zero-finding files, 2 are informative-only upstream (public key, zero-risk entropy file). The 24 risk-bearing misses cluster into recurring themes, each a candidate shape for the corpus and for rule work on this map:

• no assignment operator: .netrc, .pgpass, .htpasswd, proftpdpasswd, .esmtprc (directive/colon-delimited grammars)
• placeholder-grade values: hunter22, swordfish, password, values echoing their key (several files) - suppression working as designed, arguably
• encoded blobs: .docker/.dockercfg and config.json base64 auth values - the Base64-encoded credentials: decode, or document as out of reach #52 gate's exact territory
• hash formats: bcrypt in SQL dumps, crypt in etc/shadow - out of scope for credential rules, needs a decision
• syntax the generic rules do not parse: XML attributes (WebServers.xml, dbeaver), function-call arguments (wp-config.php define(), salesforce.js), values with brackets/parens (django SECRET_KEY)
• non-PEM key formats: PuTTY PPK

Scan outputs preserved at the session's measurement directory; leaky-repo re-clones reproducibly from Plazmaz/leaky-repo.