research(recall): diversify near-duplicate candidate clusters

[heybeaux]

Summary

Stacked on #324. Adds a default-off, reversible candidate-diversity path for near-duplicate rescue clusters.

• RECALL_CANDIDATE_POOL_DEPTH=12 expands only the rerank candidate pool; effective depth is never below the caller requested limit.
• RECALL_NEAR_DUPLICATE_CLUSTER_LIMIT=2 caps initially selected raw-text Jaccard clusters, then relevance-order backfill preserves full pages.
• Diversity is inert unless RECALL_RERANK_SCALE_FIX=true.
• Cluster-only scans are bounded.

Evidence

Targeted 20-query near-duplicate corpus, snapshot restored before every query:

• Depth alone (10/12/20/50): 14/20 gold@5, MRR 0.700.
• Depth 12 + cluster cap 2: 20/20 gold@5/@10, MRR 0.800.
• Gold@1 remains 14/20; all pages retain 10 results; repeats are bit-stable.

Corrected 81-query real-embedding gate (1,210 fixtures asserted; usage snapshot per query):

• P@5: 96.77% off and on.
• Recall@10: 97.26% off and on.
• Isolation/forbidden injection: 100% / 0 both.
• MRR: 0.91262 → 0.91282.
• No page-fullness or stability regression.

Verification

• Independent review: PASS WITH NOTES after blocker repair.
• 46 focused tests passed.
• Build: 873 files, 0 TypeScript issues.
• Default behavior remains off. No schema or production-data changes.

Rollout

Do not default-enable yet. G1/G3/G4/G5 graduated corpora have generators but no valid embedding-gated receipts because the local embedding path produced NaNs/dimension mismatch. Keep this as a research/canary flag until those runs are valid.

Dependency

This PR targets the #324 branch and should merge only after/with #324.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 397cfcaa-6fab-449d-b589-98de88042d56

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}