From 3a286b5081be10e133abdb2d2c9f8527873030f6 Mon Sep 17 00:00:00 2001 From: ishan Date: Sun, 2 Aug 2026 04:26:24 +0530 Subject: [PATCH] Fix two duplicate-detection defects; add measured findings Exact matcher could not report within-document duplicates. It grouped identical sentences, then discarded any group confined to a single document before pairing only across documents. On the sample corpus this under-reported by 85% (21 pairs against an actual 143). The "within-document" category label computed for exact pairs was unreachable by construction. Embedding matcher counted sentences as duplicates of themselves. The guard sliced off the top FAISS hit, which removes the self-match only when it ranks first; with byte-identical sentences the tied cosines are arbitrarily ordered, so the self-hit often landed at rank >= 1 and survived. 112 of 389 pairs (28.8%) were self-matches, which also inflated matched_sentences_pct in doc_metrics.csv. Verified after fix: exact 21 -> 143 (21 cross-document + 122 within-document), embeddings 389 -> 277 moderate and 374 -> 262 strict, self-match contamination 0.0%. SimHash unchanged at 186, confirming the change is isolated to the two intended paths. docs/FINDINGS.md records the full three-way comparison, per-phase timing, a Hamming threshold sweep, table-vs-prose attribution, and an explicit limitations section noting that no ground-truth labels exist, so all figures are yield/overlap/cost rather than precision/recall. Co-Authored-By: Claude Opus 5 --- corpus_dedup_runner.py | 25 +++--- docs/FINDINGS.md | 191 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 201 insertions(+), 15 deletions(-) create mode 100644 docs/FINDINGS.md diff --git a/corpus_dedup_runner.py b/corpus_dedup_runner.py index 38cadf5..e01648e 100644 --- a/corpus_dedup_runner.py +++ b/corpus_dedup_runner.py @@ -196,22 +196,13 @@ def run(args): exact_pairs = set() for norm, idxs in by_norm.items(): - # cross-doc pairs only + # all pairs of identical sentences, both cross-document and within-document. + # (previously this emitted cross-document pairs only, which silently dropped + # every repeated sentence confined to a single document) if len(idxs) < 2: continue - # group by doc - doc_groups = defaultdict(list) - for i in idxs: - doc_groups[all_items[i].doc_id].append(i) - docs = sorted(doc_groups.keys()) - if len(docs) < 2: - continue - # make combinations across different docs - for da, db in combinations(docs, 2): - for ia in doc_groups[da]: - for ib in doc_groups[db]: - a, b = (ia, ib) if ia < ib else (ib, ia) - exact_pairs.add((a, b)) + for a, b in combinations(sorted(idxs), 2): + exact_pairs.add((a, b)) # 4) SimHash + LSH buckets = defaultdict(list) @@ -248,7 +239,11 @@ def run(args): for i in range(n_sent): A = all_items[i] for j_idx, sim in zip(I[i][1:], D[i][1:]): - if j_idx < 0: + # Slicing off I[i][0] only removes the self-hit when it ranks first. + # With byte-identical sentences the cosine ties are arbitrarily + # ordered, so the self-hit can land at rank >= 1 and survive the + # slice. Exclude it explicitly. + if j_idx < 0 or int(j_idx) == i: continue B = all_items[j_idx] if sim >= args.embed_threshold_moderate: diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md new file mode 100644 index 0000000..779ed5c --- /dev/null +++ b/docs/FINDINGS.md @@ -0,0 +1,191 @@ +# Findings — three-way comparison of duplicate detection methods + +**Run date:** 2026-08-02 · **Corpus:** 5 `.docx`, 1,590 sentences (≥8 words) · **Hardware:** CPU only, no GPU +**Model:** `all-MiniLM-L6-v2` (local, no API cost) · **Reproduce:** `python corpus_dedup_runner.py --input_dir docs --out_dir out --use_embeddings` + +--- + +## The question + +This repo ships three duplicate-detection methods — exact match, SimHash near-duplicate, and embedding-based semantic match. The README describes all three and publishes no measurements. So: **do you need all three, and what does each actually contribute?** + +I expected a clean recall ladder — exact ⊂ SimHash ⊂ embeddings, each step buying recall at higher cost. The ladder is real. But measuring it surfaced two defects that changed the answer, and both are now fixed in this repo. + +--- + +## Before and after the fixes + +| Metric | Before | After | Change | +|---|---|---|---| +| Exact pairs | 21 | **143** | +581% | +| — cross-document | 21 | 21 | — | +| — within-document | **0** | **122** | previously unreportable | +| SimHash pairs (h≤8) | 186 | 186 | unchanged | +| Embedding pairs (cos≥0.88) | 389 | **277** | −112 false positives | +| Embedding self-match contamination | **28.8%** | **0.0%** | eliminated | +| Exact coverage of SimHash | 11% | **77%** | — | + +Union of all three methods: **289 unique pairs** (unchanged — the fixes removed false positives and surfaced pairs that were already in the union via SimHash). + +--- + +## Finding 1 — The exact matcher was silently under-reporting by 85% + +It reported 21 pairs. The corpus contains 143. + +Root cause, `corpus_dedup_runner.py`: + +```python +docs = sorted(doc_groups.keys()) +if len(docs) < 2: + continue # dropped every same-document group +for da, db in combinations(docs, 2): # then paired ACROSS documents only +``` + +Any group of identical sentences confined to one document was discarded, and within multi-document groups the same-document repeats were never paired. + +Confirmed three independent ways before fixing: +1. Output category split was **100% cross-document, 0% within-document**, while SimHash on the same corpus is 73% within-document. +2. An independent reimplementation over the same normalised sentences found 144 pairs against 21 reported. +3. The code path itself. + +**Dead code confirmed the intent.** The writer computed `"within-document" if A.doc_id == B.doc_id` for exact pairs — a branch unreachable by construction. The label was written for a capability the implementation didn't have. + +**Fix:** emit all pairs within each identical-sentence group. + +```python +for a, b in combinations(sorted(idxs), 2): + exact_pairs.add((a, b)) +``` + +**Verified:** 143 pairs, split 21 cross-document + 122 within-document. My pre-fix reimplementation predicted 144; the shipped fix yields 143, the one-pair difference tracing to my reimplementation counting 1,585 sentences against the shipped splitter's 1,590. + +**Why it mattered:** running the fast, cheap, explainable method alone — exactly what you'd do on a large corpus — reported 21 duplicate pairs where there were 143. In controlled documentation, "this SOP repeats itself internally" is often the *more* actionable finding, and it was the one being dropped. + +## Finding 2 — 28.8% of embedding matches were sentences matched to themselves + +112 of 389 pairs had `docA == docB` **and** `sentA_id == sentB_id`: the same sentence, same position, paired with itself at cosine 1.0. + +The subtle part is that a self-match guard was already present: + +```python +for j_idx, sim in zip(I[i][1:], D[i][1:]): # slice off the top hit +``` + +Slicing `I[i][0]` removes the self-hit **only when it ranks first**. This corpus contains many byte-identical sentences, so FAISS returns tied cosines of 1.0 whose order is arbitrary — the self-hit frequently lands at rank ≥1 and survives the slice. The guard works on corpora without exact duplicates and fails silently on corpora with them, which is the harder class of bug: correct-looking code that degrades exactly where the tool is most needed. + +**Fix:** exclude the self-index explicitly rather than positionally. + +```python +if j_idx < 0 or int(j_idx) == i: + continue +``` + +**Verified:** 389 → 277 moderate, 374 → 262 strict, contamination 0.0%. Both post-fix counts match the pre-fix analysis exactly. + +**Why it mattered:** the summary read as "embeddings find 2× what SimHash finds" (389 vs 186). The true figure is 277, and every `matched_sentences_pct` in `doc_metrics.csv` was inflated by the same 28.8%. + +## Finding 3 — Exact adds zero recall, but is now a strong cheap approximation + +`exact ⊂ SimHash` holds perfectly: **100% of exact pairs are also found by SimHash**, and exact contributes **0 unique pairs** to the union. That was true before the fix and remains true after. + +What changed is its value as a *fast path*. Exact now covers **77% of SimHash's pairs** (143 of 186), up from 11%. At ~1 ms versus ~900 ms, it is a legitimate first-pass filter — which it was not when it could only see 11% of the signal. + +## Finding 4 — The tunable threshold is nearly inert, and the cliff is outside the exposed range + +| Hamming threshold | pairs | cross-doc | within-doc | +|---|---|---|---| +| 0 | 154 | 32 | 122 | +| 2 | 154 | 32 | 122 | +| 4 | 163 | 36 | 127 | +| 6 *(shipped "strict")* | 175 | 43 | 132 | +| 8 *(shipped "moderate")* | 186 | 50 | 136 | +| 12 | 232 | 65 | 167 | +| 16 | 311 | 104 | 207 | +| 20 | **1,441** | 866 | 575 | + +- **h=0 → h=2 adds exactly zero pairs.** No pair in this corpus sits at Hamming distance 1 or 2 — over 64-bit SimHash of word 3-grams, even a one-word edit typically moves more than 2 bits. +- **The entire shipped strict→moderate band (6→8) moves results by 6%.** The interesting behaviour, a 4.6× explosion between h=16 and h=20, sits far outside any threshold a user would try. +- 82.8% of all SimHash hits are at h=0. + +The knob the config invites you to tune barely does anything, and the precision cliff is somewhere nobody will look. + +## Finding 5 — Algorithm choice is irrelevant to latency; document parsing dominates + +| Phase | Time | Share | +|---|---|---| +| **Read .docx** | **7,162 ms** | **88.0%** | +| Split + normalise | 37 ms | 0.5% | +| Exact match | 1 ms | 0.0% | +| SimHash signatures | 485 ms | 6.0% | +| SimHash all-pairs compare | 457 ms | 5.6% | + +Per sentence SimHash costs **902× more than exact** (595 µs vs 0.7 µs) — and it is still irrelevant against a 7.2-second parse. Enabling embeddings takes the run from 7.3 s to 69.5 s (**9.5×**), the only method choice visible in wall-clock. + +**The optimisation target is `read_docx_text`, not the matcher.** (The shipped SimHash uses LSH banding, 8 bands × 8 bits; the 457 ms above is a brute-force reference, so the shipped path is at least that fast.) + +## Finding 6 — Nearly two-thirds of the "duplication" is a table artifact + +`read_docx_text` appends every table cell to the paragraph text and `sentence_split` then treats those cells as prose. **46.5% of extracted characters come from table cells**; one document is 93.9% tables. + +| Corpus | Sentences | Exact pairs | Within-doc | Cross-doc | SimHash h≤8 | +|---|---|---|---|---|---| +| Paragraphs + tables *(as shipped)* | 1,585 | 144 | 122 | 22 | 186 | +| Paragraphs only | 908 | 54 | 41 | 13 | 71 | +| Tables only | 677 | 88 | 79 | 9 | 113 | + +- Tables are 42.7% of sentences but **62.5% of all exact duplication** and 61.8% of near-duplication. +- Table content duplicates at **13.0%** versus prose at **5.95%** — more than double. +- The headline "9.09% duplicated" is really **5.95% prose duplication** plus a large table-repetition artifact. + +Repeated table headers and boilerplate are not the plagiarism-style duplication the tool is framed around. They should be reported separately; today they aren't. **This one is not yet fixed** — see the open list below. + +## Finding 7 — Embeddings are the only method that finds genuine paraphrase + +103 pairs are found by embeddings alone (cosine 0.880–0.998, median 0.948), only one byte-identical — so these are substantively new. + +> **A:** "a Module may have a square or a round shape;" +> **B:** "The resulting module shape can be square as well as round." +> *cosine 0.8808 — negligible lexical overlap; SimHash cannot reach this.* + +The same threshold also admits false positives on numeric table rows: + +> **A:** "7 5.6 x5.6 4.2 x 4.2 Rectangular Type 12x26 1\*) 4.5 x 8 …" +> **B:** "8.1 x 14.4 6.3 x 11.2 4.5 x 8 Rectangular Type 12x36 …" +> *cosine 0.8801 — structurally similar, semantically unrelated.* + +Embeddings buy real paraphrase detection at 9.5× runtime, with a false-positive mode the other two methods don't have. + +## Finding 8 — Documentation did not match the implementation + +| `Algorithms.md` says | Code actually does | +|---|---| +| `embed_threshold_strict` default **0.8** | **0.90** | +| `embed_threshold_moderate` default **0.7** | **0.88** | +| `block_min_run` default **3** | **2** | +| Exact match "removes punctuation and special characters" | `normalize_sentence` only lowercases, folds smart quotes, collapses whitespace | +| Exact detects duplicates "across and within documents" | Was structurally impossible — **now true after the Finding 1 fix** | + +--- + +## Method recommendation + +For this corpus type: **exact as a fast first pass, SimHash at h=0–6 as the default, embeddings behind an opt-in flag.** + +Exact now covers 77% of SimHash's pairs at 1/900th the cost, so it earns its place as a filter — though never as the only stage, since it contributes no unique recall. SimHash captures 94% of what embeddings find that overlaps at all. Embeddings earn their 9.5× cost only when genuine paraphrase detection is the goal. + +## Still open + +1. **Separate table-derived from prose-derived duplicates** in reporting, or add `--exclude_tables` (Finding 6). The single number reported today conflates two different phenomena. +2. **Re-scope the Hamming config** (Finding 4) — the exposed 6–8 band does nothing useful. +3. **Optimise `read_docx_text`** if latency ever matters (Finding 5). +4. **Reconcile `Algorithms.md` with the code** (Finding 8). + +## Limitations — read before citing any number here + +- **No ground-truth labels.** Every number is yield, overlap, or cost. **No precision or recall against human judgement is reported**, because no labelled set exists. "Unique contribution" measures novelty, not correctness — a method can contribute uniquely by being uniquely wrong. +- **Single corpus, n=5 documents**, one domain, heavily table-laden. The table findings in particular may not generalise. +- The true/false-positive calls in Finding 7 are my judgement on a small sample, not adjudicated labels. +- Timing is single-run on one CPU machine; no repeats, no confidence intervals. + +**Next investigation:** hand-label a stratified sample of ~150 pairs from the union, then report precision per method and pooled recall. That converts everything above from *yield* into *quality*. Until then the defect findings (1, 2, 6, 8) stand on their own — they are properties of the code, not the corpus.