Skip to content

Fix within-document blindness in exact matcher and self-matches in embedding search #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ishankgp wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix within-document blindness in exact matcher and self-matches in embedding search #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 10 additions & 15 deletions corpus_dedup_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,22 +196,13 @@ def run(args):

exact_pairs = set()
for norm, idxs in by_norm.items():
# cross-doc pairs only
# all pairs of identical sentences, both cross-document and within-document.
# (previously this emitted cross-document pairs only, which silently dropped
# every repeated sentence confined to a single document)
if len(idxs) < 2:
continue
# group by doc
doc_groups = defaultdict(list)
for i in idxs:
doc_groups[all_items[i].doc_id].append(i)
docs = sorted(doc_groups.keys())
if len(docs) < 2:
continue
# make combinations across different docs
for da, db in combinations(docs, 2):
for ia in doc_groups[da]:
for ib in doc_groups[db]:
a, b = (ia, ib) if ia < ib else (ib, ia)
exact_pairs.add((a, b))
for a, b in combinations(sorted(idxs), 2):
exact_pairs.add((a, b))

# 4) SimHash + LSH
buckets = defaultdict(list)
Expand Down Expand Up @@ -248,7 +239,11 @@ def run(args):
for i in range(n_sent):
A = all_items[i]
for j_idx, sim in zip(I[i][1:], D[i][1:]):
if j_idx < 0:
# Slicing off I[i][0] only removes the self-hit when it ranks first.
# With byte-identical sentences the cosine ties are arbitrarily
# ordered, so the self-hit can land at rank >= 1 and survive the
# slice. Exclude it explicitly.
if j_idx < 0 or int(j_idx) == i:
continue
B = all_items[j_idx]
if sim >= args.embed_threshold_moderate:
Expand Down
191 changes: 191 additions & 0 deletions docs/FINDINGS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# Findings — three-way comparison of duplicate detection methods

**Run date:** 2026-08-02 · **Corpus:** 5 `.docx`, 1,590 sentences (≥8 words) · **Hardware:** CPU only, no GPU
**Model:** `all-MiniLM-L6-v2` (local, no API cost) · **Reproduce:** `python corpus_dedup_runner.py --input_dir docs --out_dir out --use_embeddings`

---

## The question

This repo ships three duplicate-detection methods — exact match, SimHash near-duplicate, and embedding-based semantic match. The README describes all three and publishes no measurements. So: **do you need all three, and what does each actually contribute?**

I expected a clean recall ladder — exact ⊂ SimHash ⊂ embeddings, each step buying recall at higher cost. The ladder is real. But measuring it surfaced two defects that changed the answer, and both are now fixed in this repo.

---

## Before and after the fixes

| Metric | Before | After | Change |
|---|---|---|---|
| Exact pairs | 21 | **143** | +581% |
| — cross-document | 21 | 21 | — |
| — within-document | **0** | **122** | previously unreportable |
| SimHash pairs (h≤8) | 186 | 186 | unchanged |
| Embedding pairs (cos≥0.88) | 389 | **277** | −112 false positives |
| Embedding self-match contamination | **28.8%** | **0.0%** | eliminated |
| Exact coverage of SimHash | 11% | **77%** | — |

Union of all three methods: **289 unique pairs** (unchanged — the fixes removed false positives and surfaced pairs that were already in the union via SimHash).

---

## Finding 1 — The exact matcher was silently under-reporting by 85%

It reported 21 pairs. The corpus contains 143.

Root cause, `corpus_dedup_runner.py`:

```python
docs = sorted(doc_groups.keys())
if len(docs) < 2:
continue # dropped every same-document group
for da, db in combinations(docs, 2): # then paired ACROSS documents only
```

Any group of identical sentences confined to one document was discarded, and within multi-document groups the same-document repeats were never paired.

Confirmed three independent ways before fixing:
1. Output category split was **100% cross-document, 0% within-document**, while SimHash on the same corpus is 73% within-document.
2. An independent reimplementation over the same normalised sentences found 144 pairs against 21 reported.
3. The code path itself.

**Dead code confirmed the intent.** The writer computed `"within-document" if A.doc_id == B.doc_id` for exact pairs — a branch unreachable by construction. The label was written for a capability the implementation didn't have.

**Fix:** emit all pairs within each identical-sentence group.

```python
for a, b in combinations(sorted(idxs), 2):
exact_pairs.add((a, b))
```

**Verified:** 143 pairs, split 21 cross-document + 122 within-document. My pre-fix reimplementation predicted 144; the shipped fix yields 143, the one-pair difference tracing to my reimplementation counting 1,585 sentences against the shipped splitter's 1,590.

**Why it mattered:** running the fast, cheap, explainable method alone — exactly what you'd do on a large corpus — reported 21 duplicate pairs where there were 143. In controlled documentation, "this SOP repeats itself internally" is often the *more* actionable finding, and it was the one being dropped.

## Finding 2 — 28.8% of embedding matches were sentences matched to themselves

112 of 389 pairs had `docA == docB` **and** `sentA_id == sentB_id`: the same sentence, same position, paired with itself at cosine 1.0.

The subtle part is that a self-match guard was already present:

```python
for j_idx, sim in zip(I[i][1:], D[i][1:]): # slice off the top hit
```

Slicing `I[i][0]` removes the self-hit **only when it ranks first**. This corpus contains many byte-identical sentences, so FAISS returns tied cosines of 1.0 whose order is arbitrary — the self-hit frequently lands at rank ≥1 and survives the slice. The guard works on corpora without exact duplicates and fails silently on corpora with them, which is the harder class of bug: correct-looking code that degrades exactly where the tool is most needed.

**Fix:** exclude the self-index explicitly rather than positionally.

```python
if j_idx < 0 or int(j_idx) == i:
continue
```

**Verified:** 389 → 277 moderate, 374 → 262 strict, contamination 0.0%. Both post-fix counts match the pre-fix analysis exactly.

**Why it mattered:** the summary read as "embeddings find 2× what SimHash finds" (389 vs 186). The true figure is 277, and every `matched_sentences_pct` in `doc_metrics.csv` was inflated by the same 28.8%.

## Finding 3 — Exact adds zero recall, but is now a strong cheap approximation

`exact ⊂ SimHash` holds perfectly: **100% of exact pairs are also found by SimHash**, and exact contributes **0 unique pairs** to the union. That was true before the fix and remains true after.

What changed is its value as a *fast path*. Exact now covers **77% of SimHash's pairs** (143 of 186), up from 11%. At ~1 ms versus ~900 ms, it is a legitimate first-pass filter — which it was not when it could only see 11% of the signal.

## Finding 4 — The tunable threshold is nearly inert, and the cliff is outside the exposed range

| Hamming threshold | pairs | cross-doc | within-doc |
|---|---|---|---|
| 0 | 154 | 32 | 122 |
| 2 | 154 | 32 | 122 |
| 4 | 163 | 36 | 127 |
| 6 *(shipped "strict")* | 175 | 43 | 132 |
| 8 *(shipped "moderate")* | 186 | 50 | 136 |
| 12 | 232 | 65 | 167 |
| 16 | 311 | 104 | 207 |
| 20 | **1,441** | 866 | 575 |

- **h=0 → h=2 adds exactly zero pairs.** No pair in this corpus sits at Hamming distance 1 or 2 — over 64-bit SimHash of word 3-grams, even a one-word edit typically moves more than 2 bits.
- **The entire shipped strict→moderate band (6→8) moves results by 6%.** The interesting behaviour, a 4.6× explosion between h=16 and h=20, sits far outside any threshold a user would try.
- 82.8% of all SimHash hits are at h=0.

The knob the config invites you to tune barely does anything, and the precision cliff is somewhere nobody will look.

## Finding 5 — Algorithm choice is irrelevant to latency; document parsing dominates

| Phase | Time | Share |
|---|---|---|
| **Read .docx** | **7,162 ms** | **88.0%** |
| Split + normalise | 37 ms | 0.5% |
| Exact match | 1 ms | 0.0% |
| SimHash signatures | 485 ms | 6.0% |
| SimHash all-pairs compare | 457 ms | 5.6% |

Per sentence SimHash costs **902× more than exact** (595 µs vs 0.7 µs) — and it is still irrelevant against a 7.2-second parse. Enabling embeddings takes the run from 7.3 s to 69.5 s (**9.5×**), the only method choice visible in wall-clock.

**The optimisation target is `read_docx_text`, not the matcher.** (The shipped SimHash uses LSH banding, 8 bands × 8 bits; the 457 ms above is a brute-force reference, so the shipped path is at least that fast.)

## Finding 6 — Nearly two-thirds of the "duplication" is a table artifact

`read_docx_text` appends every table cell to the paragraph text and `sentence_split` then treats those cells as prose. **46.5% of extracted characters come from table cells**; one document is 93.9% tables.

| Corpus | Sentences | Exact pairs | Within-doc | Cross-doc | SimHash h≤8 |
|---|---|---|---|---|---|
| Paragraphs + tables *(as shipped)* | 1,585 | 144 | 122 | 22 | 186 |
| Paragraphs only | 908 | 54 | 41 | 13 | 71 |
| Tables only | 677 | 88 | 79 | 9 | 113 |

- Tables are 42.7% of sentences but **62.5% of all exact duplication** and 61.8% of near-duplication.
- Table content duplicates at **13.0%** versus prose at **5.95%** — more than double.
- The headline "9.09% duplicated" is really **5.95% prose duplication** plus a large table-repetition artifact.

Repeated table headers and boilerplate are not the plagiarism-style duplication the tool is framed around. They should be reported separately; today they aren't. **This one is not yet fixed** — see the open list below.

## Finding 7 — Embeddings are the only method that finds genuine paraphrase

103 pairs are found by embeddings alone (cosine 0.880–0.998, median 0.948), only one byte-identical — so these are substantively new.

> **A:** "a Module may have a square or a round shape;"
> **B:** "The resulting module shape can be square as well as round."
> *cosine 0.8808 — negligible lexical overlap; SimHash cannot reach this.*

The same threshold also admits false positives on numeric table rows:

> **A:** "7 5.6 x5.6 4.2 x 4.2 Rectangular Type 12x26 1\*) 4.5 x 8 …"
> **B:** "8.1 x 14.4 6.3 x 11.2 4.5 x 8 Rectangular Type 12x36 …"
> *cosine 0.8801 — structurally similar, semantically unrelated.*

Embeddings buy real paraphrase detection at 9.5× runtime, with a false-positive mode the other two methods don't have.

## Finding 8 — Documentation did not match the implementation

| `Algorithms.md` says | Code actually does |
|---|---|
| `embed_threshold_strict` default **0.8** | **0.90** |
| `embed_threshold_moderate` default **0.7** | **0.88** |
| `block_min_run` default **3** | **2** |
| Exact match "removes punctuation and special characters" | `normalize_sentence` only lowercases, folds smart quotes, collapses whitespace |
| Exact detects duplicates "across and within documents" | Was structurally impossible — **now true after the Finding 1 fix** |

---

## Method recommendation

For this corpus type: **exact as a fast first pass, SimHash at h=0–6 as the default, embeddings behind an opt-in flag.**

Exact now covers 77% of SimHash's pairs at 1/900th the cost, so it earns its place as a filter — though never as the only stage, since it contributes no unique recall. SimHash captures 94% of what embeddings find that overlaps at all. Embeddings earn their 9.5× cost only when genuine paraphrase detection is the goal.

## Still open

1. **Separate table-derived from prose-derived duplicates** in reporting, or add `--exclude_tables` (Finding 6). The single number reported today conflates two different phenomena.
2. **Re-scope the Hamming config** (Finding 4) — the exposed 6–8 band does nothing useful.
3. **Optimise `read_docx_text`** if latency ever matters (Finding 5).
4. **Reconcile `Algorithms.md` with the code** (Finding 8).

## Limitations — read before citing any number here

- **No ground-truth labels.** Every number is yield, overlap, or cost. **No precision or recall against human judgement is reported**, because no labelled set exists. "Unique contribution" measures novelty, not correctness — a method can contribute uniquely by being uniquely wrong.
- **Single corpus, n=5 documents**, one domain, heavily table-laden. The table findings in particular may not generalise.
- The true/false-positive calls in Finding 7 are my judgement on a small sample, not adjudicated labels.
- Timing is single-run on one CPU machine; no repeats, no confidence intervals.

**Next investigation:** hand-label a stratified sample of ~150 pairs from the union, then report precision per method and pooled recall. That converts everything above from *yield* into *quality*. Until then the defect findings (1, 2, 6, 8) stand on their own — they are properties of the code, not the corpus.