extractgraph

Getting the page is one problem. Getting the data out of it is a different one.

browsergraph solves the first — engine × stealth × transport, escalating when a configuration fails. This solves the second, and it is not solved by a better regex.

It is solved by asking several independent questions and comparing the answers.

from extractgraph import Schema, extract

result = extract(html, Schema.of("email", "phone", "name", "social"), url=url)

result.get("phone")            # '+13035550142'
result.confidence("phone")     # 1.00  — four paths agreed
result.conflicts               # where the paths disagreed, kept rather than hidden

https://acme.example
  + email        sales@acme.example      1.00  links +jsonld,label,pattern
  + phone        +13035550142            1.00  links +jsonld,label,pattern
    name         Acme Roofing Ltd        0.95  jsonld

---

Why paths, and not one good extractor

A single regex is one path with nothing to corroborate it. This library exists because of a specific failure: running a contact extractor against python.org reported a phone number. python.org has no phone number. Its homepage prints a Fibonacci series, and

0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987

contains 377 610 987 — nine digits in three groups. No fixture would ever have contained that. The pattern was not wrong; it was alone.

Independent paths that agree give you confidence. When they disagree, that disagreement is itself the signal — and it is reported rather than resolved silently by taking the first answer.

The paths

Ranked by how much the evidence constrains the answer, not by cleverness.

path	prior	evidence
links	0.97	a mailto: / tel: href — the author chose a scheme because they knew the type
jsonld	0.95	schema.org JSON-LD — the author's machine-readable statement
microdata	0.85	itemprop, OpenGraph, meta tags
label	0.80	a value immediately after the word "Phone", within one block
dom	0.70	a value inside an element named .contact-email
pattern	0.45	digits somewhere on the page that look right

Only the last is what most extraction code does, and it is the one that found the Fibonacci series. It stays — plenty of pages carry no better evidence — but it carries the lowest prior, so anything that disagrees with it wins.

Corroboration saturates and is capped below the gap between a strong and a weak path, so three guesses agreeing never outrank one fact.

Provenance is the product

Every value knows where it came from and what agreed with it.

v = result.values["phone"][0]
v.path              # 'links'
v.evidence          # 'tel: link'
v.corroborated_by   # ('jsonld', 'label', 'pattern')
v.confidence, v.score

A phone number from a tel: link and one from pattern-matched digits are not the same kind of fact. Anything that treats them identically is throwing away the most useful thing extraction knows.

Disagreement is kept

! name: chose 'Python.org' (microdata, 0.80) over 'Welcome to Python.org' (microdata, 0.80)

That is a real result from python.org. Both answers are defensible; the library picks one and tells you it did.

Fields decide what counts

Validation, normalisation and identity live on the field, not in the paths — so a rule is written once and no path can bypass it.

PHONE.accept(" (303) 555-0142 ")   # '3035550142'
PHONE.accept("377 610 987")        # None — nine bare digits are not dialable

identity is separate from normalise, and earns its keep: tel:+13035550142, (303) 555-0142 and the JSON-LD value are one phone number, not three findings.

Rules are deliberately strict. In extraction a false positive is silent and permanent — it lands in a dataset and is never questioned again — while a miss is a visible empty field that someone notices.

Some rules exist only because a real page produced them:

• nine bare digits are not a phone number (python.org's Fibonacci series)
• a number flanked by other numbers is part of a sequence (stats tables, code output)
• a social host in a query string is not a social link (news.ycombinator.com/from?site=github.com/x)
• github.com/owner/repo/issues is not anybody's profile (Hacker News's front page)
• a profile in the page chrome outranks one in the body — a site's own accounts live in its footer; body links belong to whatever the page is about

Two strategies

extract(html, schema)               # every path, then fuse — the only way to corroborate
cascade(html, schema, floor=0.8)    # best-first, stop when every field is answered

cascade is cheaper and right for harvesting at scale, but it returns values with no corroborated_by — fewer paths ran, so agreement was never available. That is a deliberate trade, not a default.

Failure is contained

One path failing must not lose the others — that is the entire reason there are several. Malformed JSON-LD is extremely common, and it costs exactly the JSON-LD path:

result.paths   # [PathResult(path='jsonld', found=0, error='JSONDecodeError: ...'), ...]

Install

pip install extractgraph          # stdlib only — no lxml, no C extension

Pairs with browsergraph, but depends on nothing:

from browsergraph import Engine, Spec
from browsergraph.drivers import build
from extractgraph import Schema, extract

browser = build(Spec(engine=Engine.HTTP)); browser.start()
browser.goto(url)
result = extract(browser.html(), Schema.of("email", "phone"), url=url)

MIT.