Skip to content

chore(deps): update dependency @nitpicker/crawler to ^0.21.0 - #1038

Open
renovate[bot] wants to merge 1 commit into
devfrom
renovate/nitpicker-crawler-0.x
Open

chore(deps): update dependency @nitpicker/crawler to ^0.21.0#1038
renovate[bot] wants to merge 1 commit into
devfrom
renovate/nitpicker-crawler-0.x

Conversation

@renovate

@renovate renovate Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

This PR contains the following updates:

Package Change Age Confidence
@nitpicker/crawler (source) ^0.18.0^0.21.0 age confidence

Release Notes

d-zero-dev/nitpicker (@​nitpicker/crawler)

v0.21.0

Compare Source

Note: Version bump only for package @​nitpicker/crawler

v0.20.0

Compare Source

Bug Fixes
  • crawler: stop cross-host redirect leak; honor fetchExternal in list (3b3cc12)

v0.19.0

Compare Source

Bug Fixes
  • crawler: compute page body hash once per page instead of twice (f2fbda5)

  • crawler: route self-healing and session notices away from bare console.* (deb5684)

  • feat(crawler)!: implement Symbol.asyncDispose for archive lifecycle (de60624)

  • feat(crawler)!: replace Wappalyzer-only page_tags with technology detection (4b81a47)

Features
  • crawler: assert puppeteer install is shared with beholder (5cbf0ed)
  • crawler: eliminate silent stretches across archive I/O (issue #​294) (eabdd07)
  • crawler: export setup-phase label sequences for TaskList consumers (793b952)
  • crawler: log page_tags row count before migration (#​294) (5f535a7)
BREAKING CHANGES
  • requires Node >=24.11 for native Explicit Resource
    Management (using/await using) support.

ArchiveAccessor (and, via polymorphism, Archive) and CrawlerOrchestrator
now implement Symbol.asyncDispose, delegating to their existing
close()/garbageCollect() logic. Callers can use await using instead
of a manual try/finally around teardown.

  • page_tags is removed outright in favor of two new
    tables: technology_signals (raw per-signal evidence, one row per
    signal) and page_technologies (noisy-OR confidence-combined roll-up,
    one row per page x technology). Existing archives are converted via a
    read-time migration (migrate-page-tags-to-page-technologies.ts for
    0.13+ archives, a special case in retarget-legacy-fk-tables.ts for the
    0.10->0.13 upgrade path) that drops page_tags after conversion.
    Page.getTags() and the TagsSummary/TagInventoryEntry types are
    removed along with it; use Page.getTechnologySignals() /
    Page.getPageTechnologies() instead.

Also adds a post-crawl JS license-comment scan (scanJsResourcesForTechnologySignals)
that re-fetches the leading bytes of not-yet-scanned internal JS
resources to detect technologies whose signature lives in the JS body
rather than the HTML — a distinct "post-crawl network enrichment"
category from the existing post-hoc-marking and backfill patterns,
since it is the only step that performs network I/O after the crawl
itself finishes. Each resource is scanned at most once (recorded in
technology_js_scan_cache), and a match folds into every page that
references the resource by recombining that page's full signal set
under the same Scoped-Replace invariant insertTechnologies uses.

Also detects Web Components (custom elements, tag names containing a
hyphen and not one of the 8 reserved SVG/MathML names) within the main
content region, at the same grain as the existing button/table/etc.
categories (page_main_content_custom_elements, main_content_custom_element_count).

0.18.1 (2026-08-12)

Bug Fixes
  • crawler: close two more demotion sites the /code-review pass on bf75031 found (69c3597)
  • crawler: guard is_external against demotion on an already-scraped internal page (bf75031)
  • query: exclude out-of-scope pages from the directory tree (56a2c6b)

v0.18.1

Compare Source

Bug Fixes
  • crawler: close two more demotion sites the /code-review pass on bf75031 found (69c3597)
  • crawler: guard is_external against demotion on an already-scraped internal page (bf75031)
  • query: exclude out-of-scope pages from the directory tree (56a2c6b)

Configuration

📅 Schedule: (UTC)

  • Branch creation
    • At any time (no schedule defined)
  • Automerge
    • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

github-actions[bot]
github-actions Bot previously approved these changes Aug 26, 2026
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch 4 times, most recently from d95c14d to 171aff4 Compare September 1, 2026 02:59
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch from 171aff4 to 91b382e Compare September 1, 2026 10:40
github-actions[bot]
github-actions Bot previously approved these changes Sep 1, 2026
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch 3 times, most recently from f93bd0a to 620ea18 Compare September 2, 2026 22:34
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch from 620ea18 to deaf643 Compare September 3, 2026 03:28
@renovate renovate Bot changed the title chore(deps): update dependency @nitpicker/crawler to ^0.19.0 chore(deps): update dependency @nitpicker/crawler to ^0.20.0 Sep 3, 2026
github-actions[bot]
github-actions Bot previously approved these changes Sep 3, 2026
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch from deaf643 to da40a2e Compare September 3, 2026 13:30
@renovate renovate Bot changed the title chore(deps): update dependency @nitpicker/crawler to ^0.20.0 chore(deps): update dependency @nitpicker/crawler to ^0.21.0 Sep 3, 2026
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch 3 times, most recently from 52ddab7 to 637d666 Compare September 7, 2026 20:13
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch 5 times, most recently from 7ee46a1 to a47a304 Compare September 8, 2026 10:55
@renovate
renovate Bot force-pushed the renovate/nitpicker-crawler-0.x branch from a47a304 to f57b700 Compare September 8, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants