Report per-viewport image-scan outcome from beforePageScan/beholder - #962
Merged
Conversation
…ttles but frame is usable Add an opt-in continueOnDegradedNetwork option: when both the networkidle0 and networkidle2 waits time out, check whether the frame is nonetheless usable (main frame attached, URL matches the target, document past loading) instead of failing outright. Some pages (analytics beacons, chat widgets, open WebSocket connections) never go network-idle even though navigation itself completed. Defaults to false, so every existing caller of beforePageScan keeps throwing on a double timeout exactly as before. BeforePageScanResult gains a settled: 'idle' | 'degraded' field. A double timeout with the frame unusable throws the new NavigationUnsettledError instead of the raw Puppeteer timeout error, so callers can distinguish "navigation never settled" from frame/session loss without parsing error message strings. isFrameSettled's URL comparison normalizes a trailing slash and hash before comparing, since Chrome normalizes a root navigation's page.url() to end in '/' even when the requested URL string omits it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
BREAKING CHANGE: PageData.imageScan is a new required field ({ desktop,
mobile }: ImageScanCode | null per viewport), following the same
precedent as the 4.0.0 addition of mainContents/scrollHeight. Anyone
constructing a PageData object directly (rather than only reading
scrapeStart()'s result) must add this field.
Scraper#fetchImages now classifies each device preset's outcome via the
new IMAGE_SCAN_CODE vocabulary (0=ok, 1=degraded, 2=nav-unsettled,
3=frame-lost, 4=scroll-height-exceeded, 255=unknown) instead of only
emitting a free-text changePhase message: success records ok/degraded
based on @d-zero/puppeteer-page-scan's new settled result, a
scrollHeight-limit skip records scroll-height-exceeded, and a caught
exception is classified by the new classifyImageScanError (reusing
puppeteer-scroll's isTransientFrameError, plus "Not attached to an
active page"/"Target closed" for reload/goto-specific failures).
#fetchImages opts into @d-zero/puppeteer-page-scan's new
continueOnDegradedNetwork so a page whose network never settles still
yields images (flagged degraded) instead of losing that viewport
outright.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
beforePageScan'snetworkidle0/networkidle2fallback used to fail outright when both timed out. Add an opt-incontinueOnDegradedNetworkoption: on a double timeout, check whether the frame is still usable (main frame attached, URL matches, document pastloading) and continue assettled: 'degraded'instead of throwing — pages with analytics beacons, chat widgets, or open WebSocket connections never go network-idle even though navigation completed. Defaults tofalse, so every existing caller keeps its current throw-on-double-timeout behavior unchanged.NavigationUnsettledErrorinstead of the raw Puppeteer timeout error, so callers can classify "navigation never settled" without parsing error message strings.beholder'sScraper#fetchImagesopts into the new option and records each device preset's outcome as a small integer (IMAGE_SCAN_CODE:ok/degraded/nav-unsettled/frame-lost/scroll-height-exceeded/unknown) onPageData.imageScan, via the newclassifyImageScanErrorhelper — instead of only emitting a free-textchangePhaselog message that a caller would have had to string-match to know why a viewport's image scan was abandoned.Breaking change
PageData.imageScanis a new required field, the same category of change as the4.0.0addition ofmainContents/scrollHeight. Anyone constructing aPageDataobject directly (rather than only readingscrapeStart()'s result) needs to add it.@d-zero/puppeteer-page-scan's change is additive/opt-in and is not breaking.Test plan
yarn lint/yarn build/yarn testall pass at the repo rootpuppeteer-page-scan: newbefore-page-scan.spec.tscases cover idle / networkidle0→idle2 fallback / degraded-continue / eachNavigationUnsettledErrortrigger (detached frame, URL mismatch, readyState) / non-timeout errors at both layers / thecontinueOnDegradedNetwork: falsebackward-compat pathpuppeteer-page-scan: newis-frame-settled.spec.tscovers the trailing-slash/hash URL normalizationbeholder: newclassify-image-scan-error.spec.tscovers all classification branchesbeforePageScanconsumers (print,puppeteer-screenshot,a11y-check-core,replicator,anatomist) to confirm no behavior change for callers that don't opt in🤖 Generated with Claude Code