Skip to content

test: end-to-end suite for the extension with visual regression - #84

Open
evg4b wants to merge 25 commits into
mainfrom
test/e2e-playwright
Open

test: end-to-end suite for the extension with visual regression#84
evg4b wants to merge 25 commits into
mainfrom
test/e2e-playwright

Conversation

@evg4b

@evg4b evg4b commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Adds an end-to-end suite that runs the packed extension in a real Chromium, plus committed screenshots so any change that alters rendering shows up as a CI failure with a diff.

What it covers

Spec
formatter.spec.ts tree rendering, big numbers kept intact, collapse/expand, non-JSON pages left alone, dark + light screenshots
toolbar.spec.ts raw tab returns the response byte for byte, tab switching, download dropdown
query.spec.ts a jq expression typed and submitted, and the inline error for an undefined filter
options.spec.ts every section renders, and unchecking a toolbar button reaches the content script through synced storage
faq.spec.ts manual renders, sidebar navigation

20 tests, ~7s.

Reaching the UI

The content script builds everything inside closed shadow roots, which Playwright locators cannot enter. e2e/support/shadow.ts resolves paths over CDP instead — segments separated by >>>, each queried inside the shadow root of the previous match:

await (await shadow.find('body >>> mjf-toolbox >>> button[data-type="raw"]')).click();

Clicks go through page.mouse at the element's real box, so they are genuine user input. Known paths live in e2e/support/ui.ts rather than in the specs. The options and FAQ pages use open shadow roots and need none of this.

Pages are served by fulfilling the request in Playwright, so there is no fixture server to manage.

Visual comparison

Baselines are committed under e2e/*.spec.ts-snapshots/. Font rendering is host-specific, so the CI job runs inside mcr.microsoft.com/playwright:v1.63.0-noble — the same image make e2e-update uses to regenerate them. Only the Linux PNGs are committed; macOS and Windows ones are ignored.

make e2e           # build, then run the suite
make e2e-update    # regenerate the baselines (Docker)

CI

New E2E Tests job, depending on build and reusing the extension artifact it already uploads. The Playwright HTML report (with actual/diff images) is uploaded on failure.

Noticed while doing this

The Linux baselines render in a fallback font rather than the bundled Monaco: chrome-extension://__MSG_@@extension_id__/Monaco.woff ships verbatim in content-script.js, and that placeholder is only substituted in manifest-declared CSS, not in styles injected at runtime. macOS hides it via local('Monaco'). Left alone here — fixing it would rewrite these baselines — but worth a follow-up.

🤖 Generated with Claude Code

evg4b
evg4b commented Sep 15, 2026
View reviewed changes
Comment thread src/content-script/extension.ts Outdated
evg4b
evg4b commented Sep 15, 2026
View reviewed changes
Comment thread src/content-script/content-script.scss
evg4b and others added 25 commits September 17, 2026 14:18
@evg4b @claude
chore(e2e): add playwright for browser-level tests
10f65c9 
The suite drives a real Chromium with dist/ loaded as an unpacked
extension, so it needs a production build rather than the rstest
environment. Visual baselines are kept per platform and only the Linux
ones are committed, since that is where CI compares them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): reach the extension UI through its closed shadow roots
e3094b9 
Playwright locators stop at closed shadow roots, and the content script
puts its whole UI behind two of them. CDP still walks through, so the
`shadow` fixture resolves paths by hand: segments separated by `>>>`,
each queried inside the shadow root of the previous match.

Matches are turned into remote object handles right away, because node
ids are discarded every time the document is re-read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover the formatted view
62b4c03 
Checks the rendered tree against a document with values the formatter
handles specially — a number larger than JavaScript can hold, null,
empty containers — plus collapsing and expanding, and that pages without
JSON are left untouched.

The screenshots cover both themes, so a styling change that survives the
text assertions still has to be looked at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover the toolbar tabs and download menu
f566e6f 
Asserts the raw tab hands back the response byte for byte, that
switching tabs keeps the formatted tree, and screenshots the open
download dropdown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover jq queries
26fb4a0 
Runs an expression through the input the way a user would — typing and
pressing Enter — and checks both the rendered result and the inline
message an undefined filter produces.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover the options and manual pages
9311961 
Beyond rendering, unchecking a toolbar button on the options page has to
reach the content script through synced storage, which is the one path
no unit test can exercise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
ci: run the end-to-end suite against the built extension
f83f89d 
The job reuses the artifact the build job already uploads and runs in
the same Playwright image the committed screenshots were generated in —
otherwise the runner's fonts leak into the comparison and every visual
assertion fails. `make e2e-update` pins the same image for regenerating
them locally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
docs: describe the end-to-end suite
bb1ce9a 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): check the formatted tree copies back as valid JSON
b8b8a4d 
Selecting the tree and copying it has to round-trip through JSON.parse,
including with a node collapsed, where the `...` and `// N items`
markers sit in the middle of the text and are kept out of the selection
by `user-select: none`.

Chromium handles select all and copy as editing commands rather than as
plain key presses, and the selection only reaches a closed shadow root
once it already sits inside one, so the fixture clicks into the tree
first and drives both commands over CDP.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): keep one screenshot per view instead of one per platform
b981097 
Text rendering depends on the fonts installed on the machine, so a
baseline is only meaningful next to the renderer that produced it. That
was handled by suffixing every image with its platform and committing
only the Linux ones, which left macOS and Windows runs silently writing
baselines nobody compares against.

Both `make e2e` and `make e2e-update` now run in the image CI uses, so
there is one renderer and one image per view. Repeat runs reuse a named
volume for node_modules rather than reinstalling each time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
fix(content-script): load the bundled Monaco face where it is not ins…
7090964 
…talled

`@font-face` is document scoped — Chrome ignores it inside a shadow
root — and `__MSG_@@extension_id__` is substituted only in CSS the
manifest declares, not in styles injected at runtime. The rule shipped
into the shadow root with the placeholder intact, so the bundled face
never loaded and the view fell back to sans-serif. macOS and any other
machine with Monaco installed matched on `local()` and hid both faults.

The face now goes into the document head with the URL resolved through
`chrome.runtime.getURL`, and `Monaco.woff` is web accessible so pages
being formatted can fetch it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): retry select all until the clipboard actually changes
d65531e 
The commands act on whatever the page shows at that instant, so a view
still settling copied nothing and left the assertion to fail on an empty
string. Seeding a sentinel first makes "nothing was copied" distinct
from a stale read, and the copy is retried until the clipboard moves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): compare screenshots exactly instead of within a tolerance
d85a9a6 
`maxDiffPixelRatio: 0.005` allowed ~4600 differing pixels on a 1280x720
frame, which is more than the whole text area of the query view. Both of
its screenshots kept matching through the font change that rewrote every
other image, so a regression there would have passed.

Rendering is pinned to one container, so there is nothing for the
tolerance to absorb. The two images it had been masking are regenerated
here; the suite is stable over repeated runs without it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): hide the caret before screenshotting the query view
3371f01 
Playwright hides the text caret by injecting CSS into the document,
which never reaches the input inside the closed shadow root. It blinked
between captures and left a 14 pixel diff — invisible under the old
tolerance, a flaky failure without it.

`ShadowElement.evaluate` runs a function against a pierced element, with
`text` now going through it too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): wait for a collapse to reach the page before reading it
e513611 
Collapsing sets a class the browser then has to lay out, and both the
rendered text and a copy of it read that layout, not the class. Asserting
straight after the click caught the tree still expanded, which showed up
as a copy containing the items that were supposed to be hidden.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): support dragging out a selection and copying just that
f0e10cf 
`clipboard.ts` becomes `selection.ts` now that it covers making a
selection as well as copying one: `dragSelect` sweeps the mouse between
two elements the way a reader would, and `copySelection` copies whatever
is selected rather than the whole document.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover expanding, per-item collapsing and partial copies
cc266df 
Collapsing was only checked one way and only on a whole property. These
add expanding a node back, collapsing a single array item while its
siblings stay open, and dragging across one nested object to confirm the
copy holds that object alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover linked strings and multi-result expressions
478fbbd 
Strings that look like a URL or an address render as links and light up
while the modifier is held, and a jq expression yielding more than one
value renders each result rather than only the first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover the download mode and the recorded query history
81e5a27 
Both settings only mean anything once they reach another part of the
extension: the download mode changes the toolbar button on a formatted
page, and a query run there has to travel through the background worker
into the history the options page lists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover navigating the manual
01f3f8d 
The sidebar was only checked for existing and for scrolling to one
section. It is built from the headings the markdown produces, tracks
where the reader is, and is how the query panel's help icon is meant to
land somewhere useful — none of which was covered.

Sidebar entries are now checked against the headings actually rendered
rather than a fixed list, so adding a section to the manual does not
quietly go unlisted. The mark on the current section is asserted after
navigating and after scrolling, and the help icon has to open the manual
itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): wait for an element to stop moving before clicking it
2837962 
Playwright settles a locator's box before clicking, and none of that
applies to an element resolved by hand over CDP: the box was measured
once and the mouse sent to wherever it had been. Layout keeps moving
after the tree first renders — the bundled font finishing loading
reflows all of it — so the click could land on nothing, leaving a
collapse that never happened.

Seen twice on CI and never reproduced locally, including at ten repeats
across four workers, so this closes the gap in the helper rather than a
diagnosed race.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): cover opening a linked url in a new tab
8e7fcd0 
This was left uncovered because an earlier attempt never opened
anything: `page.mouse.click` ignores a `modifiers` option, which only
exists on locator clicks, so the modifier was never actually held and
the handler never ran. Holding it through the keyboard works.

The tab is given something to land on by serving the address from the
test, so nothing leaves the machine. A plain click is covered alongside
it, since "opens on modifier-click" means little without "does not open
otherwise".

The address form stays uncovered: it hands a `mailto:` to whatever the
machine has registered, which is not something to invoke from a test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
style(e2e): drop the explanatory comments
e774c29 
The reasoning they carried is in the commit messages and in the two
guides; the code reads on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
test(e2e): screenshot every view in both themes
b0016c0 
Only the formatted view was captured light, and it took a duplicated
test to do it. The suite now runs as two projects, and a test tagged
`@screenshot` runs in both, so each view is committed once per theme and
the project name supplies the suffix. Everything else stays on one
theme rather than running twice for no gain.

The light theme was carrying six uncovered views, including the options
page and the manual.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b @claude
fix(content-script): declare the Monaco face in manifest CSS
3a46d16 
Back to a stylesheet, as asked, but not the one it came from:
`content-script.scss` is injected into the closed shadow root, where
Chrome ignores `@font-face` entirely. Declaring the file through
`content_scripts.css` puts it in the document and is also the one place
Chrome substitutes `__MSG_@@extension_id__`, so the original rule works
unchanged and the runtime-built stylesheet is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@evg4b
evg4b force-pushed the test/e2e-playwright branch from 1258734 to 3a46d16 Compare September 17, 2026 18:20
@sonarqubecloud

sonarqubecloud Bot commented Sep 17, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@evg4b