Fix the flaky demo e2e step that turned main red - #29
Merged
Merged
Conversation
main went red on the merge of #28: the demo step reported "demo is missing 'Overall alignment'" while the page dump printed directly beneath it contained "Overall alignment · 63% · from 22 shared answers". The assertion was right about what it wanted and wrong about when to look. The step waited for the creature names, which render as soon as the model resolves, and then snapshotted document.body. Every compare panel is a lazily loaded component that arrives after that, so the snapshot caught a page still filling in. It has been a race since the demo step was written and only lost it on a slower runner — three CI runs and every local run happened to win. It now waits for the panels it actually asserts on before reading the body. Same lesson as the networkidle fix a commit ago: wait for the thing you are asserting, not for a proxy that usually arrives first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UaDanzXTm6kVSgbNh1eNrs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mainwent red on the merge of #28. This fixes it. The app is fine; the test was wrong.What happened
…and the page dump printed directly beneath that failure contains the missing text:
The assertion was right about what it wanted and wrong about when to look.
Why
The step waited for the creature names (
brave-azure-otter), which render as soon as the demo model resolves, and then snapshotteddocument.body. But every compare panel is a lazily loaded component —provideComparePanelresolves them asynchronously — so they arrive after the names. The snapshot caught a page still filling in.It's been a race since the demo step was written in Wave 2, and it only lost on a slower runner: three CI runs and every local run happened to win it.
The fix
Wait for the panels the step actually asserts on before reading the body. Same lesson as the
networkidlefix one commit earlier — wait for the thing you're asserting, not for a proxy that usually arrives first.Verification
format:checkande2egreen locally against a fresh production build. Nothing bute2e/run-e2e.mjschanges, so no app behaviour is touched.Generated by Claude Code