feat(website): add 'What Fixture Replay Can't Catch' blog post - #854
Merged
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Body across all sections, plus two validation fixes: - Rewrote the transcript-settling sentence, which had 6 eight-grams of overlap with the subgraphs docs guide (same author, two days apart). - Restored Let's transitions and trimmed demonstrative tics. Dated 2026-08-29 (plan said 08-28; the date rolled and siblings date to their merge day). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Remove a self-contradiction the thesis forbids: the conclusion claimed 'both have a plan' while the body says the drift check cannot see meaning drift. Only one of the two has a plan; say so. - Answer the obvious objection (replay the recorded chunk boundaries): faithful boundaries make the fence failure deterministic, not absent. - Defend the trade by displacement, not documentation — progressive rendering moved to the unit-level variance tables, which do test it. - Name the drift check as the second instance of the law, earning 'every' in the opening line. - Attribute affordability to replacing the model, not to the seam position. - Link the subgraphs post as the source of the worked example; 1 link -> 4. - Register: trim idiom, intensifiers, and doubled phrasing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Spec-compliance review found the post overstated its own thesis. All verified in source: - chunkSize 4096 is a DEFAULT, not a law. A second harness serves the two example apps, and its comment says targeted streaming regressions opt into smaller per-fixture chunks — those fixtures set 3/4/6/23/36 with 25-750ms latency, and there are e2e tests sampling the mid-stream DOM. 'We deleted time' was true of the cockpit tier only. Reframed around which tier opted back in, which is a better question anyway. - 'spawns a real langgraph dev subprocess' is false for 8 of 34 apps; the ag-ui setup spawns uvicorn. The seam generalizes, the server doesn't. - The drift check does not run at all: it shells out to , which is not a declared option under strict parseArgs, so it exits before comparing anything. The post said it re-records and compares. Now says what is true. - Quote the full source comment rather than half of it; the omitted lines were what the following paragraphs presented as original analysis. - Drop the 60ms/56-samples figures — real, but from an ad-hoc session and unverifiable from the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The check went from 'has never run' to merged and proven (#846) between drafts. §5 now carries the real design: a @drift-tagged contract-only subset of the e2e suite runs weekly against the live provider through the record proxy — the assertions are the drift check, so meaning drift is caught by construction. Includes the first run's finding: the differ reported tool-call drift that was actually the recorder failing to parse tool-call deltas — the instrument's own blind spot, now its own category. Conclusion updated to match: both gaps covered by the same move, widening is tagging specs, not new design. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Keystone stands alone as its own paragraph; third restatement cut. - Anecdote no longer announces itself; actor fixed (the recorder warns, not the check); adds that it has run clean against the live model since. - Conclusion scoped honestly: widening drift = tagging specs plus porting record mode to the shared harness; widening streams = per-fixture chunk opt-in. Both reviews independently flagged the original overclaim. - Tagging rule examples now match what the specs actually assert (prompt-derived patterns, not shape-only). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Soften first-person discovery narration to impersonal constructions.
The events are real, but they were my discoveries narrated under
Brian's byline; the claims now belong to the work, not a narrator.
- Restore the flagged opinion on the streaming trade ('I think that's
the right trade') — the post's strongest judgment was reading as fact,
and siblings carry 1-2 flags.
- Fix bold in body prose (italics-only convention).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
blove
enabled auto-merge (squash)
August 29, 2026 21:52
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Post #12, the final post of the GSC-driven sequence — after #836, #837, #839.
Design: spec · plan
Thesis
A deterministic test harness buys determinism by deleting a dimension. Ours deletes time — deliberately, with the reason written in the source — and that makes one bug class structurally invisible to a green suite. Not a "how to mock your LLM" post; that one exists everywhere.
This is the only post in the sequence with no search evidence behind it — a bet on being differentiated rather than on capturing existing demand.
What the reviews changed
Two full review rounds. The first found the post overstating its own thesis, which is the one failure a post about overstatement can't survive:
chunkSize: 4096is a default, not a law. A second harness serves the two example apps and lets targeted streaming regressions opt into per-fixture chunks (3–36, 25–750ms latency), with e2e tests sampling the mid-stream DOM. "We deleted time" was true of the cockpit tier only. Reframed around which tier opted back in — a better question, and the tier that didn't is where the worked example's bug lived.langgraph dev" is false for 8 of 34 apps (the AG-UI ones spawn uvicorn). The seam generalizes; the server doesn't.Both reviewers then independently flagged the conclusion's "widening either is tagging more specs" — each catching a different half of why it was wrong (record mode isn't in the shared harness; the stream check was never a tagging mechanism). Now scoped honestly, keeping the confident close.
Drift detection shipped mid-draft
The post originally reported that our drift check had never run — the broken
--outflag. That got fixed properly in #846, so §5 now describes the shipped design: a@drift-tagged contract-only subset runs weekly against the live provider through the record proxy. The assertions are the drift check — meaning drift caught by construction, no thresholds invented. Proven by three green live runs (local, branch CI, andmain).Verification
main(5 documented pre-existing + 2 files on unresolvedposthog-node)🤖 Generated with Claude Code