From e0d471ee05e162dddefc880a6ebc0e2f2ae7e919 Mon Sep 17 00:00:00 2001 From: Brian Love Date: Sat, 29 Aug 2026 18:11:36 -0700 Subject: [PATCH] docs(website): add an on-ramp to the fixture-replay post MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The post opened on its conclusion — 'every harness deletes a dimension' — before establishing any premise. It never stated the problem it solves (models don't return the same thing twice), never defined 'fixture' or 'replay' despite both being in the title, and 'ours deletes time' was unparseable until section three. Adds ~150 words up front: the nondeterminism problem, what record-and- replay is and buys, then the thesis, which now has a referent. Also defines what a fixture entry is before describing its match block. Co-Authored-By: Claude Opus 5 --- ...2026-08-29-what-fixture-replay-cant-catch.mdx | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/apps/website/content/blog/2026-08-29-what-fixture-replay-cant-catch.mdx b/apps/website/content/blog/2026-08-29-what-fixture-replay-cant-catch.mdx index 52cdd2b96..5cfc11a74 100644 --- a/apps/website/content/blog/2026-08-29-what-fixture-replay-cant-catch.mdx +++ b/apps/website/content/blog/2026-08-29-what-fixture-replay-cant-catch.mdx @@ -8,8 +8,16 @@ featured: false draft: false --- -Every deterministic test harness buys its determinism by deleting a dimension. -Ours deletes time, deliberately, and the reason is written in the source. +Agents are hard to test end to end for one boring reason: the model doesn't return the same thing twice. +Ask it the same question in two runs and you get two different sentences, two different orderings, sometimes a tool call and sometimes not. Write an assertion against that and you've written a coin flip. + +The usual fix is to stop calling the model. +You capture its responses once, save them to disk as _fixtures_, and _replay_ them on every run — same request in, same bytes back, forever. The tests go deterministic, CI stops spending money on tokens, and the agent under test never knows it's talking to a recording. + +We test our whole demo fleet that way. This post is about the bill, because replay isn't free and the charge doesn't show up where you'd look for it. + +Here's the shape of it. Every deterministic harness buys its determinism by deleting a dimension of the real thing. +Ours deletes _time_ — deliberately, with the reason written in the source — and one specific class of bug vanishes along with it. So the interesting question about a harness isn't whether it's green. It's which dimension you deleted, because that's the list of bugs it can't report. @@ -49,7 +57,9 @@ This is the outer tier. For in-process fakes at the unit level, the [testing gui The shape of the request — and the order you list the entries decides which one wins. -Each entry carries a `match` block. The obvious discriminator is the user message, but there are richer ones: a parent LLM's first call and its continuation after a tool round carry the same user message. Something has to tell them apart. +A fixture file is a list of entries, and each one is a pair: a `match` block describing which request it answers, and the response to hand back when a request fits. + +The obvious discriminator is the user message, but there are richer ones: a parent LLM's first call and its continuation after a tool round carry the same user message. Something has to tell them apart. That something is `hasToolResult`, and matching is first-match-wins: