The fake server in #21 is what almost every test on this board runs against, and
its fixtures are responses recorded from real servers. Recorded once. From that
moment the suite is measuring the core against a snapshot, and a snapshot is
green forever.
The way this fails is quiet. A server line changes a field's type, stops sending
a property, or returns a list where it returned an object. The core is then wrong
against the real server and right against the recording, so the suite passes and
the first report comes from an operator. Nothing about that failure looks like a
test problem when it arrives, which is why it costs so much to find.
This takes nothing away from #20. The comparison needs a real server, so it lives
in the harness in #22 and the headless suite never depends on it.
What this produces
A run in that harness which fetches every endpoint in the record from #10 from a
real server and compares the shape of what came back against the stored fixture.
Shape rather than content, because values differ per server and per library while
the field that disappeared does not.
A report naming each difference one at a time, so that a fixture which genuinely
has to move is moved deliberately. Re-recording the whole set is how a hostile
fixture gets replaced by a healthy one without anybody noticing that the case it
proved is gone.
The server version each fixture was recorded against, stored beside the fixture,
so a difference reads as a version change rather than as a mystery.
A refusal to run rather than a silent skip when no server is reachable, which is
the rule #22 already sets for everything in that harness.
Done when
The comparison exists in the harness, a run against a server on the version a
fixture was recorded from reports no differences, and a deliberately edited
fixture is reported by name and not merely by count. Every fixture in the tree
carries the server version it came from. Blocked on #21 and #22.
The fake server in #21 is what almost every test on this board runs against, and
its fixtures are responses recorded from real servers. Recorded once. From that
moment the suite is measuring the core against a snapshot, and a snapshot is
green forever.
The way this fails is quiet. A server line changes a field's type, stops sending
a property, or returns a list where it returned an object. The core is then wrong
against the real server and right against the recording, so the suite passes and
the first report comes from an operator. Nothing about that failure looks like a
test problem when it arrives, which is why it costs so much to find.
This takes nothing away from #20. The comparison needs a real server, so it lives
in the harness in #22 and the headless suite never depends on it.
What this produces
A run in that harness which fetches every endpoint in the record from #10 from a
real server and compares the shape of what came back against the stored fixture.
Shape rather than content, because values differ per server and per library while
the field that disappeared does not.
A report naming each difference one at a time, so that a fixture which genuinely
has to move is moved deliberately. Re-recording the whole set is how a hostile
fixture gets replaced by a healthy one without anybody noticing that the case it
proved is gone.
The server version each fixture was recorded against, stored beside the fixture,
so a difference reads as a version change rather than as a mystery.
A refusal to run rather than a silent skip when no server is reachable, which is
the rule #22 already sets for everything in that harness.
Done when
The comparison exists in the harness, a run against a server on the version a
fixture was recorded from reports no differences, and a deliberately edited
fixture is reported by name and not merely by count. Every fixture in the tree
carries the server version it came from. Blocked on #21 and #22.