test(e2e): declarative Scenario layer — compose model, steps, expected observables - #14047
Open
ndeloof wants to merge 5 commits into
Open
test(e2e): declarative Scenario layer — compose model, steps, expected observables#14047ndeloof wants to merge 5 commits into
ndeloof wants to merge 5 commits into
Conversation
…trator Introduce a small declarative layer over the e2e framework so a test reads as its intent: an inline compose.yaml, a sequence of steps (command + expected observables), and nothing else. The framework owns project naming (derived from the test name), cleanup, and failure reporting: on the first unmet expectation the test fails with a step transcript, the failing command output, the project's container state, the engine events since the scenario started and the container logs. Checks observe real state (container IDs across steps, labels, image manifest platform) instead of grepping the progress writer output, which stays available as an explicit last-resort check. The e2e tests introduced by #14041 are converted as a demonstrator; their dedicated fixture files are inlined in the scenarios and removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Add Eventually(check, timeout) to the scenario layer: it retries a state-based check, re-observing the project containers between attempts. ServiceState(service, state) joins the check library, backed by a State field in the per-step snapshots. TestRestart is converted as the demonstrator: it replaces a WaitForCmdResult polling `ps --format json` for a JSON fragment and a bare time.Sleep with two declarative expectations, and now also locks the real invariant — restart must reuse the same container — via NotRecreated, which the lock-file trick of the original fixture was proving only indirectly. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…alive On failure, a scenario now writes the untruncated diagnostic material to a stable per-project directory (compose.yaml, each step's full command and output, project containers, engine events, full container logs, per-step state snapshots), announced at the top of the failure report. The inline report stays truncated for readability; the artifacts let a failure be diagnosed without re-running the scenario. E2E_KEEP_FAILED=1 skips the teardown of a failed scenario so the live project state remains inspectable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
… the contract Checks are the shared vocabulary between scenarios: move them to a dedicated checks.go so the whole vocabulary is read — and reviewed — as one file, independently of the tests using it. SCENARIO.md codifies the rules that so far only lived in the PR description: one intent = one invariant, step names as behavior sentences, inline compose model, state-based checks first with OutputContains as an explicit last resort, new checks generic and named after their observable — plus how to exploit the failure artifacts and E2E_KEEP_FAILED. AGENTS.md points new e2e work at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A scenario could only declare a lone compose.yaml; anything needing a Dockerfile, an env file or a config file fell back to fixture directories, breaking the "self-contained in the test source" rule. Files() takes the project files as a txtar archive (golang.org/x/tools/txtar, already in the module graph): each file introduced by a `-- name --` line, extracted into the project directory. txtar is the format Go's own cmd/go tests are written in — diff-friendly, and one every human and coding agent already knows by heart, which is the point: e2e scenarios are the main reading surface between the two. TestUpBuildUnchangedContext demonstrates the need — compose.yaml + Dockerfile + a copied file — and locks a real invariant: an unchanged `up --build` hits the build cache and must not recreate the service. Project() is exposed so a scenario can Defer removal of the image it built. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof
force-pushed
the
e2e-scenario-dsl
branch
from
August 16, 2026 15:11
c1028c2 to
f3597c0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With coding agents writing most of the production code, e2e tests become the document humans actually read: they should state an intent, a compose model, commands and expected observables — and nothing else.
This PR prototypes a small declarative
Scenariolayer over the existing e2e framework:compose.yaml, steps as(command → expected observables), intent as a one-linerOutputContainsstays as an explicit last resortEventually(check, timeout)replaces hand-rolled polling loopsDemonstrators, converted in place:
TestRestart, whoseWaitForCmdResultJSON-grep polling and baretime.Sleepbecome two declarative expectations — and now locks the actual invariant (restartreuses the same container) viaNotRecreatedExample failure report:
Draft to discuss the approach; the two styles coexist, existing tests convert opportunistically.