test: Split unit, integration, and e2e tiers with just recipes - #55
Merged
Merged
Conversation
Previously, the justfile only had benchmark recipes and running the suite, the e2e subset, or a real measurement meant remembering the right `go test`/`go run` incantation. Now, `just test`, `just test-e2e`, `just measure-cloudflare`, and `just measure-apple` cover those, and the bare `just` lists the recipes. `test-e2e` derives its `-run` regex from the `Test*` functions in the `e2e_test.go` files at run time rather than introducing a build tag, so `go test ./...` and CI keep running the e2e tests unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B4xBvbUJ5Ci1b99X5x8LJL
Previously, `go test ./...` ran every tier in one pass and `-short` skipped only the soak, so a unit-level iteration cost about 150 s. Now the two `e2e_test.go` files carry `//go:build e2e`, and every test that starts a listener or a process calls `skipIfShort`, so `go test -short ./...` finishes in about a second per package. "Unit" here means no I/O with anything external to the test: an in-process nqserver over loopback, a spawned `nq` binary, and `nqserver` bound to a port all count as external. The guard lives in the shared server-starting helpers where one exists and at the top of the test otherwise. The helpers the e2e files shared with other tests move to untagged `helpers_test.go` files so the plain build still compiles. CI, the release workflow, vet, and golangci-lint pass `-tags e2e` so the tagged files stay tested and linted; without that they would silently drop out of both. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B4xBvbUJ5Ci1b99X5x8LJL
Previously, `test-e2e` grepped test names out of two hard-coded file names and `test` ran the e2e tests too. Now `test` runs unit and integration tests only, `test -short` unit only, and `test-e2e` discovers every file tagged `//go:build e2e` and runs just the tests it declares, under the tag. The README testing section documents the three modes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B4xBvbUJ5Ci1b99X5x8LJL
korya
changed the title
korya
marked this pull request as ready for review
September 11, 2026 16:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A developer iterating on a unit change waited about 150 seconds for
go test ./..., because every tier ran in one pass and-shortskipped only the soak. There was also no single command for the suite, the end-to-end subset, or a real measurement, and a barejustprinted an error instead of a menu.What & how
Split the tests into three tiers selected by
-shortand a build tag, and addjustrecipes for each plus real measurements.just test -shortjust testjust test-e2e//go:build e2ego test -tags e2e ./...-racejust measure-cloudflare,just measure-appleDefinition of unit. No I/O with anything external to the test. An in-process nqserver over loopback, a spawned
nqbinary, andnqserverbound to a port all count as external. Every site that starts one callsskipIfShort: inside the shared server-starting helper where one exists (five of them), at the top of the test otherwise. The soak already skipped under-short.Why a build tag needs more than two lines. The two
e2e_test.gofiles defined helpers that other tests use, so those helpers move to untaggedhelpers_test.gofiles or the plain build breaks. And a tag hides files from every tool that is not told about it, so CI, the release workflow, vet, and golangci-lint now pass-tags e2e. The lint job already carried the same warning forwindows-tagged files.Why
test-e2estill filters by name.-tags e2eadds files to a package, it does not select them, sogo test -tags e2ewould run every tier. The recipe discovers files carrying the tag and runs only the tests they declare. Moving the e2e tests to their own package would be cleaner, but both files depend on package internals (package netquality, and the CLI's unexported entry point).Out of scope: the root
e2e_test.goruns full measurements against an in-process server, much likerun_test.go, so its "e2e" label is loose. It keeps the tag as-is to limit churn; reclassifying it is a separate decision.No public responsiveness servers exist beyond Apple and Cloudflare. Self-hosted servers remain reachable through
nq --well-knownor--config-url.Other changes
just --fmt, and a barejustlists the recipes.🤖 Generated with Claude Code
https://claude.ai/code/session_01B4xBvbUJ5Ci1b99X5x8LJL