Move toward modular layout + automatic setup, localize snapshots

[mjambon]

Tentative plan (as of Aug 22, 2026):

1. Make it easier to define named snapshots in the folder containing the current source file (e.g. xxx/test_foo.ml will define tests that keep snapshots in xxx/snapshots/).
2. Provide a testo init command that sets things up. Must work at least for a Dune project initially. No need to support other subcommands just yet.
3. (a) Support a testo.conf file that contains settings such as how to determine the project root (e.g. "here" or "dune"), and commands to rebuild the test program or to locate the test executable. (b) Add support for subcommands that require testo.conf: testo run, testo review, testo approve.

---

Messy, unsatisfying, and outdated "summary" of notes, written by Claude:

Move toward modular layout + automatic setup, localize snapshots

Goal

Two coupled changes:

1. Automatic setup. A testo command inspects the project tree and generates/updates the dune wiring for tests, so adding tests to a library doesn't mean hand-writing dune files and hand-maintaining a list of test libraries somewhere central.
2. Localized snapshots. Snapshots live next to the test definitions that own them, named after those tests, so they're easy to find and trivial to relocate when a library is detached into its own project.

These are big, intentionally-incompatible changes. There are few users and they're the adventurous kind, so single cutover, no back-compat layer.

Architecture

• Discovery by convention, aggregation by codegen, tests stay values. Each test group exposes val tests : env -> Testo.t list. The testo command walks the tree, finds conforming modules/folders by name pattern, and generates an aggregator that names each one explicitly. Codegen (not runtime registration) because partial linking / -linkall makes register-on-link unreliable in OCaml, and because a registry discards the structure we want for bulk reclassify/rewrap.
• Per-library tests/ folders. Each depends on its parent library (+ declared extras). These are local and travel with the library on detach.
• One root aggregator is the only project-global artifact. It holds the (libraries …) list and the calls into each group. It's the only thing that breaks when a library is detached, and testo init regenerates it in the new repo.

Invariants (the load-bearing decisions — don't lose these)

• The user-named-path snapshot feature already is the localization mechanism. This is a default flip + path derivation, NOT new snapshot storage. File content format is unchanged; migration (if any) is pure relocation.
• Snapshot path derives from discovery data, not the test value. The path is <group.source>/snapshots/<name>, where source is the folder the group was discovered in — the same field used for prefixing/tagging. No ppx, no __FILE__ capture. The test supplies a local name; the group supplies the root.
• Drop the hash. It only existed to make names collision-free in one global flat folder. Folder-local names need only be unique within a few dozen human-authored tests → use the sanitized test name, handle rare collisions locally. If a test has multiple outputs, the unit is a per-test subfolder snapshots/<test>/ so it relocates as one.
• Catalog construction is effect-free. tests : env -> Testo.t list builds descriptions only — it must NOT glob or read snapshot files. All snapshot IO defers into the Testo.t run thunks. So testo list / name-filtering / dry enumeration never touch the filesystem.
• Snapshot IO resolves against the source root (from the config), never the runtime sandbox. Addressing snapshots by the process's cwd writes promotions into throwaway _build/ copies under dune exec. testo resolves the source root once — from the real invocation cwd, before building — and pins all snapshot read/write to it, so approve always updates versioned files. The suite is never run via dune exec; testo run builds then execs from the source root.
• Generated files are inert data, carrying zero policy. The aggregator emits handles (source, lib name, tests reference), not List.map (prefix …). Prefix/tag/filter live in the Testo runtime over the group list. This keeps the regeneration trigger narrow: generated files change ONLY when the set of test libraries changes, never on a Testo version bump.
• env enters at exactly one seam: leaf test construction. Discovery, aggregation, path derivation, prefixing, filtering, listing are all env-free. env is fixed by the runtime, not user-extensible, so nothing project-specific is smuggled through it.
• Users invoke the installed testo CLI; the compiled suite exe is a hidden artifact under _build/, never launched by path. No root-level launcher file → no test/-vs-./test collision, and no reliance on test (a shell builtin).

Invocation: testo run, not ./test

The compiled suite exe lives under _build/ and should never be invoked directly. The old ./test launcher (symlink/script at the root) collides with a test/ folder — exactly the folder this layout puts at the root — and test is a shell builtin, so the launcher artifact is a hazard twice over. Eliminate it rather than work around it.

The installed testo CLI becomes the single entry point (mental model: testo is to the suite what dune/cargo is to a build — one install, operates on whatever project you're in):

• testo run [filters…], testo status, testo approve … — testo walks up to the project root, builds the runner target, then execves the compiled exe so it becomes the test process: clean exit codes, signals, TTY passthrough, dune out of the process tree during the run. Works from any subdirectory.
• The exe's name and location stop being part of the UX, which is what makes the buried-in-_build problem disappear — nothing addresses it by path anymore.
• Promote correctness (third independent reason for the launcher). dune exec chdirs into a _build/ copy, so approve writes snapshots into the throwaway sandbox, not the source tree. Because testo run builds-then-execs from the resolved source root instead, approve updates the versioned files. The same launcher decision is now justified three ways: the test/ collision, clean exit codes via execve, and source-tree promotion.

Two real decisions this forces:

• Subcommand namespacing. The suite exe already exposes Testo's own CLI (run/status/approve). testo adds tree-level meta-commands (init/gen). Decide the rule: a small reserved set handled by testo itself, everything else forwarded verbatim to the located exe. Getting this wrong shadows a suite subcommand or makes adding one require a testo release.
• Build-system coupling (touches the founding constraint). Shelling to dune to build re-couples testo to a build system. Keep the independence claim precise: it meant no build-system plugin to hold test state, NOT never invoke the build system. Confine the coupling to one recorded runner target + one configurable build command (dune by default); after the build, exec the artifact directly so dune is a default, not a load-bearing assumption.

Bonus: testo run can fold the regen pass in front of the build (ensure-wiring → build → exec), absorbing the staleness-on-add cost — adding a test lib and running it becomes one command.

Project root & config (testo-project / testo-workspace)

Primary motivation: the config is the seam for accommodating specific build systems — dune above all. A build system isn't just "how to compile the runner"; it imposes a sandbox with quirks Testo has to work with. The config lets Testo carry per-build-system handling as data instead of inferring it and getting it wrong. The case that forces the issue:

dune exec runs the suite from a copy of the source tree under _build/, chdir'd into the sandbox. approve (≈ dune's promote) then writes updated reference snapshots into that build copy, which is discarded on the next build — so approve silently fails to update the source tree. Fix: the config marks the real source root; testo run resolves it from the user's actual location (in the source tree, not _build), builds via dune, then execves the artifact with snapshot IO pinned to the source root — so approve writes back to the versioned files. Never run the suite via dune exec. Trap to avoid: dune copies testo-project into _build too, so the runner must NOT re-derive the root by walking up from its own cwd — testo resolves the root once, before building, and passes it down explicitly.

So the config carries build-system-specific options (a dune profile to start) on top of root-finding — that's the reason it exists, not an afterthought. With that set:

Mirror dune's two-marker root-finding (independent implementation, but familiar to dune users — testo's audience):

• testo-project: topmost-wins, "part of a project." The settings-bearing marker — runner target + build command live here (this absorbs the earlier testo.conf; don't keep a separate conf as a third file). Always present.
• testo-workspace: explicit root-pin that halts the ascent. Present only when you need to pin a sub-root (vendoring) or configure workspace-wide context.

It's a root artifact, generated by testo init. A not-yet-detached library does NOT carry a testo-project; it gets one on detach. So the common tree has exactly one testo-project and no testo-workspace.

testo-project writes/reads the runner target → single source of truth for both testo gen and testo run; resolves "where does invocation find the target."

Build context (separable from the marker question above — about which build output to run, not root-finding): default to dune's default context → _build/default/<runner>. Covers the single-switch case (almost everyone) with zero config. Multiple contexts (a dune-workspace with several switches) → still default to default, expose testo run --context NAME as a passthrough of dune's --context, let testo-project record an optional default context name. Don't reimplement dune's context resolution or parse dune-workspace to enumerate — pass the name through, let dune's error surface for invalid contexts. Always the host artifact, never a cross-compiled -x target. Only un-guessable case: a workspace with contexts but no default → error, require --context. Run = dune build the (optionally context-qualified) target, then execve it.

Open: does testo-workspace carry content, or is it pure root-pinning ceremony? In dune each marker earns its place via distinct content (project metadata vs. build contexts); root-finding rides on top. Copying both files only to use one as a bare "stop here" imports dune's structure without dune's reason. BUT a test framework has a real workspace-level concern dune-workspace already serves: running the suite across multiple switches / OCaml versions / profiles (a context matrix). If that's on the roadmap, testo-workspace is justified on its own terms — two markers, clean. If not, collapse to one marker with an explicit (root) stanza for the pin (one concept; the pin becomes a field, not a filename — less greppable).

Note: testo-workspace pins a root and configures contexts; it does NOT aggregate runners across projects. Multiple testo-projects under one workspace = the vendoring case, where each was meant to stay separate. The "one runner per project" / minimize-executables constraint is untouched — the workspace marker separates things already separate, never links across them.

Settings are project-global → they live ONLY at the project marker. A standalone marker file preserves build-system independence but admits testo-root ≠ dune-root (footgun: runner target is dune-relative, mismatch breaks the build) — so testo should warn when its root doesn't match the nearest dune root. Diagnostic, not dependency.

dune notes

• data_only_dirs snapshots so include_subdirs doesn't try to compile reference files. The generator emits this automatically.
• Generator owns its dune files; humans don't hand-edit them. Extra deps/flags go in a sidecar (e.g. tests/testo.sexp) the generator reads, so regeneration is lossless.
• Consider (include dune.inc) + (mode promote) so dune build @runtest --auto-promote keeps wiring fresh; committed files give a one-pass steady state. Known cost: adding/removing a test library needs a generation pass before dune can link (dune resolves deps up front). Editing tests inside an existing lib needs no regen.

Open question

Within-folder granularity:

• Per-folder entry module — human writes a tests.ml that concats the folder's files; generator sees one handle per lib. Dumber generator, small manual concat per folder, signature checkable.
• Per-module discovery — generator finds every *_tests module; no manual concat, but wider name-pattern surface and the conformance check (module _ : Testo.TESTS = M) has to be emitted per module.

Pick based on how the existing big projects actually lay tests out.

Also open: testo's subcommand namespacing — the reserved meta-set (init/gen/…) vs. what's forwarded verbatim to the suite exe, and whether testo run auto-regenerates wiring or keeps gen explicit.

Also open: whether testo-workspace carries real content (a switch/version context matrix) or is pure root-pinning — i.e. two markers vs. one testo-project with an explicit (root) stanza.