feat(testing): CSD flows run on the five-platform matrix; state: now asserts something

.github/workflows/build.yml

@@ -111,7 +111,7 @@ jobs:
           python3 -m pip install --quiet pytest pyyaml
           python3 -m pytest testing/test_driver_rules.py testing/test_gate_vendoring.py \
             testing/test_bringup.py testing/test_five_platform_workflow.py \
-            testing/test_csd_state_tags.py -q
+            testing/test_flows.py testing/test_csd_state_tags.py -q
 
       - name: A published version must offer a universal wheel
         # Only for versions already on the index — a PR's VERSION is normally

.github/workflows/five-platform-live-qa.yml

@@ -141,6 +141,12 @@ jobs:
         with:
           python-version: '3.10'
 
+      # THE CSD FLOWS NEED PyYAML (testing/gate/flow_spec.py reads the flow and
+      # its CSD's typed blocks). Everything else the legs run is stdlib; this is
+      # the one install, and it goes into the interpreter every leg below uses.
+      - name: The flow runner's one dependency
+        run: python3 -m pip install --quiet pyyaml
+
       - name: The node this client will be a client of
         id: node
         env:
@@ -224,12 +230,18 @@ jobs:
       - name: Start the node
         uses: ./.github/actions/ciris-node
 
+      # THE SMOKE WALK, THEN THE CSD FLOWS, IN ONE PROCESS AGAINST ONE APP.
+      # `--flows` runs every testing/flows/*.yaml after the walk passes, in the
+      # app the walk just proved, signing in once (session_fixture). A flow
+      # refused by its `client:` floor is reported and does not redden the leg;
+      # a flow that fails, or never reaches its first screen, does. See
+      # testing/flows/README.md. Every leg below passes the same flag.
       - name: Linux desktop
         run: |
           python3 -m testing.gate.run_platform --platform desktop --xvfb \
             --jar "${{ steps.art.outputs.jar }}" \
             --node-version "${{ steps.node.outputs.version }}" \
-            --shots shots --report reports/linux.json
+            --shots shots --report reports/linux.json --flows testing/flows
 
       # WITHOUT THIS THE EMULATOR RUNS IN SOFTWARE AND DIES.
       #
@@ -289,7 +301,7 @@ jobs:
           #
           # after the emulator had booted and the whole leg had been paid for.
           # It reads worse on one line and it is the form that runs.
-          script: python3 -m testing.gate.run_platform --platform android --apk "${{ steps.art.outputs.apk }}" --node-version "${{ steps.node.outputs.version }}" --shots shots --report reports/android.json; rc=$?; adb logcat -d > logcat.txt 2>&1; exit $rc
+          script: python3 -m testing.gate.run_platform --platform android --apk "${{ steps.art.outputs.apk }}" --node-version "${{ steps.node.outputs.version }}" --shots shots --report reports/android.json --flows testing/flows; rc=$?; adb logcat -d > logcat.txt 2>&1; exit $rc
 
       # ALWAYS. A failure you cannot diagnose from the artifact costs a re-run
       # to learn what this run already knew.
@@ -325,6 +337,17 @@ jobs:
           distribution: temurin
           java-version: '17'
 
+      # A PROVISIONED INTERPRETER for the macOS desktop leg, so the flow runner's
+      # PyYAML goes into a Python this job owns rather than the image's
+      # externally-managed one. The iOS leg switches to 3.10 further down and
+      # installs it again there.
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: The flow runner's one dependency
+        run: python3 -m pip install --quiet pyyaml
+
       - name: The node this client will be a client of
         id: node
         env:
@@ -350,7 +373,7 @@ jobs:
           python3 -m testing.gate.run_platform --platform desktop \
             --jar "$(python3 -m testing.gate.candidate_artifacts --kind desktop | tail -1)" \
             --node-version "${{ steps.node.outputs.version }}" \
-            --shots shots --report reports/macos.json
+            --shots shots --report reports/macos.json --flows testing/flows
 
       # ── THE iOS BUNDLE, MATERIALIZED THE WAY THE AGENT'S GATE DOES IT ──────
       #
@@ -541,6 +564,9 @@ jobs:
       - name: iOS simulator
         run: |
           set -euo pipefail
+          # `python3` is the 3.10 set up above now, not the 3.12 the macOS leg
+          # installed PyYAML into — the flow runner needs it here too.
+          python3 -m pip install --quiet pyyaml
           # THE TASK NAME, AND THE DIRECTORY, BOTH WRONG — AND THE SECOND HID THE FIRST.
           #
           # `:shared:assembleDebugXCFramework` does not exist. The framework is
@@ -632,7 +658,7 @@ jobs:
           rc=0
           python3 -m testing.gate.run_platform --platform ios --app "$app" --udid "$UDID" \
             --node-version "${{ steps.node.outputs.version }}" \
-            --shots shots --report reports/ios.json || rc=$?
+            --shots shots --report reports/ios.json --flows testing/flows || rc=$?
 
           # THE APP'S OWN ACCOUNT, EITHER WAY. Run 35359571538 got the iOS app
           # to a real screen — "Engine Failed to Start: server did not become
@@ -732,10 +758,11 @@ jobs:
       - name: Windows desktop
         shell: bash
         run: |
+          python3 -m pip install --quiet pyyaml
           python3 -m testing.gate.run_platform --platform desktop \
             --jar "$(python3 -m testing.gate.candidate_artifacts --kind desktop | tail -1)" \
             --node-version "${{ steps.node.outputs.version }}" \
-            --shots shots --report reports/windows.json
+            --shots shots --report reports/windows.json --flows testing/flows
 
       - if: always()
         uses: actions/upload-artifact@v4

client/VENDORING.md

@@ -40,7 +40,7 @@ source is the pair a bisect wants:
 The tree's current recorded state — sha256-of-sha256s over every git-tracked
 file under `client/` except this one:
 
-**state digest:** `dece5915e6100c07639cb5cb597ee35cd9701d527ca0588969aa5e7ceba3b2fc`
+**state digest:** `1ab9a22b7e23dea8b1e2dc3aac8a8c6e3cc95c5ca7d6783a4d73bf40321f2bbd`
 
 `packaging/check_vendoring.py` asserts it on every push, and refuses any
 tracked file matching a §2 never-vendor class. **Any commit that touches

client/shared/src/androidMain/kotlin/ai/ciris/mobile/shared/testing/TestAutomationServer.android.kt

@@ -94,7 +94,9 @@ class AndroidTestAutomationServer(private val port: Int = 9091) {
                 // Input text to element
                 post("/input") {
                     val request = call.receive<InputRequest>()
-                    call.respond(TestAutomationHandler.handleInput(request))
+                    val resp = TestAutomationHandler.handleInput(request)
+                    // A failed input is not a 200 (desktop answers 404; iOS now does too).
+                    call.respond(if (resp.success) HttpStatusCode.OK else HttpStatusCode.NotFound, resp)
                 }
 
                 // Scroll the screen (recovery after an off-screen refusal)

client/shared/src/iosMain/kotlin/ai/ciris/mobile/shared/testing/TestAutomationServer.ios.kt

@@ -265,7 +265,11 @@ class IOSTestAutomationServer(private val port: Int = 9091) {
                 }
                 method == "POST" && path == "/input" -> {
                     val req = json.decodeFromString<InputRequest>(body)
-                    200 to json.encodeToString(TestAutomationHandler.handleInput(req))
+                    val resp = TestAutomationHandler.handleInput(req)
+                    // A failed input is not a 200: desktop answers 404, and a
+                    // harness that checks the status (the gate's driver did)
+                    // otherwise believes it typed into a field that took nothing.
+                    (if (resp.success) 200 else 404) to json.encodeToString(resp)
                 }
                 method == "POST" && path == "/wait" -> {
                     val req = json.decodeFromString<WaitRequest>(body)

testing/driver.py

@@ -138,9 +138,15 @@ def _call(self, method: str, route: str, body: dict | None = None) -> Any:
         if not raw.strip():
             return None
         try:
-            return json.loads(raw)
+            parsed = json.loads(raw)
         except json.JSONDecodeError:
             return raw
+        # A body that says it failed has failed, whatever the HTTP status: the
+        # iOS server answered a failed /input with 200 until 0.5.225, and the
+        # walk then "typed" into fields that took nothing.
+        if isinstance(parsed, dict) and parsed.get("success") is False:
+            raise DriverError(f"{method} {route} -> {parsed.get('error') or parsed}")
+        return parsed
 
     # ---- reads --------------------------------------------------------
 

testing/flows/README.md

@@ -0,0 +1,142 @@
+# CSD flows
+
+A CSD's §4 says what its surface must do. A flow here is that §4 made
+executable, and the five-platform gate (`five-platform-live-qa.yml`) runs every
+flow in this directory on every leg — Linux, macOS and Windows desktops, the
+Android emulator, the iOS simulator — against a real `ciris-server`.
+
+That is what `testable` means in `CSD.md` §1: *"floor flips off unreleased; flow
+runs on the matrix"*.
+
+## The file
+
+```yaml
+flow: people                  # the flow's id; unique across this directory
+csd: CSD-005                  # the CSD it tests — REQUIRED here
+title: People on a node with no contacts yet
+description: >-               # optional; say what is NOT driven and why
+  …
+client: ">=0.5.224"           # the client that carries every tag it names
+
+steps:
+  - step_id: landing
+    title: Signing in on a bare node lands on People
+    requires:                 # checked BEFORE the step; the first step's is the entry
+      screen: Contacts
+    do:                       # click / input / scroll_to / wait (one per entry)
+      - wait: card_contacts_add
+        wait_ms: 5000         # a `wait` waits wait_ms × 4
+    expect:                   # checked AFTER
+      visible: [card_contacts_add]
+      absent: [contacts_empty]
+
+  - step_id: search_no_match
+    title: A search nothing matches is the empty state
+    do:
+      - input: {input_contacts_search: "zz-no-such-contact"}
+    expect:
+      state: empty            # read from the CSD's `states:` block
+```
+
+The language is `testing/gate/flow_spec.py`'s — vendored from CIRISAgent, with
+the CSD/3 §3 predicates (`count`, `number`, `matches`, `one_of`, `each`,
+`relation`, `state`) and one local addition, `csd:` (see
+`testing/gate/VENDORED.md`). Unknown keys anywhere are a load error.
+
+### How a flow is tied to its CSD
+
+`csd: CSD-NNN` resolves to the one `FSD/CSD/CSD-NNN-*.md`, and the runner reads
+two of its typed blocks (`testing/gate/csd_doc.py`):
+
+- **`csd:shows`** → field id → tag. A `relation:` names `ceg:` field ids
+  (`capacity:composite`), not tags, and this is how they resolve.
+- **`csd:states`** → state → tag. `state: empty` holds when the `empty` tag is on
+  screen **and no other state's tag is** — so an error cannot pass for "nothing
+  here".
+
+Each of these is a **load error**, found before any app is started:
+
+- the CSD does not exist, is ambiguous, or a typed block does not parse;
+- the flow names a tag the CSD still marks `proposed:` — in `requires`,
+  `expect` or `do`. No client carries it, and it would fail as "element not
+  found", which looks exactly like a broken app;
+- `state: X` where the CSD gives X no tag, or a proposed one;
+- a `relation:` operand that is not one of the CSD's `shows:` fields.
+
+`testing/test_flows.py` also checks that every literal tag a flow here names is
+a string in the client's `commonMain` source.
+
+## Verdicts
+
+| verdict | when | leg |
+|---|---|---|
+| `pass` | every step held | green |
+| `refused` | the `client:` floor is above the client under test (`>=X`, `>X`, `unreleased`) | green — reported, not passed |
+| `cannot-start` | floor met, but the flow never reached its first screen (no hop, a hop tag missing, or the first `requires` never held) | **red** |
+| `fail` | it started and a step broke | **red** |
+
+`cannot-start` is red on purpose. The floor is how a flow waits for a surface
+that has not shipped; once the floor is met, a flow that never reached its first
+screen is a flow that silently never ran.
+
+Each leg's report (`reports/<leg>.json`) carries a `flows` list with every
+outcome and its step-level detail; screenshots and per-flow JSON land under
+`shots/flows-<leg>/`. The client version the floor is checked against is this
+tree's `VERSION` — on the matrix the artifact is asserted to be this tree's.
+
+## From `building` to `testable`
+
+1. Every tag the flow needs is real: no `proposed:` left on the rows it drives,
+   and the PR that adds them has shipped.
+2. Write `testing/flows/<surface>.yaml` with `csd:` pointing at the CSD and
+   `client: ">=<the release that carries it>"`. Until a release carries it, use
+   `client: unreleased` — the flow loads, is checked, and is refused on the
+   matrix instead of failing.
+3. Run it locally (below), then let the nightly matrix run it.
+4. When it is **green on the platforms the CSD's §5 declares**, the pen-holder
+   moves `stage:` to `testable`. Nothing advances the stage automatically — a
+   green run is evidence for the edit, not the edit.
+
+## Running one flow locally, against a desktop client
+
+The runner drives whatever client answers on the test-automation port. Keep it
+off your own install: the node takes `--home`, the client reads `CIRIS_HOME`
+(`testing/gate/session_fixture.py` explains why they differ).
+
+```bash
+# 1. a throwaway node
+python3 -m testing.gate.node_fixture --version v0.5.224 --home /tmp/flows-node \
+    --platform x86_64-unknown-linux-gnu        # or aarch64-apple-darwin
+
+# 2. the desktop client in test mode, with its own home
+( cd client && ./gradlew :desktopApp:packageUberJarForCurrentOS )
+CIRIS_TEST_MODE=true CIRIS_HOME=/tmp/flows-client \
+    java -jar "$(python3 -m testing.gate.candidate_artifacts --kind desktop | tail -1)" &
+
+# 3. the flow — it signs in (running first-run setup if the node has no owner)
+python3 -m testing.gate.run_flows --platform desktop \
+    --flows testing/flows/people.yaml --report /tmp/flows.json
+```
+
+`--client-version` checks floors against another version (default: `VERSION`);
+`--no-sign-in` drives whatever screen the client is already on; `--url` points at
+a test server other than `http://127.0.0.1:9091`. Exit status is 0 only when
+every flow passed or was refused.
+
+On the matrix the same code runs inside `run_platform` (`--flows testing/flows`)
+after the smoke walk, in the app the walk just brought up, so there is one
+bring-up per leg and one session.
+
+## What this does not do yet
+
+- **Navigation is to the first screen only.** Before step one the runner walks
+  the hop `testing/gate/nav_map.py` derives for the flow's first
+  `requires: screen:` on this build (node or agent tree, from `/state`),
+  waiting for each tag before clicking it. A missing hop tag is `cannot-start`
+  naming the tag; a screen with no hop that is not flow-only is `cannot-start`
+  with "no nav hop for Screen.X"; a flow-only screen (pre-login, wizards,
+  leaves) is waited for, not walked to. Hops between later steps are the flow's
+  own `do:` clicks.
+- **Only what a bare node can show.** The matrix stands up one node with no
+  contacts, no agent and no peers, so CSD-005's populated list and receipt sheet
+  are not driven here.

testing/flows/people.yaml

@@ -0,0 +1,62 @@
+flow: people
+csd: CSD-005
+title: People on a node with no contacts yet
+description: >-
+  The matrix's first CSD flow, and the one that proves the wiring: every leg signs
+  in on a bare ciris-server, which has no contacts, so People lands in the
+  "empty and unsearched" shape CSD-005 §4 describes — the add card INSTEAD of an
+  empty block. A search nothing matches then shows the empty state proper, and
+  clearing it brings the add card back. Every tag here ships in 0.5.224
+  (PeopleTags in ContactsScreen/PeopleSupport.kt).
+
+  Not driven here, and why: the populated list and the receipt sheet need a
+  second node to be a contact of, which the matrix does not stand up; the
+  `proposed:` trust chip cannot be named by a flow at all.
+client: ">=0.5.224"
+
+steps:
+  - step_id: landing
+    title: Signing in on a bare node lands on People, offering to add someone
+    description: >-
+      `requires` is the entry precondition, stated rather than assumed: a client
+      that landed anywhere else reports "cannot start", not "a People element is
+      broken". The wait absorbs the first /v1/contacts read (the loading state).
+    requires:
+      screen: Contacts
+    do:
+      - wait: card_contacts_add
+        wait_ms: 5000
+    expect:
+      screen: Contacts
+      visible: [input_contacts_search, card_contacts_add, input_contacts_add_key,
+                btn_contacts_add_submit, btn_contacts_refresh]
+      # The add card stands in for the empty block over an EMPTY, UNSEARCHED
+      # list; and a node that answered is neither loading, nor in error, nor
+      # too old for contacts.
+      absent: [contacts_empty, contacts_list, contacts_loading, contacts_error,
+               contacts_unsupported]
+
+  - step_id: search_no_match
+    title: A search nothing matches is the empty state, not an error
+    description: >-
+      `state: empty` is checked against CSD-005's `states:` block: contacts_empty
+      on screen, and the populated, loading and error tags all absent — so an
+      error cannot pass for "no matches".
+    do:
+      - input: {input_contacts_search: "zz-no-such-contact"}
+      - wait: contacts_empty
+        wait_ms: 2500
+    expect:
+      state: empty
+      absent: [card_contacts_add]
+
+  - step_id: clear_search
+    title: Clearing the search brings the add card back
+    do:
+      - input: {input_contacts_search: ""}
+      - wait: card_contacts_add
+        wait_ms: 2500
+    expect:
+      screen: Contacts
+      visible: [card_contacts_add]
+      absent: [contacts_empty, contacts_error]

testing/gate/VENDORED.md

@@ -67,6 +67,24 @@ tagged is drivable" about a screen with no elements on it.
 `check_csd.py`, and two implementations of one DSL is the drift this repo exists
 to measure.
 
+### `flow_spec.py` — local delta: a flow names its CSD (`csd:`)
+
+Added so CSD flows can run on this repo's matrix (`testing/flows/`,
+`testing/gate/run_flows.py`). Each change is marked `LOCAL DELTA` in the source:
+
+| change | reason |
+|---|---|
+| `csd` added to `_FLOW_KEYS`; `FlowSpec.csd_id` / `FlowSpec.csd` | a flow says which CSD it tests, so the runner can read that CSD's `shows:` (→ `field_tags`, for `relation`) and `states:` (→ `state_tags`, for `state:`) instead of every caller wiring them by hand |
+| `FlowSpec.load(path, csd_root=None)` binds the CSD at load | a CSD that is missing, ambiguous or does not parse is a **load error**, never a skip — a flow that silently loses its CSD loses the checks that make `state:`/`relation:` mean anything. The CSD is read by `csd_doc.py` (ours), which takes its block grammar from `packaging/check_csd_v3.py` rather than re-typing it |
+| a `proposed:` tag named in `requires`/`expect`/`do`, a `state:` whose tag is proposed or absent, or a `relation` operand that is not a `shows:` field → load error | no client carries a proposed tag, so asserting it fails as "element not found" — indistinguishable from a broken app. `count`/`each` globs are deliberately NOT checked: CSD-005's real `contacts_row_*` rows share a prefix with its proposed `contacts_row_trust` chip |
+| `state:` is now CHECKED: that state's tag on screen, every other state's tag not; with no `states:` map it fails | upstream's `state:` body was `pass` — the one predicate CSD/3 makes mandatory asserted nothing, a vacuous green. Upstream flows do not use `state:`, so none of them changes behaviour |
+| `FlowRunner(state_tags=…)`; `run()` fills both maps from `spec.csd` when the caller did not | one source for the maps: the CSD |
+| `write_report` records `csd` | a result that cannot say what it tested cannot be acted on |
+
+A flow without `csd:` still loads and runs exactly as before; `run_flows.py`
+is what requires the key for flows in this repo. Upstream needs the same key
+before CIRISAgent's flows can be checked against their CSDs the same way.
+
 ### `platforms.py` — and a mistake this file previously recorded as a fact
 
 An earlier version of this document said all three vendored modules were