MCP Phase 3b: Tasks (progress/cancel) — deferred, needs reindex_files cancel/progress infra + ownership model

[jamie8johnson]

Phase 3b of #2124 (MCP Tasks — progress/cancel for the long-runners), split out of the Phase-3 arc after a four-adversary attack on the design doc (docs/plans/2026-07-03-mcp-phase3-decisions.md) convicted the Tasks half on false premises. Phase 3a (the HTTP transport) ships independently; this tracks the deferred Tasks work.

Trigger to enter: an eval-core lift (#NEW-eval-issue) + a GPU-contention policy — the same gate as the eval descope.

What 3b must BUILD (the draft under-budgeted all of these):

1. Progress + cancellation added to reindex_files (the daemon's actual index engine) — it has NEITHER today; the run_index_pipeline atomics/cancel-checks the draft cited are CLI-only and unreachable from the daemon.
2. A DECIDABLE cancel rule: job-cancel honored only while QUEUED; once pending_files drains, cancel = detach-only, job runs to completion. ("job was task-initiated" is undecidable — file provenance is destroyed when a task walk and inotify events merge into one HashSet.)
3. Store-side cancel-safety: requeue unprocessed files (the events.rs Err arm doesn't today) + don't skip the post-run SPLADE encode for committed files (else the dense-only drift window becomes client-triggerable). The manifest-generation "abort-anywhere safe" claim covers only the HNSW sidecar.
4. An OWNERSHIP model: shared token + always-advertised tasks/list means any holder can list+cancel any task (re-entering cancel-one-cancels-all). Decide single-principal or taskId-as-capability. Resolve current-vs-next attachment.
5. tasks/result as a transport-side poll loop, NOT a blocking relay (a blocking relay times out AND exhausts the capped daemon handler pool = self-DoS).
6. Tasks spec conformance: task-gone ⇒ JSON-RPC -32602 not HTTP 404; createdAt/lastUpdatedAt; related-task _meta; capability at init; taskSupport -32601. Verify session-scoping of tasks vs the survive-restart property.

Full corrected design in the plan doc's PHASE 3b section.

[jamie8johnson]

Scout report: anatomy of cqs index as a long-runner (for the Tasks design)

A read-only scout pass mapped exactly the infra this issue's premise says is missing (progress/cancel on the reindex path). Persisting it here as the design's ground truth. Caveats: snapshot taken 2026-07-04 ~04:20Z — pre-#2224 (cache eviction has since moved out of the watch loop's flush_due branch) and pre-#2228 (the build.rs "daemon holds a shared HNSW flock" comment it flagged as a discrepancy is now fixed — detection is a socket probe). Line numbers will drift; the structural facts below were spot-verified today.

Progress checkpoints that already exist

• File count known up front from enumerate_files — a natural denominator.
• Shared parsed_count/embedded_count AtomicUsize counters already cross the pipeline threads (src/cli/pipeline/mod.rs) — the existing cross-thread progress state a Tasks progress stream could read.
• Per-batch loops with natural checkpoints in parser stage, both embed stages, store stage (drives the indicatif bar via set_position), SPLADE (while idx < total_chunks, adaptive batch), enrichment (progress.inc(1) per chunk).
• Opaque once started: HNSW build_batched_with_dim — no progress callback, no cancel hook inside.
• No machine-readable incremental progress stream exists anywhere; JSON output is end-only (IndexSummaryOutput). The daemon's per-tick WatchSnapshot (read by cqs status --watch) is the only out-of-band progress surface today.

Cancellation machinery that already exists

• Global two-phase INTERRUPTED: AtomicBool (src/cli/signal.rs): first signal = finish current batch, second = hard exit 130. Stage granularity: parser between file batches, embedders between batches, store stage between embed batches.
• Daemon: separate SIGTERM handler + per-rebuild shutdown: Arc<AtomicBool> on the HNSW rebuild thread (signaled before bounded join, skip-save on shutdown).
• No cooperative cancellation token inside pipeline stages beyond the global flag; none at all inside HNSW build, enrichment (zero check_interrupted sites — runs to completion once entered), or a single SPLADE/embed batch.

Cancel-safety per stage (what an abort leaves behind)

• SQLite: per-file fused transactions — abort leaves every file fully committed+stamped or untouched; staleness pre-filter resumes the remainder exactly.
• Embedding cache written per batch as computed → completed batches survive abort; re-run is a cache hit (the main resumability asset).
• HNSW dirty flags set before first chunk write, cleared only after successful saves → abort degrades to brute-force search, no corruption, self-heals.
• Generation-scheme saves: atomic manifest commit; crash orphans are GC'd by the next save; flocks are kernel-held (auto-release on death).
• Leaks on abort: orphan gen<N> files until next save's GC; index.db.bak after an interrupted --force (not auto-restored); needs_embedding=1 sentinels (self-healing next run).

Daemon-side shape (what Tasks would actually drive)

• dispatch_reconcile only flips a coalescing AtomicBool; the walk QUEUES divergent files; actual parse/embed/write runs synchronously on the main watch thread inside its 100ms tick loop — not on a worker thread. The only background index work is the HNSW rebuild thread (own read-only Store, mpsc results, delta replay on drain).
• --force under a live daemon bails (socket probe); the daemon detects the inode swap tick-based, not transactionally.

Full detail (locks, generation machinery, --force sequence) is in the scout transcript; the structural summary above is what the ownership/cancel-predicate design has to work with. It confirms this issue's premise: Tasks needs the progress stream and the cooperative cancel token built first — neither exists.

[jamie8johnson]

Scout report: serve-stack facts relevant to 3b's SSE half

Companion to the index-anatomy comment above; same caveats (2026-07-04 ~04:20Z snapshot, pre-#2223 — its SECURITY.md quotes predate the MCP-over-HTTP section that now exists; line numbers drift).

Facts that bear on the SSE/resumability design folded into this phase:

• No streaming exists anywhere in src/ today. serve is single-shot Json<T> responses only — no SSE, no WebSocket, no chunked bodies. Zero text/event-stream hits outside historical docs (the removed v0.9.x MCP server had HTTP+SSE; its audit trail noted an SSE keep-alive/no-idle-timeout finding worth remembering when we build the replay buffer).
• serve's handler shape: sync Store calls behind spawn_blocking + a shared permit semaphore; axum on a tokio runtime built inside run_server. The mcp --http listener reuses this stack — SSE would be the first long-lived response either surface holds, which interacts with: the serve idle-shutdown clock (bumped per request — a long-lived SSE stream would need to decide whether it counts as activity), the concurrency-cap semaphore (a permit held for the stream's lifetime eats the budget), and the blocking-permit pool (must NOT hold one across the stream).
• serve relays exactly one route to the daemon (/api/search_legs) over a blocking unix-socket round-trip with a 16 MiB response cap — the same relay shape mcp --http uses; a Tasks event stream is a different animal (server push), which is why SSE was deferred to ride on the Tasks event model rather than built speculatively (P3a-2 → here).
• Auth model available for reuse: per-launch 256-bit token, three channels, constant-time compare, port-scoped cookie; mcp --http already reuses the serve auth primitives. Sessions on the mcp side are server-issued (fixation-closed) — the SSE resumability design (Last-Event-ID replay) should key its replay buffer to those sessions.

[jamie8johnson]

Phase 3b design — CONVERGED (2026-07-05), ready for review before any build

Full decision record: docs/plans/2026-07-05-mcp-3b-decisions.md. Produced by a 4-scout design pass + a 3-adversary attack (spec-fidelity / seam / red-team), the same pattern that hardened #2218 and #2217. Not built — the design is the deliverable; 3b opens a new network-facing stateful subsystem, so it gets eyes before code.

Headline: 3b is far smaller than the Phase-3 doc feared

The spec-fidelity scout (MCP 2025-11-25 Tasks + schema.ts) established the minimal conformant floor, and it dissolves the three hardest problems:

• No tasks/create — task creation is augmenting a tools/call with a task field; the server returns CreateTaskResult instead of the normal result.
• No SSE / no server-push — the floor has ZERO server-initiated messages (progress is pure-MAY), so the "how does a daemon-client transport push progress" crux simply doesn't arise. P3a-2 stays deferred.
• No ownership fork — tasks/list/tasks/cancel are opt-in; the spec SHOULD-NOTs listing under a shared token, which picks our answer.

A post-cutoff 2026-07-28 RC removes tasks/list and goes stateless-extension, ratifying the floor's direction (re-fetch at build time).

The floor (each item spec MUST or a marked SHOULD/MAY)

Task-augment ONE tool — cqs_eval (taskSupport: "optional", preserving the shipped one-shot path) — with tasks/get (cheap poll) + tasks/result (block-until-terminal). Daemon-scoped taskId-capability (spec-aligned, not a deviation). Single-flight = the concurrent-task cap. A daemon-side TaskStore (spawn_hnsw_rebuild-shaped worker, GC on the 60s sweep). eval_core untouched.

What the attack changed (all folded)

• red-team RT-DOS-2 (HIGH): the original "separate budget" for the 20-min tasks/result block was ineffective — the concurrency cap is the outermost layer and holds a permit before the method is known, so 256 blocks would 503 everyone (re-opening the MCP-HTTP listener has no concurrency cap (RT-DOS-1) #2220 gap). Fixed to an isolated K≪256 fan-out-capped budget that doesn't hold the outermost permit.
• red-team RT-DOS-3 (HIGH): single-flight caps running to 1 but not retention — tiny fixtures pile terminal entries between GC sweeps. Fixed with an admission-time count cap + total-bytes bound + a fixture-read byte cap.
• seam 1b (the real new bug): a worker dying without publishing leaves a zombie task at working forever (terminal-only GC never collects it). Fixed with a publish-on-drop failed handle.
• seam 2b (the sharpest): a 20-min eval stamps freshness: fresh at spawn but the index can move mid-run — tasks/result would hand the recall-gate a "fresh, resident" verdict for an index that no longer exists. Fixed by making the epoch stamp load-bearing at read with _meta staleness injection.
• Plus: panic=abort reality (worker panic aborts the daemon — same as today, but the safety story is the panic fence, not unwind-release), RT-LEAK-1 (no taskId in the thread name), RT-ORACLE-1 (both at-cap reasons → one opaque code), and 5 spec-fidelity corrections (failed-task error passthrough, isError→failed, wire-shape asymmetry, ISO-8601, SECURITY.md doc).

The doc carries the full requirement checklist for the build lanes. Awaiting review before building the two floor lanes.

[jamie8johnson]

P3a-2 PR1 landed: #2244 merged (57e57787). Daemon-side eval-task progress plumbing per docs/plans/2026-07-05-p3a2-sse-decisions.md §D2 — ProgressCounters side-channel on EvalCoreParams, the one-frame {done, total} (processed-count / M-at-admission, mirror-bug tripwire), status-before-counters read order, descriptor_json skip-if-none wire emission. Fable review APPROVE (risk LOW); CI 5/5 per-job green.

Remaining on this ladder: PR2 = SSE transport (http.rs) consuming this progress; PR3 = the pre-scheduled second trust-boundary probe pass against the live binary once PR2 deploys.

[jamie8johnson]

P3a-2 PR2 (SSE server-push) landed — PR #2250 merged (70d0bf8e). The GET flip from the 405 floor to a notifications-only SSE progress stream over the MCP HTTP transport, consuming the #2244 progress plumbing.

Pre-land trust-boundary probe (Opus session, per the decisions doc requirement): 41 invariants HELD / 0 BREACH. Five independent probers, each on its own isolated live cqs watch --serve + cqs mcp --http stack, crafted real raw-HTTP PoCs against every claim in SECURITY.md's "MCP over HTTP" section — uniform auth on POST/GET/DELETE, the 16-stream budget + takeover-transfer + SSE-escapes-256-cap, tasks/result fan-out cap, zero secret leakage (session-id/taskId/progressToken/token absent from every log + thread name), crypto-random id entropy, the mutation triple-gate, and SSE-injection defeat (crafted CRLF/event:/data: in a progressToken is JSON-escaped inside one data: line). Every probed invariant already has a named guard in the 64-test http suite; the pre-scheduled PR3 second probe against the deployed binary remains as the post-deploy pass. Full probe writeup on PR #2250.

Remaining under #2216: the P3a-2 ladder is complete (PR1 #2244 progress plumbing + PR2 #2250 SSE); 3b Tasks floor (lanes 1/2) landed earlier (#2237/#2238/#2242). No daemon redeploy for PR2250 (http.rs is mcp-http+unix-gated, out of the deployed default build).