From a2416db90658b4c94daf1468041b43893f7c6c5e Mon Sep 17 00:00:00 2001 From: Aniket Wattamwar Date: Sat, 29 Aug 2026 22:08:41 -0600 Subject: [PATCH] docs: add ECP-0001 and ECP-0002 design proposals --- spec/proposals/0001-protocol-maturity.md | 409 ++++++++++++++++++ .../0002-execution-traces-and-inspector.md | 300 +++++++++++++ 2 files changed, 709 insertions(+) create mode 100644 spec/proposals/0001-protocol-maturity.md create mode 100644 spec/proposals/0002-execution-traces-and-inspector.md diff --git a/spec/proposals/0001-protocol-maturity.md b/spec/proposals/0001-protocol-maturity.md new file mode 100644 index 0000000..c376470 --- /dev/null +++ b/spec/proposals/0001-protocol-maturity.md @@ -0,0 +1,409 @@ +# ECP-0001: Protocol Maturity Roadmap + +Status: Draft — for discussion +Author: ECP maintainers +Targets: protocol `1.0`, runtime `1.x` + +## Summary + +Four phases that take ECP from "a working eval runner with a protocol attached" +to "a protocol other people can build on, and a benchmark others can trust." + +They are strictly ordered. Each depends on the one before it, and the last is +the one everyone wants to do first. + +| Phase | Work | Why it comes here | +| --- | --- | --- | +| 1. Now | Version negotiation and real capabilities | The window closes as adoption grows. Everything else needs a safe way to evolve. | +| 2. Next | Grader depth | Makes ECP a policy-testing tool rather than a string matcher — the story the README already tells. | +| 3. Then | Determinism and variance | Unglamorous prerequisite for any comparable number. | +| 4. Then, and only then | Publishable benchmarks | Credible only on top of 1–3. | + +The temptation is to jump to phase 4. A leaderboard built on single-run, +unpinned-judge, unversioned manifests produces numbers nobody can reproduce — +and a protocol only gets to be wrong about that once. + +--- + +# Phase 1 — Version negotiation and capabilities + +## The problem + +**There is no protocol version anywhere in the wire format.** `agent/initialize` +exchanges a name and an empty object. A runtime cannot tell whether it is +talking to an agent built against today's spec or one from six months ago, so +there is no safe way to ever make a breaking change. We detect incompatibility +by watching things fail strangely. + +**`capabilities` is a dead field.** The runtime validates it is an object +([conformance.py](../../runtime/python/src/ecp_runtime/conformance.py)) and then +ignores it. Nothing is ever declared and nothing ever reads it. + +That has concrete costs today: + +- A manifest with `tool_usage` graders run against an agent that cannot report + tool calls fails with `"No tool_calls present"` on every step. That is + **indistinguishable from an agent that legitimately used no tools.** One is a + broken integration, the other is a real test failure, and we report them the + same way. +- `agent/reset` returns `true` unconditionally. In + [server.py](../../sdk/python/src/ecp/server.py), `_handle_reset` runs the hook + if one is registered and returns `true` either way — so an agent with no reset + support claims successful reset, and scenario isolation silently does not + happen. +- `params.config` is documented in the spec and **dropped entirely** by + `_handle_init`. Either it means something or it should not be in the spec. + +## Proposal + +### Version exchange + +Both directions of `agent/initialize` carry `protocol_version`, a +`MAJOR.MINOR` string. + +- **MAJOR** increments on breaking changes. Different major versions are + incompatible. +- **MINOR** increments on additive changes. A runtime speaking `1.2` must work + with an agent speaking `1.0`, and vice versa, degraded to the lower minor. + +This is deliberately **not** the package version. `ecp-runtime` is at `0.9.0` +and moves weekly; the protocol should move yearly. Coupling them was already a +source of drift in the docs. + +Request: + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "agent/initialize", + "params": { + "protocol_version": "1.0", + "runtime": { "name": "ecp-runtime", "version": "0.9.0" }, + "capabilities": {}, + "config": {} + } +} +``` + +Response: + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "name": "SupportAgent", + "protocol_version": "1.0", + "capabilities": { + "tool_calls": {}, + "evaluation_context": {}, + "usage": {}, + "reset": {}, + "multi_turn": { "max_turns": 20 } + } + } +} +``` + +**Negotiation rules** + +1. The runtime sends the highest version it speaks. +2. The agent responds with the version it will actually speak — the runtime's + version if it supports it, otherwise its own highest. +3. If the majors differ, the runtime MUST abort the run with a clear error + rather than proceeding. +4. If minors differ, both sides operate at the lower minor. + +**Absence means legacy.** An agent that omits `protocol_version` is assumed to +speak `0.1` — every agent built against the current SDK. The runtime MUST +continue to drive those agents, warning once. This is the whole reason to do it +now rather than after there are thousands of them. + +### Capabilities + +Presence of a key means supported. The value is an options object, empty when +there are no options. This mirrors MCP and leaves room to add options without a +version bump. + +Proposed agent capabilities, each motivated by something broken today: + +| Capability | Meaning | Runtime behaviour when absent | +| --- | --- | --- | +| `tool_calls` | Reports tool invocations | Fail fast if the manifest has `tool_usage` graders | +| `evaluation_context` | Populates evaluator-safe context | Fail fast if a grader targets that field | +| `usage` | Reports token accounting | Audit `usage` reported as unavailable, not zero | +| `reset` | Honours `agent/reset` | Warn that scenario isolation is not guaranteed | +| `multi_turn` | Maintains state across steps within a scenario | Warn on multi-step scenarios | +| `deterministic` | Same input yields same output | Required by phase 4; absent means benchmark scores need `repeat` | + +The payoff is the failure mode getting honest. Instead of: + +``` +FAIL | tool_usage on tool_calls — No tool_calls present. (x40) +``` + +you get, once, before anything runs: + +``` +Manifest requires tool_usage graders, but agent 'SupportAgent' does not +declare the 'tool_calls' capability. Either the adapter is not reporting +tool calls, or this agent cannot be evaluated by this manifest. +``` + +**Unknown keys MUST be ignored** by both sides. That is what makes minor +versions additive. + +**Vendor extensions** use an `x-` prefix with a vendor segment: +`x-acme.trace_export`. Reserved so a vendor can extend without collision and +without the core spec having to know. + +### `config` + +Keep it, define it, and wire it up: manifest-provided configuration passed +through to the agent, exposed via a new `@on_initialize` SDK hook. A manifest +gains an optional `agent_config` block. If we would not do this, `config` should +be struck from the spec instead of remaining documented and ignored. + +### New error codes + +Reserve an ECP range within the JSON-RPC implementation-defined space: + +| Code | Name | Meaning | +| --- | --- | --- | +| `-32001` | `VERSION_UNSUPPORTED` | Major version mismatch | +| `-32002` | `CAPABILITY_REQUIRED` | Manifest needs a capability the agent lacks | +| `-32003` | `CONFIG_INVALID` | Agent rejected the supplied `config` | + +Currently everything is `-32000`, so a runtime cannot distinguish "your agent +crashed" from "your agent refuses this configuration." + +## Compatibility + +Fully additive. Old agents omit the fields and keep working under the `0.1` +assumption. Old runtimes ignore fields they do not know. The SDK sets +`protocol_version` and infers capabilities from registered hooks, so most users +get this by upgrading. + +The one behaviour change: `reset` capability makes the current +"always return `true`" a lie we stop telling. Agents without an `@on_reset` hook +will stop declaring `reset`, and the runtime will warn on multi-scenario runs. + +## Open questions + +- `MAJOR.MINOR` versus MCP's date-based versions. Dates avoid arguments about + what counts as breaking; semver is easier to reason about. Leaning semver. +- Should the SDK infer capabilities from hooks, or require them explicit? + Inference is friendlier and risks being wrong; explicit is honest and is + boilerplate. +- Does `deterministic` belong here or in phase 3? + +--- + +# Phase 2 — Grader depth + +## The problem + +Three graders: `text_match`, `llm_judge`, `tool_usage`. `tool_usage` can only +assert a tool **was** called. The README positions ECP as a policy and tool-use +testing tool, and the most important policy assertions cannot be written: + +- "must **not** call `issue_refund`" — the single most valuable assertion for an + agent with authority it should not exercise +- "`lookup_order` **before** `issue_refund`" — ordering, i.e. did it check before + it acted +- structured output assertions — PydanticAI agents serialize JSON into + `public_output`, so today you `contains`-match against JSON text +- cost and latency as pass/fail, now that the audit record carries both + +## Proposal + +Reuse the existing `condition` field, which `GraderConfig` already has for +`text_match`: + +```yaml +# Negative assertion — the agent must not exercise authority it lacks +- type: tool_usage + condition: not_called + tool_name: issue_refund + +# Ordering — it must check the policy before acting on it +- type: tool_sequence + mode: ordered_subsequence # or: exact, unordered_subset + tools: [lookup_order, check_refund_policy, issue_refund] + +# Structured output +- type: json_match + field: public_output + path: "$.refund.eligible" + equals: true + +# Budgets, from the phase-0 audit record +- type: budget + max_latency_ms: 5000 + max_total_tokens: 8000 +``` + +`json_match` needs a JSONPath dependency in the runtime; the SDK stays +zero-dependency. + +`budget` graders make the audit record load-bearing rather than informational, +which is a good forcing function for keeping it accurate. + +## Compatibility + +Additive. New grader types and one new `condition` value. Existing manifests are +unaffected. Schema and the `GraderConfig` validator both need the new types, and +`ecp validate` should reject a `not_called` grader that also specifies +`arguments`, which is meaningless. + +## Open questions + +- Should `not_called` be a `condition` on `tool_usage` or its own grader type? + Condition reuses machinery; a separate type reads better in YAML. +- `tool_sequence` default mode. `ordered_subsequence` is the forgiving choice + and probably right — agents legitimately interleave other calls. + +--- + +# Phase 3 — Determinism and variance + +## The problem + +**One run is not a score.** Agents are stochastic; the same manifest against the +same agent gives different results. Everything today reports a single pass rate +as if it were a measurement. + +**`llm_judge` silently poisons comparability.** The judge model comes from +`ECP_LLM_JUDGE_MODEL` (default `gpt-4o-mini`) read from the environment at grade +time — it is not in the manifest, so it is not in the manifest digest. When the +provider updates that model, every historical score shifts and nothing in the +report records that anything changed. + +`ecp trend` already does cross-run pass-rate analysis, but it compares runs +without any notion of expected variance, so it cannot distinguish noise from +regression. + +## Proposal + +**Repeats.** `--repeat N`, or per-scenario `repeat:`. The audit record gains a +`runs` array and a `variance` block: pass rate mean, standard deviation, and +min/max per check. A check that passes 3 of 5 times is reported as flaky rather +than as pass or fail. + +**Pin the judge in the manifest**, not the environment: + +```yaml +- type: llm_judge + model: "gpt-4o-mini-2024-07-18" + temperature: 0 + prompt: "Does the answer cite a policy?" +``` + +Environment variables become the fallback, not the source of truth. The pinned +model then lands inside the manifest digest, so a judge change produces a +different digest, which is exactly what we want. + +**Separate the headline number.** Report deterministic-grader pass rate and +judge-based pass rate independently. Only the deterministic one is comparable +across time without caveats. + +**`ecp trend` gets variance-aware**, flagging regression only when a change +exceeds observed run-to-run noise. + +## Compatibility + +Additive, except that `llm_judge` results acquire a warning when no model is +pinned. Report and audit schemas gain optional blocks. + +## Open questions + +- Default `repeat`. `1` keeps CI fast; anything higher multiplies cost. Probably + `1` by default and required `>1` for benchmark mode. +- How to surface flakiness in exit codes. Does a check passing 4 of 5 fail the + build? Probably configurable, defaulting to strict. + +--- + +# Phase 4 — Publishable benchmarks + +## The problem, and the opportunity + +An ECP manifest is already a runnable benchmark: a declarative behaviour +contract whose agent side is framework-agnostic. Someone can publish +`refund-policy@1.0` and **any** agent — CrewAI, PydanticAI, Strands, a raw Node +script — can run it and produce a comparable number. + +No framework-coupled eval tool can do that. This is the concrete payoff of +vendor neutrality, and it is worth more than the migration story neutrality is +usually sold on. + +## Proposal + +A benchmark is a manifest plus its dataset, a pinned judge, a required +`repeat`, and a version — addressed by the manifest digest the audit record +already computes. + +```bash +ecp bench run refund-policy@1.0 --target "python agent.py" +``` + +Output is a **score card** whose provenance is the point: + +```json +{ + "benchmark": "refund-policy@1.0", + "manifest_digest": "sha256:c8a39311...", + "protocol_version": "1.0", + "runtime_version": "1.2.0", + "agent": { "name": "SupportAgent", "capabilities": ["tool_calls", "usage"] }, + "runs": 5, + "deterministic_score": { "mean": 0.86, "stddev": 0.04 }, + "judge_score": { "mean": 0.79, "stddev": 0.07, "model": "gpt-4o-mini-2024-07-18" }, + "cost": { "total_tokens": 41870, "usd_estimate": 0.31 }, + "latency": { "p95_ms": 812.4 } +} +``` + +**The registry starts as a git repository of manifests, not a service.** Version +by tag, distribute by URL. A hosted registry can come later if the format proves +itself; building the service first is how this becomes a product nobody uses. + +**No leaderboard until reproducibility is demonstrable.** Self-reported scores +without a trusted runner are worthless, and a public leaderboard invites both +gaming and contamination — a published benchmark ends up in training data. If we +ever do a leaderboard, it needs held-out variants and attested runs, which is a +governance problem more than an engineering one. + +## Compatibility + +Entirely new surface. `ecp bench` is a new command; nothing existing changes. + +## Open questions + +- Is `usd_estimate` in scope? It requires a pricing table that goes stale and is + provider-specific. Tokens are objective; dollars are not. Leaning tokens only. +- How do we handle a benchmark whose manifest requires a capability the agent + lacks — unscoreable, or scored zero? Unscoreable seems honest; zero is what + people will assume. +- Contamination: do we need a private held-out split from day one, or is that + premature for a benchmark nobody has heard of yet? + +--- + +## What this roadmap deliberately excludes + +- **The proxy / silent interception design.** It is a real adoption idea, but it + is a workaround for ECP being strictly runtime → agent. Whether the protocol + should become bidirectional is a phase 1.5 question we should settle before + building the workaround. +- **Tool mocking.** Depends on the bidirectionality decision above. +- **A TypeScript SDK.** Independently valuable, on its own track, not blocked by + any of this. + +## Sequencing note + +Phase 1 is the only one with a closing window. Version negotiation is cheap now +and progressively harder with every agent built against a versionless protocol. +Phases 2 through 4 can be reordered or descoped based on what users ask for. +Phase 1 cannot be deferred without cost. diff --git a/spec/proposals/0002-execution-traces-and-inspector.md b/spec/proposals/0002-execution-traces-and-inspector.md new file mode 100644 index 0000000..13fc5c1 --- /dev/null +++ b/spec/proposals/0002-execution-traces-and-inspector.md @@ -0,0 +1,300 @@ +# ECP-0002: Execution Traces And The Inspector + +Status: Draft — for discussion +Author: ECP maintainers +Depends on: [ECP-0001](0001-protocol-maturity.md) phase 1 (capabilities) + +## Summary + +The Inspector today lists manifests from `examples/`, runs one, and shows +pass/fail per grader. The thing people actually want from it — "five agents, +ten tools, one question: show me what actually happened, what was called +correctly, and what the agent made up" — **is not something ECP can currently +express.** + +That is a data model gap, not a UI gap. No amount of Inspector work produces a +call path, because the protocol carries no path. This proposal adds the missing +protocol surface first, then rebuilds the Inspector on top of it, then covers +interop with other eval tooling. + +--- + +## Part 0 — What exists today + +`server/src/server.js` (~500 lines, zero dependencies) plus a static client. +Two distinct modes: + +**Evaluation mode.** Scans for `manifest.yaml` files, lists them, and on "Run" +spawns `python -m ecp_runtime.cli run --manifest X --json --no-fail-on-error`, +polls a job, and renders scenario → step → grader results. + +**Session mode.** Opens a live stdio or HTTP JSON-RPC session against the agent +and lets you fire `agent/step` and `agent/reset` by hand, showing raw responses +and logs. This is the direct MCP-Inspector analogue and is the more +differentiated half. + +### Honest limitations + +| Issue | Detail | +| --- | --- | +| **Only reads `examples/`** | `EXAMPLES_ROOT` is hardcoded. A developer cannot point it at their own manifest. It is a demo *of* the Inspector, not a usable tool. | +| **Only runs from a repo checkout** | Paths are `REPO_ROOT`-relative and it injects `runtime/python/src` onto `PYTHONPATH`. There is no `npx` story. | +| **Hand-rolled YAML parser** | `parseManifest` is ~60 lines of indentation heuristics with an `indent >= 14` magic number. It cannot parse the flow-style graders used in `examples/crewai_demo/manifest.yaml`, datasets, or multi-line strings. | +| **Fragile result parsing** | Runtime JSON is recovered with `indexOf("{")` … `lastIndexOf("}")` over stdout. Any log line containing braces corrupts it. | +| **No progress** | `progress` goes `0` → `1`. A fifty-scenario run shows nothing until it finishes. | +| **Ignores the audit record** | Latency, token usage, `exit_reason`, and degraded-run state all exist now and none are shown. | +| **Read-only** | No editing a grader and re-running. The tightest useful loop — tweak, re-run, compare — is absent. | +| **Not in CI** | No tests, not built or linted by any workflow. | + +--- + +## Part 1 — Why the "call path" view is impossible today + +`tool_calls` is a flat list of `{name, arguments}`. Three things are missing. + +### 1. No tool inventory, so hallucination is undetectable + +To claim a tool was hallucinated you must know which tools exist. **ECP never +learns the agent's tool inventory.** A call to `lookup_order` and a call to +`lookup_ordr` are both just strings in a list. The runtime cannot tell a real +tool from an invented one, so the single most requested view cannot be built. + +### 2. No attribution, so multi-agent structure is invisible + +`examples/two_agent_demo` models its sub-agents *as tool calls* named +`planner_agent` and `writer_agent`. That is a convention the demo invented, not +a protocol concept. Nothing says which agent made a call, or that one agent +delegated to another. Five agents and ten tools collapse into one flat list of +fifteen names. + +### 3. No outcomes and no edges + +A tool call records `{name, arguments}` — not whether it **succeeded**, what it +returned, how long it took, or what triggered it. So "called six tools +correctly" is not derivable: ECP only knows the agent *claimed* to call them. + +That last point deserves emphasis. **ECP records self-reported behaviour.** The +agent (via its adapter) says what it did; nothing verifies it. That is a +reasonable design — it is what keeps ECP framework-neutral — but it should be +stated in the spec, because "the agent hallucinated a tool" and "the adapter +failed to report a tool" are indistinguishable today, and we already know the +second happens (see ECP-0003 work on adapter conformance). + +--- + +## Part 2 — Proposal: tool inventory and call structure + +### 2.1 Declared tool inventory + +Extends the `tool_calls` capability from ECP-0001: + +```json +{ + "capabilities": { + "tool_calls": { + "inventory": [ + { "name": "lookup_order", "description": "Fetch an order by id" }, + { "name": "check_refund_policy", "description": "Policy for an order" }, + { "name": "issue_refund", "description": "Issue a refund" } + ] + } + } +} +``` + +The runtime then classifies every reported call as **declared** or +**undeclared**, and the audit record carries the counts. + +**Terminology matters here.** We report `undeclared`, not `hallucinated`. An +agent may legitimately call a dynamically-registered tool. Hallucination is an +*interpretation* of undeclared calls that a human or a grader makes; the +protocol should report the fact, not the judgement. + +New grader: + +```yaml +- type: tool_inventory + condition: no_undeclared_calls +``` + +### 2.2 Richer tool calls + +Every field additive and optional, so existing agents stay valid: + +```json +{ + "id": "call_2", + "parent_id": "call_1", + "kind": "tool", + "agent": "writer", + "name": "lookup_order", + "arguments": { "order_id": "A100" }, + "status": "ok", + "error": null, + "duration_ms": 12.3, + "result_summary": "Order A100, purchased 5 days ago" +} +``` + +- `id` / `parent_id` give the call **graph** — this is what makes a path view + possible. Absent `parent_id` means a root call. +- `agent` gives **attribution** in multi-agent systems. +- `kind` is `"tool"` or `"agent"`, so delegation and tool use live in one graph + with a discriminator, rather than in two parallel arrays. Frameworks generally + implement delegation *as* a tool call, so this matches reality. +- `status` / `error` make "called correctly" answerable. +- `result_summary` is **evaluator-safe by contract** — a short summary, never + the raw tool result. Raw results are the most likely place for PII to leak + into CI artifacts, and we already have that problem with `evaluation_context`. + +With this, the view the user described becomes a straightforward render: + +``` +Step 1 "I want a refund for order A100" 1,204ms +│ +├─ agent planner ok 310ms +│ ├─ tool lookup_order {order_id: "A100"} ok 41ms declared +│ └─ tool check_refund_policy {order_id: "A100"} ok 28ms declared +│ +└─ agent writer ok 890ms + ├─ tool fetch_customer_tier {id: "A100"} error 12ms UNDECLARED + └─ tool issue_refund {order_id: "A100"} ok 63ms declared ← policy grader FAILED + +2 agents · 4 tools · 3 ok · 1 error · 1 undeclared +``` + +### 2.3 Compatibility + +Entirely additive. Agents that report `{name, arguments}` keep working and +simply produce a flat, single-level path. Adapters gain the richer fields +opportunistically — PydanticAI and LangChain both expose call ids and errors +already, so they can populate more than CrewAI can. + +The `tool_usage` grader is unchanged. New graders (`tool_sequence` from +ECP-0001 phase 2, `tool_inventory` here) build on the new fields. + +--- + +## Part 3 — Proposal: the Inspector as a real tool + +### 3.1 Make it usable outside this repo + +The blocking issue. Publish as `@ecp/inspector`, runnable via `npx`, operating +on the user's project: + +```bash +npx @ecp/inspector # discover manifests under cwd +npx @ecp/inspector --manifest evals/support.yaml +``` + +It should shell out to whatever `ecp` is on `PATH` rather than injecting +`PYTHONPATH` into a repo checkout. + +### 3.2 Stop parsing YAML in JavaScript + +Delete `parseManifest`. Call `ecp validate --json` and consume the runtime's +own parsed manifest. One parser, one source of truth, and the Inspector +automatically understands datasets, flow style, and every future manifest +feature for free. + +This requires adding `--json` output to `ecp validate`, which is small and +useful independently. + +### 3.3 Streaming progress + +`ecp run --progress` emits NDJSON events on stderr: + +```json +{"event":"scenario_start","name":"Refund inside window","index":1,"total":12} +{"event":"step_done","scenario":1,"step":2,"status":"ok","duration_ms":812} +{"event":"run_done","passed":31,"total":34,"exit_reason":"ok"} +``` + +The Inspector streams these instead of polling a job that reports `0` then `1`. +This also benefits plain CLI users on long runs, who currently get silence. + +### 3.4 Surface what we already collect + +The audit record has latency, tokens, `exit_reason`, and planned-vs-executed +steps. Show them. A degraded run — timeout, skipped steps — should be visually +distinct from a clean failing run, because they mean completely different +things and are currently rendered identically. + +### 3.5 The edit-and-rerun loop + +The thing that makes an inspector sticky is a tight loop: tweak a grader, +re-run, see what changed. Editing the manifest in the UI plus a run-over-run +diff ("3 checks newly failing, 1 newly passing") is higher value than any +individual view, including the trace. + +### 3.6 Read result payloads, not stdout + +Run with `--json-out` and `--audit-out` to temp files and read those, instead of +brace-scanning stdout. + +--- + +## Part 4 — Interop with other eval frameworks + +Three tiers, in order of how much work they are and how far they reach. + +### 4.1 The file is the integration (works today) + +`--json-out` and `--audit-out` are schema-validated artifacts. Any tool can +consume them. This is the honest baseline and should be documented as the +primary integration path rather than treated as a fallback. + +### 4.2 OpenTelemetry export (recommended) + +Rather than writing N bespoke exporters, emit OTel spans following the GenAI +semantic conventions. One integration reaches Langfuse, Arize, Datadog, +Honeycomb, Braintrust, and anything else that speaks OTLP. + +The call-graph work in Part 2 is what makes this possible: `id`/`parent_id` +map directly onto span/parent-span, `duration_ms` onto span duration, +`status`/`error` onto span status. **The trace view and OTel export are the same +underlying model** — build the model once and get both. + +```bash +ecp run --manifest evals/support.yaml --otlp-endpoint http://localhost:4318 +``` + +### 4.3 Native test-framework integration (partly exists) + +The pytest plugin already lets ECP run inside an existing suite. The equivalent +for Vitest/Jest arrives with the TypeScript SDK. This is the "use ECP without +adopting ECP's runner" path and is worth promoting more loudly — many teams will +never run `ecp run`. + +The existing `--export langsmith` should probably be **deprecated** in favour of +4.2. It currently creates runs with only input and output, dropping tool calls +and evaluation context, so it under-represents ECP to the one audience that +already understands evals. + +--- + +## Sequencing + +1. Tool inventory (2.1) — small, and it unblocks the most-requested view. Ships + with ECP-0001 capabilities. +2. Inspector portability (3.1, 3.2, 3.6) — turns a demo into a tool. Independent + of everything else. +3. Richer tool calls (2.2) — the protocol work behind the path view. +4. Progress streaming (3.3) and audit surfacing (3.4). +5. Trace view and OTel export (2.2 consumers, 4.2) — same model, two outputs. +6. Edit-and-rerun (3.5). + +## Open questions + +- **Is `agent` a string or a structured reference?** A string is simple; a + reference allows per-agent capability declaration in nested systems. Leaning + string until someone needs more. +- **Should `result_summary` exist at all?** It is genuinely useful for debugging + and is a PII risk by construction. Options: omit it, gate it behind a + capability, or make redaction mandatory before it reaches artifacts. +- **Do we verify anything?** Everything here is still self-reported. A stricter + mode where the runtime observes tool calls directly would require the proxy / + bidirectional work deferred in ECP-0001. Worth deciding whether "trusted + self-report" is a permanent design stance or a temporary one. +- **Does the Inspector belong in this repo?** A published npm package with its + own release cadence may want its own repository, like the TypeScript SDK.