Skip to content

CL-6477: validate the model create_agent asks for - #222

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-6477-model-validation
Aug 21, 2026
Merged

TheGreatAxios merged 3 commits into
mainfrom
cl-6477-model-validation

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Myra's create_agent tool call invents a free-text modelPreference — observed live as "gpt-4o" in a tenant whose catalog contains only Ollama models. Nothing validated it, so the agent was created pinned to a model that cannot exist, and could never answer.

  • The create route (workflow-create-routes.ts) now checks a requested model against the tenant's own capability inventory (the same inventory already resolved there for toolPackagePins — reused, not re-fetched). A model the catalog doesn't offer is never baked in: it falls back to the tenant's catalog default (or leaves the definition modelless if there is none), and the response carries a modelNote explaining the substitution. A model the catalog does offer is used exactly as asked, no fallback consulted.
  • create_agent's own tool schema now tells the calling model to look up a real catalog name (via list_agents/a models tool) or omit the field to inherit the default, rather than inviting a guess — the tool result also relays modelNote honestly so the calling model can tell the user what happened.
  • Fixes CL-6480 in the same trace: the client parsed currentVersion as "number", but the create route serializes workflow_definition.current_version — a text column — verbatim, always a string on the wire. Every genuine create was therefore failing this shape check, reporting a successful creation as an error and silently dropping the follow-on invite. Now parsed as "string", matching the real API.

Validation shape chosen, and why

Validation lives server-side in @corbits/agent-directory's workflow-create-routes.ts, against the same CapabilityInventoryProvider already wired there for toolPackagePins (models: [{ canonicalName }], backed by listMyraModels). This is the actual trust boundary the tool call crosses, matches the existing fail-closed pattern for tool-package pins, and needed no new plumbing. Preferred fallback over hard error, since the ticket asks for a working agent over a rejected call — a bad model name degrades gracefully to the tenant default instead of blocking creation outright.

Known gap, not fixed here

listMyraModels (the source behind capabilityInventory.models) is a flat per-tenant query, while real launch-time model resolution also inherits offerings from ancestor tenants (provenance: "set-here" | "inherited"). So a modelPreference naming an inherited model — one that would actually resolve fine at launch — could be wrongly treated as "not in catalog" here and substituted unnecessarily. The failure direction is safe (never a wrongful accept of a truly-unresolvable model, only a possible unnecessary fallback), and the asymmetry is pre-existing — the same capabilityInventory is already used identically for toolPackagePins today — but it deserves its own ticket rather than being fixed as a drive-by here.

Fixed after first review pass

CI caught a fixture break I missed in scoped testing: workflows/assistant/test/manager-tools-scenario.test.ts hand-mocked the create-agent response with currentVersion: 1 (a number) and no modelNote field — CL-6480's exact bug, reproduced in a fixture. That made the mocked response fail the client's own (now-correct) schema, so create_agent reported isError: true inside the fixture. Updated the fixture to currentVersion: "1", modelNote: null, matching what the real route returns. Re-grepped the whole repo for every fixture touching workflow-agent-directory/definitions (three total, all now consistent) rather than trusting the scoped package list alone.

Not verified

  • Did not boot a live stack (a live testing lane held the machine) — verified via scoped bun test + tsc --noEmit on agent-directory, agent-directory-tools, workflows/assistant, and apps/hub, plus root bun run lint, not an end-to-end Ollama-tenant trace.
  • Did not touch the person-facing tenant-session route (routes.ts) or its own currentVersion/model handling — out of scope for this ticket (that surface has no equivalent free-text-from-LLM boundary), so I left it as-is.
  • CL-6491 (bare 500 at wake time for a model that can't resolve) is out of scope here. Confirmed for that lane: InferenceResolutionError is thrown in packages/folded-runs/src/launch.ts inside deployAtHead, and propagates uncaught from wakeByAddress's call to wakeFoldedRun in packages/chat/src/platform-adapter.ts (no catch there today).

Fixes CL-6477. References CL-6480.

…n string parsing

Covers CL-6477 (an invented/unavailable modelPreference must fall back
to the tenant's default rather than create a dead agent) and CL-6480
(currentVersion is a string on the wire, and a successful create must
still report success and complete the invite).
…fix currentVersion parsing

modelPreference is free text a language model supplies at a tool-call
boundary. The create route now checks it against the tenant's
capability inventory (already resolved for toolPackagePins) and, when
it isn't offered, falls back to the tenant's catalog default instead
of baking in a name that can never resolve, surfacing the substitution
as a modelNote the tool relays honestly. The tool's own description
now tells the model to look up a real catalog name or omit the field,
rather than inviting a guess.

Also fixes CL-6480: the client parsed currentVersion as a number, but
the create route serializes a text DB column, so every genuine create
failed this shape check and reported as an error, silently dropping
the follow-on invite. Parses it as a string, matching the API.
…nse shape

This fixture hand-mocked the create-agent-definition response with
currentVersion as a number and no modelNote field — CL-6480's exact
bug, reproduced in a fixture. With the client's schema now requiring
currentVersion as a string and modelNote as string|null, the mocked
response failed to parse and create_agent reported isError: true.
Updated the fixture to match what the real route actually returns.
@TheGreatAxios
TheGreatAxios force-pushed the cl-6477-model-validation branch from a0d2378 to 575fca4 Compare August 21, 2026 11:27
@TheGreatAxios
TheGreatAxios merged commit f769f27 into main Aug 21, 2026
2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6477-model-validation branch August 25, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant