CL-6477: validate the model create_agent asks for - #222
Merged
Merged
Conversation
…n string parsing Covers CL-6477 (an invented/unavailable modelPreference must fall back to the tenant's default rather than create a dead agent) and CL-6480 (currentVersion is a string on the wire, and a successful create must still report success and complete the invite).
…fix currentVersion parsing modelPreference is free text a language model supplies at a tool-call boundary. The create route now checks it against the tenant's capability inventory (already resolved for toolPackagePins) and, when it isn't offered, falls back to the tenant's catalog default instead of baking in a name that can never resolve, surfacing the substitution as a modelNote the tool relays honestly. The tool's own description now tells the model to look up a real catalog name or omit the field, rather than inviting a guess. Also fixes CL-6480: the client parsed currentVersion as a number, but the create route serializes a text DB column, so every genuine create failed this shape check and reported as an error, silently dropping the follow-on invite. Parses it as a string, matching the API.
…nse shape This fixture hand-mocked the create-agent-definition response with currentVersion as a number and no modelNote field — CL-6480's exact bug, reproduced in a fixture. With the client's schema now requiring currentVersion as a string and modelNote as string|null, the mocked response failed to parse and create_agent reported isError: true. Updated the fixture to match what the real route actually returns.
TheGreatAxios
force-pushed
the
cl-6477-model-validation
branch
from
August 21, 2026 11:27
a0d2378 to
575fca4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Myra's
create_agenttool call invents a free-textmodelPreference— observed live as"gpt-4o"in a tenant whose catalog contains only Ollama models. Nothing validated it, so the agent was created pinned to a model that cannot exist, and could never answer.workflow-create-routes.ts) now checks a requestedmodelagainst the tenant's own capability inventory (the same inventory already resolved there fortoolPackagePins— reused, not re-fetched). A model the catalog doesn't offer is never baked in: it falls back to the tenant's catalog default (or leaves the definition modelless if there is none), and the response carries amodelNoteexplaining the substitution. A model the catalog does offer is used exactly as asked, no fallback consulted.create_agent's own tool schema now tells the calling model to look up a real catalog name (vialist_agents/a models tool) or omit the field to inherit the default, rather than inviting a guess — the tool result also relaysmodelNotehonestly so the calling model can tell the user what happened.currentVersionas"number", but the create route serializesworkflow_definition.current_version— atextcolumn — verbatim, always a string on the wire. Every genuine create was therefore failing this shape check, reporting a successful creation as an error and silently dropping the follow-on invite. Now parsed as"string", matching the real API.Validation shape chosen, and why
Validation lives server-side in
@corbits/agent-directory'sworkflow-create-routes.ts, against the sameCapabilityInventoryProvideralready wired there fortoolPackagePins(models: [{ canonicalName }], backed bylistMyraModels). This is the actual trust boundary the tool call crosses, matches the existing fail-closed pattern for tool-package pins, and needed no new plumbing. Preferred fallback over hard error, since the ticket asks for a working agent over a rejected call — a bad model name degrades gracefully to the tenant default instead of blocking creation outright.Known gap, not fixed here
listMyraModels(the source behindcapabilityInventory.models) is a flat per-tenant query, while real launch-time model resolution also inherits offerings from ancestor tenants (provenance: "set-here" | "inherited"). So amodelPreferencenaming an inherited model — one that would actually resolve fine at launch — could be wrongly treated as "not in catalog" here and substituted unnecessarily. The failure direction is safe (never a wrongful accept of a truly-unresolvable model, only a possible unnecessary fallback), and the asymmetry is pre-existing — the samecapabilityInventoryis already used identically fortoolPackagePinstoday — but it deserves its own ticket rather than being fixed as a drive-by here.Fixed after first review pass
CI caught a fixture break I missed in scoped testing:
workflows/assistant/test/manager-tools-scenario.test.tshand-mocked the create-agent response withcurrentVersion: 1(a number) and nomodelNotefield — CL-6480's exact bug, reproduced in a fixture. That made the mocked response fail the client's own (now-correct) schema, socreate_agentreportedisError: trueinside the fixture. Updated the fixture tocurrentVersion: "1", modelNote: null, matching what the real route returns. Re-grepped the whole repo for every fixture touchingworkflow-agent-directory/definitions(three total, all now consistent) rather than trusting the scoped package list alone.Not verified
bun test+tsc --noEmitonagent-directory,agent-directory-tools,workflows/assistant, andapps/hub, plus rootbun run lint, not an end-to-end Ollama-tenant trace.routes.ts) or its owncurrentVersion/model handling — out of scope for this ticket (that surface has no equivalent free-text-from-LLM boundary), so I left it as-is.InferenceResolutionErroris thrown inpackages/folded-runs/src/launch.tsinsidedeployAtHead, and propagates uncaught fromwakeByAddress's call towakeFoldedRuninpackages/chat/src/platform-adapter.ts(no catch there today).Fixes CL-6477. References CL-6480.