CL-6484: adversarial model-output catalogue - #223
Merged
Merged
Conversation
Covers each seeded scenario -- malformed tool name, tool-name length boundary, textless tool-only completion, wrong-shaped and truncated tool arguments, refusal, oversized output, hallucinated tool name -- plus sequence() for scripting per-turn replies, and a regression-guard demo for CL-6478's "does the room survive the next turn?" contract.
Adds named, composable ollama.reply scenarios for the specific ways real local models misbehaved last night, each documented with which model produced it and what broke: - malformedToolName() -- CL-6478's flagship case, qwen3.8:27b's \n</parameter leak into a function name - toolNameOfLength(n) -- the 63/64/65-char boundary three shipping tools sit against (encodeToolName's 64-char wire cap) - textlessToolCall() -- a tool-only completion with no text - wrongShapedToolArgs() / truncatedToolArgs() -- valid-but-wrong-shape and unparseable tool arguments - refusal() and oversized() -- a decline and a truncation-exercising blob - hallucinatedToolName() -- round 1's skills_load vs. the real load_skill OllamaToolCall gains an optional rawArguments override so a scripted reply can put a non-JSON string on the wire byte-for-byte, and sequence() scripts one reply per turn so multi-turn contracts (does the room survive the turn after a malformed one?) are directly testable.
Moves the catalogue from the roadmap into the quickstart now that it ships, with the scenario table, the sequence() multi-turn example, and a pointer to the real CL-6478 fix in vendor/intx/hub-sessions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the adversarial output catalogue to
@corbits/mocks/ollama: named, composableollama.replyscenarios reproducing specific, observed ways real local models misbehaved, plus asequence()helper for scripting a reply per turn. Builds on unit 1 (#219) and follows its establishedollama.reply.text/toolCall/toolCallsidiom exactly.Scenarios seeded (each documents which model produced it / what broke)
malformedToolName()-- flagship case.qwen3.8:27bleaked\n</parameterinto a tool-call function name (CL-6478). Reproduces the exact wire value the real fix (sanitizeToolNameForPersistenceinvendor/intx/hub-sessions/src/sanitize-tool-name.ts) already has a regression test for.toolNameOfLength(n)-- a tool-call name of exactlyncharacters; call with 63/64/65 to prove the boundary three shipping tools sit at against the 64-char wire capencodeToolNameenforces.textlessToolCall(name, args?)-- aninference.donewith no text, only a tool call -- a tool-only round, not an empty reply.wrongShapedToolArgs(name, args)-- valid JSON that doesn't match the declared schema.truncatedToolArgs(name, rawArguments)-- arguments that are NOT valid JSON, reaching the wire byte-for-byte (newOllamaToolCall.rawArgumentsoverride, bypassingJSON.stringify).refusal(text?)-- the model declines.oversized(approxChars?)-- a large text blob,finishReason: "length".hallucinatedToolName()-- round 1'sskills_loadvs. the realload_skill.sequence([...])scripts one reply per turn (repeats the last once exhausted) so a multi-turn contract like "turn 1 malformed, turn 2 normal" is a one-lineonChathandler.CL-6478 regression guard
src/ollama/cl-6478-demo.test.tsdemonstrates the shape: a turn assembler that persistsmalformedToolName()'s broken name unchanged carries the fragment straight into the next turn's history (the bug), versus a sanitized history where the room survives with a normal turn-2 reply. This is demonstrated at the mock layer only -- it does not wire into the real@workbench/hub-sessionsturn assembler (vendored, out of scope for this unit's timebox). That guard already exists for the persistence step itself invendor/intx/hub-sessions/src/sanitize-tool-name.test.ts; an equivalent test through the real turn assembler is a follow-up.Not verified
oversized()at very large sizes.toolNameOfLength's generated names are synthetic (namespaced_tool_probe_...), not the actual encoded formencodeToolNameproduces -- this exercises the wire-level character-count boundary, not the escape-scheme internals.CL-6484 stays open -- OpenAI-compatible and Anthropic provider mocks remain.
Test plan
bun testinpackages/mocks-- 31 pass, 0 failbunx tsc --noEmitinpackages/mocks-- cleanbun run lintfrom repo root -- 0 errors (12 pre-existing warnings in unrelated files)bun run check:packagesfrom repo root --check:packages: okbun run check(broader test suite); not booted the product stack per instructionsDO NOT MERGE -- a peer reviewer should merge this.