Skip to content

feat(agent): allow nextTurnParams to set toolChoice - #114

Merged
mattapperson merged 2 commits into
mainfrom
feat/next-turn-tool-choice
Aug 20, 2026
Merged

feat(agent): allow nextTurnParams to set toolChoice#114
mattapperson merged 2 commits into
mainfrom
feat/next-turn-tool-choice

Conversation

@mattapperson

@mattapperson mattapperson commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What

Adds toolChoice to NextTurnParamsContext, so a tool can change which tools the model may call on the following turn.

Three lines of behaviour change:

  • toolChoice on NextTurnParamsContext (tool-types.ts)
  • buildNextTurnParamsContext reads it off the request
  • 'toolChoice' added to the isValidNextTurnParamKey allowlist — unknown keys are silently dropped, so without this the hook is a no-op

toolChoice is already a modelled ResponsesRequest field, so applyNextTurnParamsToRequest carries it through unchanged. No serialization work, no SDK regeneration.

Why

This is what a tool-search tool needs. Declare every tool up front, keep the not-yet-needed ones out of reach behind an allowed_tools choice, and widen that choice as the model finds what it wants.

The alternative — rewriting tools between turns — changes the request prefix and loses the provider's prompt cache, which is the whole reason to withhold tools rather than send them all. Keeping tools byte-identical and moving only tool_choice preserves the cache across a search.

A test asserts exactly that: expect(result.tools).toBe(tools) — same reference, untouched.

Notes for review

  • Additive. No existing key changes meaning, and activeTools is untouched.
  • Callers should append to the current value rather than rebuild it: dropping a name revokes a tool the model may already have used, and reordering churns the request for nothing. Documented on the type.
  • pnpm lint and pnpm typecheck pass; 1156 tests pass, no type errors.

Open in Devin Review

A tool can now return a new `toolChoice` for the following turn, which lets a
tool-search tool widen an `allowed_tools` set as it discovers tools. The
alternative — rewriting `tools` between turns — changes the request prefix and
loses the provider's prompt cache, which is the reason to withhold tools in the
first place.

`toolChoice` is already a modelled `ResponsesRequest` field, so it needed only
the context entry and the key allowlist; `applyNextTurnParamsToRequest` carries
it through unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@mattapperson

Copy link
Copy Markdown
Collaborator Author

e2e-tests failure is not from this change

Every other check is green. The e2e job fails on 6 live-provider tests in call-model-state.test.ts and call-model-tools.test.ts, all with BadRequestResponseError: Provider returned error.

The same failure is on #110 (lukeparke/dev-817-drop-preliminary-results), whose CI ran ~90 minutes before this branch's — identical signature:

  • same two test files
  • same Tests 6 failed | 106 passed | 2 skipped
  • same BadRequestResponseError: Provider returned error
  • e2e-tests the only failing job on both

Two unrelated branches failing identically points upstream, not at either diff. I re-ran the failed job once and it reproduced, so it is not a one-off flake either.

This diff is additive — one optional key on a context type plus its allowlist entry — and touches nothing in state persistence or parallel tool execution.

`applyNextTurnParams` merged the computed choice into `resolvedRequest`, but
`makeFollowupRequest` then ran `applyForcedToolChoicePolicy`, which rebuilds
`toolChoice` purely from `configuredToolChoice`. The tool's value was dropped
before the request was sent, so the hook was a no-op in the main loop — only
the separate `tool-orchestrator` path preserved it.

Route a computed `toolChoice` through `applyResolvedForcedToolChoicePolicy` so
it becomes the new configured choice, re-stamping the forced-choice consumption
key alongside it and keeping relaxation consistent on later turns.

The existing tests exercised `applyNextTurnParamsToRequest` in isolation and so
could not see this. Adds a loop-level test that asserts against the dispatched
request; it fails without the fix.

Co-Authored-By: Claude <noreply@anthropic.com>
@mattapperson

Copy link
Copy Markdown
Collaborator Author

Fixed in fae208c — the Devin Review finding was correct and this was a real no-op.

applyNextTurnParams merged the computed choice into resolvedRequest, but makeFollowupRequest then ran applyForcedToolChoicePolicy, which rebuilds toolChoice purely from configuredToolChoice (model-result.ts:4989-4993). The tool's value was discarded before dispatch, so the hook only worked on the separate tool-orchestrator path.

A computed toolChoice now routes through applyResolvedForcedToolChoicePolicy, so it becomes the new configured choice and its forced-choice consumption key is re-stamped alongside it — keeping the DEV-785 relaxation behaviour consistent on later turns.

The root cause of the miss was test placement: the original tests exercised applyNextTurnParamsToRequest in isolation, which cannot see a value being overwritten downstream. Added tests/unit/next-turn-tool-choice-loop.test.ts, which drives the real callModel loop and asserts against the dispatched request. I confirmed it fails without the fix:

× reaches the dispatched follow-up request
AssertionError: expected { type: 'allowed_tools', …(2) } to deeply equal { type: 'allowed_tools', …(2) }

It also pins the property this feature exists for — tools stays byte-identical across turns while only the choice widens, so the provider's prompt-cache prefix survives.

1159 tests pass, lint and typecheck clean.

@mattapperson

Copy link
Copy Markdown
Collaborator Author

Status: review resolved, e2e-tests is a repo-wide break

Review: the Devin finding is fixed in fae208c and the bot has marked its own thread ✅ Resolved. No open threads.

Checks: lint, typecheck, unit-tests, package-boundaries, structural-gate all green. e2e-tests is red — and it is red on main too.

main's scheduled CI today failed with exactly this shape:

run date e2e-tests everything else
main 2026-08-19 06:09
main 2026-08-20 06:09
#110 2026-08-20 01:49
#114 (this) 2026-08-20 03:17 & 13:00

Same 6 tests every time — 3 in call-model-state.test.ts, 3 in call-model-tools.test.ts, all BadRequestResponseError: Provider returned error, all on openai/gpt-4o-mini multi-turn / parallel-tool paths, always Tests 6 failed | 106 passed | 2 skipped.

Since a main commit that predates this branch reproduces it, nothing in this PR can be the cause. It broke somewhere between 2026-08-19 06:09 and 2026-08-20 01:49, and it is currently blocking every PR in the repo.

I have deliberately not tried to fix it here: it is unrelated to a two-line context-type addition, it needs a live API key to iterate on, and it wants its own PR with whoever owns the e2e suite. Happy to pick it up separately if useful.

main has no required status checks (its ruleset is deletion / non-fast-forward / pull-request-with-0-approvals), so this PR reports MERGEABLE; UNSTABLE here only reflects the non-required e2e job.

@mattapperson
mattapperson merged commit 66d7232 into main Aug 20, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant