Skip to content

fix(runtime): land permission switches before the next turn's first tool call (#3349) - #3615

Open
chinawch007 wants to merge 12 commits into
apache:mainfrom
chinawch007:fix/permission-mode-next-turn-3349
Open

fix(runtime): land permission switches before the next turn's first tool call (#3349)#3615
chinawch007 wants to merge 12 commits into
apache:mainfrom
chinawch007:fix/permission-mode-next-turn-3349

Conversation

@chinawch007

Copy link
Copy Markdown
Contributor

Summary

Fixes #3349

A permission switch (Auto→Bypass) was not observed by the next turn. Under Goal continuation the switch was rejected with session_busy — the quiescent mutation bailed eagerly whenever any execution claim existed, and claims are near-continuous while a Goal admits successor turns back to back. When a switch did commit mid-turn, tools still acted on a permissionMode frozen at backend build time, so the picker said Bypass while Bash stayed sandboxed and approvals kept prompting. The issue asks for one property in any session, not just Goals: a permission change is observed by the next turn that starts after it, before that turn's first tool call.

What changed

Kernel — execution serialization:

  • New runSessionQueuedQuiescentMutation: a config change closes a per-session admission gate and waits for quiescence — every claim that predates the request has settled and no run is active — before its operation joins the mutation tail. The gate (not the tail) is what new claims observe, so admission mutations a running turn depends on — graph operator provisioning — still pass; waiting never holds a resource the waited-on execution needs, which keeps the queue deadlock-free. Claims only cover admission (a turn's claim settles at run bind), so runs are watched through hasActiveRuns; a run registers before its claim settles, leaving no instant where an in-flight admission is invisible.
  • setPermissionMode, setExecutionBoundaryKind and transitionSessionConfiguration route through it. The eager hasActiveRuns rejections are gone — quiescence is now the kernel's single authority. The wait is scoped to the primary session; descendant activity is rejected at commit time (session_busy) instead of waited on, a truthful failure rather than a potential hang. waiting_for_user still rejects, now also when it appears mid-queue. Behavior change: a switch during an active turn waits (bounded by one turn) instead of rejecting.

Read model — the boundary as the single authority (#1611):

  • ctx.permissionMode is derived live from the durable boundary at every tool dispatch (executionBoundaryDisplayMode plus the shared plan-mode downgrade), so a committed switch reaches the very next tool call without waiting for a backend rebuild; an external boundary falls back to the last known header mode.
  • resolveCollaborationPermissionMode and executionBoundaryMatchesPermissionMode move to core, so the composer (build time) and the tool runtime (dispatch time) share one rule. The matcher now derives structurally: a read-only profile widened by an approved expansion no longer reads as explore, and a custom read-only profile is preserved instead of being reset.
  • The catalog no-op short-circuit requires boundary consistency, so a header/boundary divergence is repaired instead of blessed; externally isolated sessions keep benign no-op updates on the header comparison.
  • Revision guard: each backend generation records the boundary revision it was composed against; ensureActive rebuilds on drift — immediately when idle, or via the invalidation flush once live runs exit. Any write path that skips backend disposal self-heals within one activation.

How this meets the issue's stated goals

  • "Observed by the next turn that starts after it, before that turn's first tool call" — the switch queues behind live execution, commits in the inter-turn gap, and turns admitted after the request wait at the admission gate until it commits; tools then read the boundary live on every call. Both facts the author named are covered at the dispatch site.
  • Not Goal-specific — the fix is session-level. The reporter's plain-session case (switch between two ordinary turns, no Goal) is the primary regression test and asserts exactly what was asked: the successor turn's first tool call sees executionBoundary.kind === 'bypass' and ctx.permissionMode === 'bypass' together.
  • Never silently drop a switch — return a truthful outcome — busy sessions delay the commit (bounded by one turn) instead of rejecting; a turn pausing on an interaction mid-queue rejects explicitly; descendants active at commit reject session_busy.
  • The author's root-cause pointers are each addressed: the eager bail (replaced by queueing), the frozen permissionMode (live derivation), ensureActive reusing a stale generation (revision guard), and the catalog short-circuit that could bless a divergence (consistency check).

Verification

  • Regression tests were written first and verified failing against the pre-change behavior: a gated turn with a mid-turn Auto→Bypass switch (queues, commits in the gap, next turn rebuilt from the committed mode), and the plain no-Goal session from the issue — the successor turn's first tool call sees executionBoundary.kind === 'bypass' and permissionMode === 'bypass' together, the reporter's primary case.

  • A seeded interleaving sweep (100 iterations, fixed seed) alternates widening and narrowing switches across idle, mid-turn, and racing-the-release interleavings, asserting every turn started after a switch resolved observes the committed configuration.

  • External review findings each carry their own regression test: the admission-gate deadlock interleaving, the widened read-only and external-boundary matcher cases, and the interaction-pause rejection.

  • Not run: the full runtime-host suite and the remaining workspace suites, and manual Desktop verification of the picker.

  • npm run format:check — clean (8 formatting nits auto-fixed and folded in)

  • npm run lint — 2304 files, no issues

  • npm run typecheck

  • npm --workspace @maka/runtime run build — clean

  • npm --workspace @maka/runtime run test:dist — 3110 tests, 3097 pass, 13 skip

Root cause

Three cooperating defects: the eager bail in runSessionQuiescentMutation, the build-time freeze of header.permissionMode into the tool context, and a catalog short-circuit that compared only header fields. The fix converges on the boundary as the single read model and moves quiescence authority into the kernel.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: ZCode (Z.ai GLM) authored the implementation, tests, and review fixes; the contributor directed the design, reviewed each finding, and made the rebase decisions.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

…che#3349)

runSessionQuiescentMutation bails with session_busy whenever any execution
claim exists, so a permission switch can never land while a Goal keeps
admitting successor turns back to back. Add runSessionQueuedQuiescentMutation:
it reserves a slot on each session's mutation tail synchronously, then runs
the operation only once the session is at rest — every claim that predated
the reservation has settled, and no run is active.

Claims only cover admission (a turn's claim settles once its run is bound),
so live turns are observed through hasActiveRuns instead; a run registers on
its backend generation before its claim settles, so an in-flight admission is
never invisible to both checks. Claims created after the reservation carry it
in their admission barrier, so neither they nor runs started through them can
appear first; waiting chains run strictly backwards in claim-creation order,
which keeps the queue deadlock-free. Wakeups fire from claim settlement and
run unregistration. Worst-case delay is one turn. The eager variant keeps its
semantics unchanged.

Generated-by: ZCode (Z.ai GLM)
…che#3349)

setPermissionMode, setExecutionBoundaryKind and transitionSessionConfiguration
rejected with session_busy whenever a turn was running or an admission claim
existed, so a switch could never land while a Goal keeps admitting successor
turns back to back. Route commitExecutionResourceTransition through the
kernel's queued quiescent mutation instead: the switch waits out the claims
and runs that predate it, commits in the inter-turn gap, and the successor
turn — admission-barrier-gated on the reserved slot — observes the new
configuration before its first tool call.

The eager hasActiveRuns guards come out (quiescence is now the kernel's
single authority); the waiting_for_user rejections and
relocateSessionWorkspace's fail-fast keep their semantics. Existing
assertions expecting the reject behavior are updated to the queued
semantics, and a regression test drives a gated turn through an Auto→Bypass
switch, asserting the switch commits in the gap and the next turn is rebuilt
from the committed mode.

Generated-by: ZCode (Z.ai GLM)
…che#3349)

ctx.permissionMode was frozen at backend build time from the session
header, so even a committed switch left a running turn's approvals on the
old mode — the half-applied state behind the issue's mixed symptoms.
Derive the mode per tool dispatch from the authoritative boundary
(executionBoundaryDisplayMode), with the plan-mode downgrade applied at
the same point; an external boundary falls back to the last known header
mode. resolveCollaborationPermissionMode moves to core so the composer
(build time) and the tool runtime (dispatch time) share one rule.

Also close the catalog short-circuit amplifier: session.configuration.update
now requires the durable boundary to match the requested mode before
treating the update as a committed no-op, so a header/boundary divergence
is repaired instead of blessed. executionBoundaryMatchesPermissionMode
moves to core next to the display-mode derivation.

Legacy 'execute' is audit-safe: compilePermissionProfile and the
filesystem worker treat it identically to 'ask', and the subagent
snapshot's permissionMode is unused by tool building.

Generated-by: ZCode (Z.ai GLM)
…pache#3349)

Commits 1-3 rely on every config write disposing the backend before it
commits. Keep that convention from staying implicit: each generation now
records the boundary revision it was composed against, and ensureActive
compares it against the store before reuse. On drift with no active runs
the generation is disposed and rebuilt in the same activation; with runs
still live it is only marked for invalidation — the existing flush path
retires it when they exit and the next activation composes fresh. Tools
stay correct during the grace turn: they read the boundary live on every
call. An unreadable boundary leaves the guard dormant; a self-heal test
drives a stray revision bump through both the idle and the live-run
branches.

Generated-by: ZCode (Z.ai GLM)
…sweep (apache#3349)

Complete the verification plan: the plain-session case the reporter asked
for drives an ordinary turn, an idle Auto→Bypass switch, and the next
turn, asserting the successor is composed from the committed mode and
that a probe tool call — resolving the boundary through the session's own
store, the same read a real dispatch performs — sees
executionBoundary.kind === 'bypass' and permissionMode === 'bypass' at
once. A seeded sweep then alternates ask/bypass (both widening and
narrowing, the latter through the shell-run fence) across interleaving
classes — idle, mid-turn, racing the turn's release — asserting the
invariant that every turn started after a switch resolved observes the
committed configuration. Every checkpoint awaits a deterministic event;
the seed is fixed, so failures reproduce.

Together with the earlier suites this closes the matrix: mid-turn,
gap/idle, successor-claim-pending, waiting_for_user, mid-dispatch
boundary flips, and stray revision bumps.

Generated-by: ZCode (Z.ai GLM)
…le awaiting quiescence (apache#3349)

Review finding (P0): the queued quiescent mutation held the session's
mutation tail for its entire wait, and a running turn can depend on an
admission mutation enqueued on its own session's tail — graph operator
provisioning runs on the supervisor's session exactly when the
supervisor's yield tool is waiting on a reconciliation milestone that
needs it. Tail held + run drained + run waiting on the tail = deadlock.

Decouple the two: the mutation now closes a per-session admission gate
(new claims capture it in their admission barrier) and waits for
quiescence WITHOUT the tail, so admission mutations a running turn
depends on still pass. Only after quiescence does the operation join the
tail, still serialized with other mutations; claims created after the
request stay gated until it completes. Session-manager side, the wait is
now scoped to the primary session only — waiting on descendants could
deadlock the same way through gated child claims — and descendant
activity is rejected at commit time instead, restoring the pre-queue
session_busy guard as a truthful failure rather than a hang.

Also corrects the falsified invariant in the doc comment: a running turn
CAN enqueue a mutation on its own session's tail; the kernel regression
test drives exactly that interleaving.

Generated-by: ZCode (Z.ai GLM)
…pache#3349)

Review finding (P1a): executionBoundaryMatchesPermissionMode judged
read-only-ness by profile NAME while the authoritative display mode
judges it structurally (apache#1611). A read-only-named profile widened by an
approved expansion therefore read as explore to the matcher while
presenting as ask — the catalog short-circuit could bless exactly the
profile-level divergence this series set out to repair — and a custom
structurally-read-only profile forced a transition that silently reset
it to the canonical explore profile.

Derive the match from executionBoundaryDisplayMode so both answers come
from one implementation: a widened read-only no longer matches explore
(repaired through the transition instead), custom read-only profiles
match and are preserved, legacy 'execute' never matches (forcing the
transition is the safe direction), and an external boundary stays
unverifiable.

Generated-by: ZCode (Z.ai GLM)
…solated sessions (apache#3349)

Review finding (P1b): the new boundary-consistency condition in the
session.configuration.update short-circuit made an external boundary
(always unverifiable) fail the check on every update, so even a no-op
re-apply went through transitionSessionConfiguration and hit the store's
refusal to move an externally isolated boundary — a regression for
sessions whose configuration had matched.

Skip the boundary comparison when the boundary is external: the header
comparison alone decides the no-op there, as it did before the
divergence repair.

Generated-by: ZCode (Z.ai GLM)
…action (apache#3349)

Review finding (P2): setPermissionMode and setExecutionBoundaryKind only
inspected waiting_for_user at request time. A turn that pauses on an
approval after the switch was queued never settles until the user
answers, so the queued commit waited indefinitely — the D1 'reject while
the user holds a pending decision' semantics degraded into an unbounded
hang in that window.

The quiescence wait now treats an active interaction as busy: it throws
SessionQuiescentMutationBusyError, interaction registration wakes the
waiters so the rejection is timely, and the session-manager wrapper maps
it to the same session_busy outcome transitionSessionConfiguration
already produces for the same condition. Request-time checks stay as
fast-fail; the regression test drives a switch queued behind a gated
turn that then opens a sandbox boundary approval.

Generated-by: ZCode (Z.ai GLM)
Document the boundary-revision stamping race (the stamp may trail the
revision actually composed against — safe direction, one extra rebuild),
the one-read-per-activation cost choice in the revision guard, and the
operation_unavailable cliff a kernel without the queued primitive would
create. Grow the seeded interleaving sweep from 12 to 100 iterations,
closer to the stress volume the plan promised.

Generated-by: ZCode (Z.ai GLM)
…target (apache#3349)

Main removed 'execute' from the permission-mode vocabulary, so the
P1a regression assertion now passes the legacy value through a type
cast: the runtime property it protects — a stale persisted mode never
matches, so it always routes through a transition — is unchanged.

Generated-by: ZCode (Z.ai GLM)

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful admission-gate work and the unusually thorough race coverage. I found one security-timing gap at this exact head:

[P1] Apply permission reductions before waiting for the current turn to finish

runSessionQueuedQuiescentMutation closes admission for future claims but waits until hasActiveRuns(sessionId) is false before it runs the transition (packages/runtime/src/runtime-kernel.ts:578-587,664-703). The narrower boundary, shell-run termination, and backend disposal therefore do not begin until the current turn has fully ended (packages/runtime/src/session-manager.ts:1727-1789).

For a mid-turn Bypass→Auto/Explore request, the durable boundary remains bypass throughout that wait. ToolRuntime now correctly rereads the boundary per dispatch (packages/runtime/src/tool-runtime.ts:1326-1352), but every later tool call in that same turn still reads the old unrestricted boundary, and background shell authority is not terminated yet. This window is not usefully bounded in wall-clock time: the same dispatch path explicitly supports long-running installs, builds, training, and subagent loops, and a turn can issue multiple more tools.

The tests currently encode this gap: session-manager.test.ts:5164-5185 asserts the switch remains unsettled and the old boundary remains until turn 1 ends; the seeded sweep at :5498-5527 only verifies turns begun after the switch promise resolves. The per-dispatch test manually flips a fake boundary between calls, but the production transition cannot commit that flip while a run is active.

Please split widening from tightening. Delayed widening is a UX tradeoff, but tightening should establish a dispatch fence and revoke promptly—either stop the live turn or atomically install the narrower boundary so its next dispatch uses it, with the chosen contract also fencing already-running shell/subagent resources. An integrated Bypass→Ask/Explore regression should have the current turn attempt another write-capable/Bash dispatch after the user request but before terminal completion, and assert that it sees the narrower authority or that the turn was stopped. The successor-turn assertions should remain as a separate invariant.

@zhiiw zhiiw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent second lane at exact head 8ba7c4c907ec01e56035b0ee2e2742f1ec5d7417 (MERGEABLE). I read the mechanism before reading the existing review; converging on the same conclusion independently.

The existing [P1] is mechanically correct — confirmed by my own trace, not by reading it.

The tightening path: setPermissionModerunSessionQueuedQuiescentMutation([sessionId], …) (session-manager.ts), which closes the admission gate immediately but runs the mutation — the durable boundary write included — only after quiescence (runtime-kernel.ts waitForSessionQuiescence loops on hasActiveRuns). So while the current turn is live, store.readExecutionBoundary still returns the old, wider boundary. ToolRuntime does re-read that store per dispatch (the new executionBoundaryDisplayMode derivation), but what it reads has not changed yet — the narrowing takes effect at the first dispatch after turn end, not after the user's request. The queueing machinery is direction-agnostic, so tightening inherits the same deferral as widening: session-manager.test.ts (Auto → Bypass requested mid-turn … the switch commits in the gap as soon as turn 1 settles) encodes this gap as the expected behavior, with expect((await store.readExecutionBoundary(session.id)).kind).toBe('managed') while the turn is still running.

The expensive parts the quiescence protects (backend disposal, descendant shell fencing) justify waiting — but the boundary write itself is a single store mutation, and the per-dispatch reread this PR added is precisely the mechanism that would let an early write take effect at the next tool call. Splitting "commit the narrower boundary now, dispose the old backend at quiescence" would close the window without destabilizing the running turn. Agree with the existing P1's framing: delayed widening is a UX tradeoff, delayed tightening is an authority window.

Checks note: the test check at this head is failure, but the failure is the ASF license-header gate — runtime-kernel-queued-quiescent-mutation.test.ts and tool-runtime-permission-mode.test.ts are missing headers, the job exits before running suites. So no test evidence exists on this head; the header gate failing early means the red X overstates what's known. npm run write:asf-headers should fix it.

No additional findings beyond the existing P1.

中文说明

独立复读了机制,既有 P1 成立:权限收窄请求进入排队静默 mutation,边界写入推迟到当前 turn 结束后的静默期;期间 ToolRuntime 虽然每次派发都重读边界,但读到的还是旧的宽边界。测试把这个窗口编码成了预期行为。另外这个 head 的 test 红是 ASF license header 门禁(两个新测试文件缺头文件注释),测试套件根本没跑——不是测试失败。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Permission mode switch (Auto→Bypass) is not honored by the next turn

3 participants