Skip to content

CL-6204: fire budgeted compaction, detect context overflow before send - #254

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6204-compaction
Aug 21, 2026
Merged

TheGreatAxios merged 2 commits into
mainfrom
cl-6204-compaction

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • Adds a per-model context-window budget (context-budget.ts), sized from InferenceSource.quirks via the same numCtx shape @corbits/ollama-adapter resolves for a live request, falling back to a conservative constant for sources with no quirks.
  • Adds a summarize-budgeted-turns compactor that folds history against that budget instead of a fixed turn count (deliberately lossy — a bounded recap of the active window, not a durable record).
  • WorkbenchDirector now actually fires it, at the one point in the reactor's event flow where caps.compact cannot stall a reply: a non-final tool.done inside a multi-call batch. @intx/agent's createAgent only threads env.compactors into the reactor (no ContextTransform seam exists at that layer — verified against agent.js), and compact cannot pair with infer in the same cycle with no follow-up event, so every other cycle risks stalling a reply if it compacts instead of inferring; compaction there is deferred to the next safe point.
  • Before any decision that would infer, the director checks the budget's hard limit. Past it, it replies with the same "exceeded the model's context limit" message hosted providers produce via inference.error's context_overflow category, instead of sending a request Ollama would silently truncate (HTTP 200, no error).

Why

@intx/inference's reactor resends the full turn history every inference cycle. Hosted providers fail loud on overflow (context_overflow classification → friendly reply); Ollama returns 200 and truncates server-side, so a long local-model conversation degrades into incoherence with no signal. The existing summarize-older-turns compactor was registered but never invoked by any director (CL-6204's original gap).

Scope note

The owner's broader design for this ticket also calls for: two distinct compactor strategies (conversational vs. orchestration, the latter tracking cross-thread/agent status rather than prose), and a compaction-time sweep that extracts durable facts to @corbits/memory-tools before discarding older turns (an extra inference call per compaction). Both are real, larger pieces of work — the memory-sweep needs an inference handle threaded into the compactor seam (ContextStrategy.apply doesn't have one today), and the orchestration variant needs the whole-chat-manager surface identified and scoped. Neither is half-wired here; this PR ships the budget/overflow foundation both would sit on top of.

Confirmed while reviewing: the assistant workflow (workflows/assistant) already pins @corbits/memory-tools and its system prompt directs the model to memory_search/memory_add/memory_list for anything worth recalling — the chat host (packages/chat's platform-adapter.ts, via @corbits/agent-lifecycle) runs through this same workflow, so retrieval is reachable today for the channel-agent surface. It is on-demand only (no automatic per-turn injection), matching the owner's stated design.

Test plan

  • bun test in apps/sidecar — 38 pass, 0 fail
  • tsc --noEmit in apps/sidecar — clean
  • Red/green covered: a short conversation stays untouched; a conversation past budget folds rather than resending whole; a minimum verbatim tail always survives; a prompt whose history exceeds the hard limit gets an honest reply instead of an infer that would silently truncate

Covers: a short conversation staying untouched, a conversation past
budget getting folded, a minimum verbatim tail always surviving, the
per-model numCtx budget resolver, and the workbench director's two new
behaviors (honest context_overflow reply, safe-point compact firing on
a non-final tool.done in a multi-call batch).
…send

@intx/inference's reactor resends the full turn history on every
inference cycle and Ollama silently truncates an oversized request
(HTTP 200, no error) instead of classifying context_overflow the way
hosted providers do, so a long local-model conversation degrades into
incoherence with no signal.

Adds a per-model context-window budget (apps/sidecar/.../context-budget.ts,
sized from InferenceSource.quirks via the same numCtx shape
@corbits/ollama-adapter resolves, falling back to a conservative
constant for sources with no quirks) and a new summarize-budgeted-turns
Compactor that folds on that budget instead of a fixed turn count.

WorkbenchDirector now fires it for real: @intx/agent's createAgent only
threads env.compactors into the reactor (no ContextTransform seam
exists at that layer), and the reactor's compact action cannot pair
with infer in the same cycle with no follow-up event afterward, so most
cycles can't safely compact without stalling a reply. The one point
that's always safe is a non-final tool.done inside a multi-call batch:
the batch's remaining tool.done events are already enqueued and will
still drive the eventual re-infer, so compacting there never stalls
anything. Compaction elsewhere is deferred to the next safe point
rather than forced.

Before any decision that would infer, the director also checks the
budget's hard limit (no headroom): past it, it replies with the same
"exceeded the model's context limit" message hosted providers produce
via inference.error's context_overflow category, instead of sending a
request Ollama would truncate silently.

Both compactors stay deliberately lossy folds -- a bounded recap of the
active working window, not a durable record. The assistant workflow
already pins @corbits/memory-tools and its system prompt directs the
model to memory_search/memory_add for anything worth recalling past
that window.
@TheGreatAxios
TheGreatAxios merged commit daaa833 into main Aug 21, 2026
2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6204-compaction branch August 25, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant