CL-6204: fire budgeted compaction, detect context overflow before send - #254
Merged
Merged
Conversation
Covers: a short conversation staying untouched, a conversation past budget getting folded, a minimum verbatim tail always surviving, the per-model numCtx budget resolver, and the workbench director's two new behaviors (honest context_overflow reply, safe-point compact firing on a non-final tool.done in a multi-call batch).
…send @intx/inference's reactor resends the full turn history on every inference cycle and Ollama silently truncates an oversized request (HTTP 200, no error) instead of classifying context_overflow the way hosted providers do, so a long local-model conversation degrades into incoherence with no signal. Adds a per-model context-window budget (apps/sidecar/.../context-budget.ts, sized from InferenceSource.quirks via the same numCtx shape @corbits/ollama-adapter resolves, falling back to a conservative constant for sources with no quirks) and a new summarize-budgeted-turns Compactor that folds on that budget instead of a fixed turn count. WorkbenchDirector now fires it for real: @intx/agent's createAgent only threads env.compactors into the reactor (no ContextTransform seam exists at that layer), and the reactor's compact action cannot pair with infer in the same cycle with no follow-up event afterward, so most cycles can't safely compact without stalling a reply. The one point that's always safe is a non-final tool.done inside a multi-call batch: the batch's remaining tool.done events are already enqueued and will still drive the eventual re-infer, so compacting there never stalls anything. Compaction elsewhere is deferred to the next safe point rather than forced. Before any decision that would infer, the director also checks the budget's hard limit (no headroom): past it, it replies with the same "exceeded the model's context limit" message hosted providers produce via inference.error's context_overflow category, instead of sending a request Ollama would truncate silently. Both compactors stay deliberately lossy folds -- a bounded recap of the active working window, not a durable record. The assistant workflow already pins @corbits/memory-tools and its system prompt directs the model to memory_search/memory_add for anything worth recalling past that window.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
context-budget.ts), sized fromInferenceSource.quirksvia the samenumCtxshape@corbits/ollama-adapterresolves for a live request, falling back to a conservative constant for sources with no quirks.summarize-budgeted-turnscompactor that folds history against that budget instead of a fixed turn count (deliberately lossy — a bounded recap of the active window, not a durable record).WorkbenchDirectornow actually fires it, at the one point in the reactor's event flow wherecaps.compactcannot stall a reply: a non-finaltool.doneinside a multi-call batch.@intx/agent'screateAgentonly threadsenv.compactorsinto the reactor (noContextTransformseam exists at that layer — verified againstagent.js), andcompactcannot pair withinferin the same cycle with no follow-up event, so every other cycle risks stalling a reply if it compacts instead of inferring; compaction there is deferred to the next safe point.infer, the director checks the budget's hard limit. Past it, it replies with the same "exceeded the model's context limit" message hosted providers produce viainference.error'scontext_overflowcategory, instead of sending a request Ollama would silently truncate (HTTP 200, no error).Why
@intx/inference's reactor resends the full turn history every inference cycle. Hosted providers fail loud on overflow (context_overflowclassification → friendly reply); Ollama returns 200 and truncates server-side, so a long local-model conversation degrades into incoherence with no signal. The existingsummarize-older-turnscompactor was registered but never invoked by any director (CL-6204's original gap).Scope note
The owner's broader design for this ticket also calls for: two distinct compactor strategies (
conversationalvs.orchestration, the latter tracking cross-thread/agent status rather than prose), and a compaction-time sweep that extracts durable facts to@corbits/memory-toolsbefore discarding older turns (an extra inference call per compaction). Both are real, larger pieces of work — the memory-sweep needs an inference handle threaded into the compactor seam (ContextStrategy.applydoesn't have one today), and the orchestration variant needs the whole-chat-manager surface identified and scoped. Neither is half-wired here; this PR ships the budget/overflow foundation both would sit on top of.Confirmed while reviewing: the assistant workflow (
workflows/assistant) already pins@corbits/memory-toolsand its system prompt directs the model tomemory_search/memory_add/memory_listfor anything worth recalling — the chat host (packages/chat'splatform-adapter.ts, via@corbits/agent-lifecycle) runs through this same workflow, so retrieval is reachable today for the channel-agent surface. It is on-demand only (no automatic per-turn injection), matching the owner's stated design.Test plan
bun testinapps/sidecar— 38 pass, 0 failtsc --noEmitinapps/sidecar— clean