perf(tools): scan only the chunk for a collecting tool call's end tag - #625
Open
spokvulcan wants to merge 1 commit into
Open
spokvulcan wants to merge 1 commit into
spokvulcan wants to merge 1 commit into
Conversation
`ToolCallProcessor` checked the whole buffered call for its end tag on every chunk while collecting a tagged call, on the native path and on the cross-dialect recovery path that declared tools enable, so a call costs time quadratic in its length: seconds for a call of ten thousand tokens, paid once on the live stream and again wherever the stream is replayed. Only the text a chunk appends can complete the end tag — an earlier occurrence was already in the buffer when the previous chunk was scanned — so both scanners now check the appended bytes plus the tag's overlap with what preceded them, and run the full structural frame scan only when that window holds the tag. The chunk that opens the frame still scans the whole buffer, which then holds at most the start tag and that chunk. Tests: a 40k-character call streamed one character at a time parses in linear time, with and without declared tools; the end tag is found when it is split across chunks, when it arrives in the chunk that completes the start tag, and when another call follows it in the same chunk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Fixes #624.
While collecting a tagged tool call,
ToolCallProcessorand the text recovery scanner searched the whole buffered call for its closing tag on every chunk, which is quadratic in the call's length: a 13k-tokenwritecall spent seconds in that scan alone. Only the text a chunk appends can complete the closing tag, because an earlier occurrence was already in the buffer when the previous chunk was scanned. Both scanners now check the appended bytes plus the tag's overlap with what preceded them, through one helper onToolCallFrameScanner, and run the structural frame scan only when that window holds the tag. The chunk that opens the frame still scans the whole buffer, which then holds at most the start tag and that chunk. Five tests inToolCallProcessorLongCallTests: a 40k-character call streamed one character at a time parses in linear time, with and without declared tools, and the closing tag is found when it is split across chunks, when it arrives in the chunk that completes the start tag, and when another call follows it in the same chunk.Checklist
Put an
xin the boxes that apply.pre-commit run --all-filesto format my code / installed pre-commit prior to committing changesAI usage
accurately describes the code changes.