Skip to content

feat(extract): make the extract-phase call timeout configurable - #59

Merged
tybot02 merged 1 commit into
mainfrom
feat/extract-timeout-env
Aug 26, 2026
Merged

tybot02 merged 1 commit into
mainfrom
feat/extract-timeout-env

Conversation

@lucasmaan

Copy link
Copy Markdown
Contributor

Problem

The extract phase's per-call LLM timeout was a source constant at 180s. That figure
is sized for a hosted model's generation speed, and the reason it is tighter than
the write phase's — "extract calls have predictable size (≤16K max_tokens)" — bounds
the prompt, not the duration.

A model served on localhost can be an order of magnitude slower. A local 12B model
exhausted all three attempts on a 4386-character prompt, which is a small document by
a real corpus's standards. The only way past it was to edit the source.

Extract is also where a slow model fails earliest, because it runs on every document.
So the phase that most needed the knob was the one that did not have it.

Change

KB_AI_EXTRACT_TIMEOUT_S mirrors the KB_AI_WRITE_TIMEOUT_S added in #58,
deliberately down to the details, so an operator who has learned one knob has learned
both:

  • read per call rather than at import, so setting it need not happen before kb_ai is
    imported;
  • an unusable value (zero, negative, non-numeric, NaN, infinite) is reported once on
    stderr and ignored, rather than letting a typo in an env var decide how a compile
    behaves — zero would fail every call instantly, and infinite would silently remove
    the cap the default exists to impose;
  • honoured verbatim past DEFAULT_CLIENT_TIMEOUT_S, since the client timeout is a
    default and not a ceiling.

The defaults are unchanged, so the existing
_EXTRACT_CALL_TIMEOUT_S < _WRITE_CALL_TIMEOUT_S < DEFAULT_CLIENT_TIMEOUT_S ordering
invariant still holds.

One place the phases genuinely differ

Documented rather than mirrored: extract retries in two places. _phase2_with_retry
re-dispatches its entire K-call set once on any failure, on top of the LLM layer's two
timeout retries. A hung phase-2 call therefore costs 6*timeout+60s to discover, not
3*timeout+30s — about 1140s at the 180s default and 91 minutes at 900s. Both READMEs
size extract against six attempts rather than three.

Verification

  • 13 new tests, written failing first. The end-to-end one failed with
    assert 180.0 == 1800.0, which is the defect itself: the env var was ignored.
  • 2 further tests drive both decorated entry points (extract_knowledge_summarized
    and extract_knowledge_chunked) rather than the decorator in isolation, because the
    dispatch between them depends on document size and transcript detection — a knob
    reaching one path and not the other would show up as a timeout only some documents
    respect.
  • 236 passed across the extract and merge test files; 1683 passed in the full suite.
  • kb_ai.core.extract at 91% statement coverage, with the new code fully covered.
  • Five mutations of the implementation (env ignored, inf accepted, zero/negative
    accepted, warn-once defeated, decorator reverted to the frozen constant) all killed,
    each with a green control and reverse control.

🤖 Generated with Claude Code

The 180s per-call extract timeout was a source constant. It is sized for a
hosted model's generation speed, and "extract calls have predictable size" --
the reason it is tighter than the write phase's -- bounds the prompt, not the
duration. A model served on localhost can be an order of magnitude slower, at
which point extract fails on documents a hosted model handles without trouble:
a local 12B model exhausted all three attempts on a 4386-character prompt,
which is a small document by a real corpus's standards.

Extract is also where a slow model fails earliest, because it runs on every
document. So the phase that most needed the knob was the one that did not have
it, and the only way past a timeout was to edit the source.

KB_AI_EXTRACT_TIMEOUT_S mirrors KB_AI_WRITE_TIMEOUT_S deliberately, down to
the fallback and the warn-once behaviour, so an operator who has learned one
knob has learned both. It is read per call rather than at import, so setting it
does not have to happen before kb_ai is imported. A value that cannot serve as
a timeout -- zero, negative, non-numeric, NaN, or infinite -- is reported once
on stderr and ignored: zero would fail every call instantly and infinite would
silently remove the cap the default exists to impose, and failing a whole
compile over a typo in an env var is the worse outcome.

One way the two phases genuinely differ is documented rather than mirrored:
extract retries in two places, because _phase2_with_retry re-dispatches its
entire K-call set once on top of the LLM layer's two timeout retries. A hung
phase-2 call therefore costs 6*timeout+60s to discover, not 3*timeout+30s, so
the README sizes extract against six attempts rather than three.

The autouse fixture that kept KB_AI_WRITE_TIMEOUT_S out of the suite's view now
covers both variables, so the operator these knobs exist for does not get a red
suite for a reason unrelated to their change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tybot02
tybot02 merged commit 7a90f04 into main Aug 26, 2026
10 checks passed
@tybot02
tybot02 deleted the feat/extract-timeout-env branch August 26, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants