feat: batched write, reasoning-aware sizing, and section-level merge

[wangdahoo]

Problem

A 500-document distillation run took 9.97 hours wall clock. The RCA attributed
~82% of LLM time to the write phase: documents ran through per-doc pipeline
calls at 4-way document concurrency, every pipeline call rebuilt the KB indexes
(534 rebuilds per run), and any article over 30 KB took the diff path with no
way to tune that threshold per deployment.

Follow-up replay analysis on the batched path showed the remaining cost was
generation economics, not structure: ~50% of article writes hit the 16,384-token
truncation rung (one replay round discarded ~1.1M completion tokens), the
restart ladder threw away every truncated partial, the diff path applied
patches to a truncated view of large articles (silently dropping content), and
extract spent roughly half its output on pre-JSON prose.

Change

Six commits on this branch: two feature commits and four review-hardening
fixes.

1. Batched write, debounced indexing, tunable merge threshold

Batched write

Per-task pipeline items now flow through a batcher that groups same-article
items into one write LLM call (max_items=16, 2 s flush window, bounded
concurrent calls). The concurrent-call cap is runtime-tunable via
KAAS_WORKER_PIPELINE_BATCH_MAX_INFLIGHT. A per-article lock serializes
same-article writes across concurrent calls, closing a lost-update window that
the old one-call-per-doc path never had. Rollback: pipeline_batch_max_items=1.

Debounced index rebuilds

Pipeline calls stop rebuilding the indexes per call (rebuild_index=false);
completions mark them dirty and one rebuild follows index_debounce_sec after
the last completion, forced when dirty state ages past index_max_stale_sec.
Measured ~4x fewer rebuilds under load. The refresher runs on its own circuit
breaker, so a deterministically broken index cannot open the breaker gating
extract/pipeline. Rollback: index_debounce_sec=0.

Tunable merge threshold

The 30 KB full-rewrite threshold is configurable (KB_MERGE_FULL_REWRITE_LIMIT,
warns and falls back on invalid values), and a degraded diff result now falls
back to full rewrite when the rewrite fits the prompt budget instead of
silently dropping content.

Config plumbing

extract_workers is renamed to document_workers with a deprecation shim. New
tuning keys get env overrides through a shared warn-and-fallback envInt
helper — invalid values never abort startup — and validate() rejects resolved
values that cannot work (workers < 1, inflight < 1 with batching on).

2. Reasoning-aware sizing, continuation mode, section merge, prompt tightening

Truncation handling (py/src/kb_ai/llm/)

• Length-cut finish_reason values are matched case-insensitively across
  gateway spellings (length / max_tokens / max_output_tokens); restarts
  emit alerts carrying the discarded token count.
• New continuation mode (continue_on_length=True) keeps a truncated partial
  as an assistant message and re-calls for the continuation, deduplicating seam
  overlap on raw text — plain-text write paths stop discarding completed work.
• estimate_max_tokens() sizes the first max_tokens rung from the expected
  output length plus a 16,384-token reasoning headroom: reasoning-style models
  burn 8-15K invisible tokens per call before any text appears, and billing is
  per generated token, so over-granting is free while under-granting pays twice
  (a restart discards, a continuation re-reasons).
• reasoning_effort passthrough (env knob) with graceful 400 degrade, and a
  retry for reasoning-only empty completions (finish_reason=stop, empty body).

Section-level merge (py/src/kb_ai/core/merge.py)

• Articles >= 12 KB with >= 3 sections route changed sections through a
  section-merge router (sized, semaphore-bounded) and rewrite only the affected
  sections; a cost guard chooses full rewrite when more would change than a
  rewrite costs, and a section -> diff -> full-rewrite fallback chain bounds
  the worst case. Cost scales with changed sections, not article size.
  Rollback: KB_MERGE_SECTION_MODE=off.
• Diff patches are applied to the full on-disk article (previously to a
  truncated view) and patch responses are validated before application.
• Full-rewrite and new-article call sites size their first rung from the
  expected output and use continuation mode.

Extract prompt tightening

• Prompts suppress pre-JSON prose and hard-require the reply to end with the
  JSON object; measured -44% completion tokens per call (9,349 -> 5,235) on the
  replay corpus, with sampled extractions clean.
• Multi-chunk extraction retries only the failed phase-2 groups, and summarize
  calls are sized.

Observability

The orchestrator logs per-batch write/classify LLM call counts, LLM time, and
completion tokens — the lines the replay harness parses for per-phase cost
accounting.

3. Review hardening (four follow-up commits)

Four review rounds on this PR surfaced issues in the new code itself; every
finding was verified against the tree before fixing, with tests added per fix:

• Data safety
  • Section merge degrades to the legacy chain on duplicate ## headings:
    reassembly keyed rewritten bodies by heading text, so a duplicate would
    have replaced both occurrences with one rewrite and silently deleted the
    other's content. The router also remaps (with a diagnostic alert)
    a new-section proposal that names an existing heading — dropping it
    silently unmerged its material while the task Acked "merged".
  • A degraded diff with no fitting rewrite raises instead of returning the
    unchanged article: the error lands in the task's Ack payload / the compile
    error list, and the frontmatter never records an unmerged source.
  • The heading-echo strip matches the first line exactly (normalized), so
    an echo cannot eat a shared-prefix body or slip through with trailing
    whitespace.
• Batch deadline made real
  • The hardcoded 2,400 s batch deadline is configurable
    (pipeline_batch_deadline_sec, env KAAS_WORKER_PIPELINE_BATCH_DEADLINE_SEC;
    values < 1 are rejected — zero would silently lift every bound at once).
  • Every LLM attempt's HTTP timeout is clamped to the deadline remainder and
    recomputed per attempt, so a batch cannot outrun its deadline by a
    write-timeout ladder while an orphaned daemon request keeps writing. The
    Go-side watchdog timeout then only catches a daemon that cannot answer at
    all — previously nothing bounded a wedged call, and shutdown could hang.
• Degradation correctness
  • EmptyCompletionError / OutputTruncatedError / DeadlineExceededError
    flow through the designed fallbacks instead of escaping to a generic
    per-article error (EmptyCompletionError was not even exported before).
    Deadline exhaustion propagates as itself rather than being relabelled
    "diff result unparsable".
  • A 400 is blamed on reasoning_effort only when the error body names the
    param or a without-param probe succeeds outright; flaky-but-refusing
    gateways converge on their first call instead of failing every request.
• Operational isolation
  • ErrBatcherClosed abandons the task like an open breaker: a shutdown
    racing Submit made no call and burns no attempt.
  • An explicitly invalid document_workers reaches validation instead of
    being masked by the deprecated alias; the seam-dedup window covers
    realistic re-emitted blocks (2,048 chars).

Deliberately left as follow-ups (not regressions vs main): attempt-burn
correlation on batch call failures (each task's burn rate is unchanged from
the direct path), and the bounded seam-dedup heuristic (overlaps beyond 2,048
chars pass through).

Verification

• Replay benchmarks on a 150-doc production corpus (deepseek-v4-flash; full
  reports: wangdahoo/kaas#1):
  • Throughput rounds: baseline (no batching) 136.0 min -> 95.3 min at
    inflight=2 (1.43x) -> 58.9 min at inflight=8 (2.31x, +41% LLM work — not
    the shipped default).
  • Post-sprint round C (inflight=2, 150/150 success): extract -44%
    tokens/call, write-side discards -87% (1.10M -> 144K tokens), zero merge
    fallbacks, zero continuation seams across 46 continuation events, list
    coverage +5% vs round A.
  • Honest wall-clock note: round C came out 6% slower than round A (101.1 vs
    95.3 min). The dominant cause — write-call sizing under-provisioning the
    per-call reasoning burn — is what the 16,384 headroom addresses; its
    projected effect (~0.80x round-A total tokens, wall back to ~90 min) has
    not been re-measured end to end yet.
• Review rounds: ~35 new tests across merge / llm / config / worker covering
  the hardening above. Per round, the full Python suite was failure-list
  diffed against a clean-tree run of the same selection (zero new failures;
  the flaky-on-Windows compile suite was compared via intersected stable
  sets), and all four rounds verified zero regressions in earlier fixes.
• Go: internal/config suite green; GOOS=linux go build ./... and go vet
  clean (bridge/daemon.go is Unix-only, so worker/cmd packages are
  cross-compile checked on Windows).
• Python: merge/write/extract/llm suites green; vs the pre-sprint tree +104
  newly passing tests with zero new failures (the 175 Windows-environment
  failures are pre-existing and unchanged).

🤖 Generated with Claude Code