fix: the four defects a ~500-article corpus surfaces

[lucasmaan]

Running the knowledge-base pipeline over a ~500-article corpus surfaced four defects a small KB never
reaches: chat silently lost all KB grounding once the catalog outgrew the prompt budget, the classifier's
article ranking scored 0.0 on a Chinese title and 0 on a kebab-case topic no article title welds the same
way, a bulk ingest through the queue ran 4 documents at a time where the equivalent CLI run does 16, and a
rate limit or a half-open circuit breaker burned a healthy document's retry budget until it failed
permanently. This branch fixes all four across 18 commits organised as four tasks, each carried through code
review and mutation testing; three further commits ride along, one adding a test main was missing and two on
the tokeniser module this work introduces. Figures below are measured against data/distill-2026-06.log
(108 documents, Sonnet 4.6, workers=12; 527 calls, 2450.88s of wall clock, so ~4.9 calls and ~22.7s of
wall clock per document, or ~272s of serialized per-document work) and the real KB in data/kb-knowledge.
Every corpus figure below is over the 675 entries in that KB's index/master-index.md, which is the listing
both classify and retrieval read; wiki/ holds 682 files and 7 of them parse no frontmatter title, so they
never reach the index and no figure here rests on 682.

Retrieval: the catalog no longer overruns the prompt budget

An operator with a large KB saw chat answer from the model's own knowledge with no citations, the only clue
a line on stderr. Page selection sent every catalog line in one prompt; at the 348.6 chars a catalog line
averages on the reference KB (median 342.0, measured with the shipped render_catalog_line over the 675
indexed articles), an 80K budget is spent at about 229 articles. Past that the client refuses
the prompt before sending, _select_relevant catches the refusal, and the answer is ungrounded. A
500-article KB is therefore silently un-queryable.

retrieval/retrieve.py:69 (_fit_catalog) now ranks the catalog by lexical overlap with the question and
keeps the lines that fit, so an oversized KB costs recall instead of dropping grounding whole. Ranking
needs a tokeniser that survives CJK, hence the new py/src/kb_ai/_text.py: each ideograph is its own
token, every other script is a word run matched through \w. Restricted to [a-zA-Z0-9], a Chinese title
tokenises to the empty set, every article scores 0.0, and the cut keeps whatever came first.

Three further defects review found in the fit, each reproducing the same silent loss of grounding:

• Nothing capped a chat message, so a pasted document arrived as the query, drove the catalog budget
  negative, emptied the listing and overran the prompt limit on its own. Selection now reads a bounded
  prefix (_MAX_QUERY_CHARS, retrieve.py:41, applied at :112); the answering call still gets it all.
• The scan stopped at the first line too long to fit, discarding every shorter article ranked below it:
  for one 930-char line against a tight budget, that kept none of 21. It now skips and carries on.
• \w carries the underscore, so cb_cooldown_sec stayed one token and a question about "the cooldown"
  scored 0 against the article defining it. Reference-table keys are why the catalog line carries a keys
  column at all, so those articles were dropped first. _ is now out of the word class (_text.py:35).

Rate limits: 429 is retryable, and it is what gates concurrency

429 sat with the malformed-request statuses in the non-retryable branch, so a rate limit ended the call on
the first response and took the document's phase with it. It is the one 4xx that clears on its own, and it
fires harder the more concurrency a run is driven with, which made it a blocker for raising worker counts
at all. llm/_completion.py:132-164 retries it on the existing backoff, floored by the server's
Retry-After when the header carries a usable delay and capped at _RETRY_AFTER_CAP_S = 120.0 (:51) so
one absurd value cannot park a worker for the rest of a run. The HTTP-date form of Retry-After is
RFC-legal and deliberately ignored (honouring it means trusting our clock against the server's), and a
test now says so rather than only a docstring. The label stays distinct from the gateway family: both are
retryable, but only this one says concurrency is the cause.

Queue ingest: 4 documents at a time against the CLI's 16

A bulk ingest through the queue route ran 4 documents at a time. One queue task carries one document, so
that was the whole of a run's document-level parallelism, while kb-ai compile does the same work 16-wide.
The 500-document run took 8 hours, observed rather than instrumented. Against the baseline's ~272s of
serialized work per document, 500 documents at 4 lanes predicts about 9.5 hours, 18% pessimistic against
that 8, so the observed time is a throughput ceiling rather than a regression. The same arithmetic gives
about 3.2 hours at 12 lanes, which is an extrapolation and not a measurement. worker.extract_workers goes
4 -> 12 and ai.daemon.concurrency 8 -> 16
(internal/config/config.go:103,138, etc/kaas.toml:27,52). 12 rather than the CLI's 16 because a
queue-route ingest shares the daemon's 16-slot semaphore with chat, derive and retrieval, so 16 would
saturate it for the length of the run, while kb-ai compile owns its process and needs no such margin.
The daemon pool rises with it because every dispatched document holds a daemon slot for its whole
pipeline, so a pool of 8 would just become the new limit.

Config: refuse a worker pool the daemon cannot serve

The daemon >= worker invariant was guarded by a test over the shipped defaults only, so a deployment
writing extract_workers = 32 loaded cleanly and got the daemon's 16-slot semaphore as the silent real cap,
so the raise read as having bought nothing. validate() now refuses the unpaired raise and names both
figures (internal/config/config.go:249-253). Refused rather than clamped: clamping would ignore the
number the operator wrote down.

The comment justifying 12 was also wrong in a way future tuning would have relied on. It claimed the
per-phase fan-out inside Python collapses to one group because a queue task carries one document. It does
not: a document over 16,000 characters splits and each phase fans out to min(chunks, KB_WORKERS) calls of
its own, so the worst case is the product: 12 x 16 = 192 in-flight calls, the figure a gateway's rate
limit is actually met with. KB_WORKERS has no config field, but the daemon inherits the backend's
environment (bridge/daemon.go:98 passes os.Environ() through), so exporting it is how the second factor
gets bounded; etc/kaas.toml:19-26 now says so. 192 is labelled a ceiling for uniformly long documents
rather than an observed figure; the baseline log's documents average 2.8 chunks and peak at 9, so its
typical fan-out was near 34.

Classify: the ranking was dead, the fit wasted the budget, and the dedup was wrong

Two defects made the classifier create duplicate articles once a KB outgrew the prompt budget, and they
compounded. _title_words stripped everything outside [a-zA-Z0-9\s], so a Chinese title tokenised to the
empty set and _relevance_score (core/classify.py:122) returned 0.0 for every article. The sort was
stable and therefore a no-op, leaving the cut to keep whichever articles came first. The same regexp welded
a hyphenated topic into one token, so the damage was never limited to Chinese: across all 982 extractions in
the reference KB, 13,721 of 16,289 topics are kebab-case slugs (84%), the mean best score across the
675-article catalog was 0.195 against
0.691 now, and 15% of extractions scored zero against every article, now 0%. Separately,
_fit_articles_to_budget (core/classify.py:149) halved the list until it fit, discarding 41-43% of what
the budget holds: 84 entries kept where 143 fit at a 38K budget, 168 where 292 fit at 78K. It now walks
greedily, keeping every entry that fits and skipping (rather than stopping at) one whose block does not,
matching retrieve._fit_catalog. Combined, the merge target survives the cut for 89% of the 675 articles
at 38K (was 83%) and 93% at 78K (was 88%).

Migrating dedup_create_new to the shared tokeniser turned a missed-duplicate failure into a false-merge
one, which is the expensive direction: a false merge writes a document's knowledge into an article that
never claimed the subject and nothing later undoes it, while a missed duplicate leaves an article a later
compile can still merge. Five review rounds each found a class the previous fix had not closed, every one
killed by a measurement through the shipped run_dedup_phase rather than by argument:

• Date-differing siblings and CJK character subsets: 发言复盘 2026-01 against 发言复盘 2026-03 scored
  0.83, 上海 against 海上运输 1.0, 数据安全 against 安全数据库 1.0.
• A length-ratio guard binding twice as hard on Chinese as on English, because bigrams roughly double a CJK
  title's token count, so 向量检索/向量检索基础 stopped deduplicating while the English
  Vector Search/Vector Search Basics pair still did, reopening the collision the dedup exists for.
• A one-character negation reads as an addition rather than a substitution, since 不支持向量检索 carries
  every token of 支持向量检索: that pair scored 0.91, 启用灰度发布 against 停用灰度发布 0.80. Both were
  0.000 before the branch only because non-ASCII tokenised to nothing, and the same shape was already
  reachable in English, where Rate Limiting Enabled/Disabled On Gateway scored 0.80 on both sides of the
  branch. The polarity gate below puts all three at 0.000.
• The arithmetic that broke all of it: for two n-token titles differing in ONE token, Dice is exactly
  1 - 1/n, so _NEAR_IDENTICAL = 0.85 was a bar on title length, not on rewordings, and it was
  cleared by every pair of 7 tokens or more (581 of the 675 titles). 允许跨境数据传输的合规评估结论 against
  拒绝跨境数据传输的合规评估结论 scored 0.857, an approve/refuse inversion. No constant fixes that, so the
  arm and its constant are deleted.
• Set containment cannot see a transposition: 腾讯云到阿里云迁移方案 and 阿里云到腾讯云迁移方案 carry
  exactly the same tokens, so every set-based score read 1.000 and the pair merged, as did
  Migration From Redis To Kafka Decisions against the same title with Redis and Kafka swapped.

The rule that survived: a duplicate title is the same title with words added in place: nothing
substituted, nothing reordered, no disagreeing number, no unpaired negation. core/classify.py:248-369
implements it as a subsequence test over _text.bigram_sequence (_text.py:45, bigram tokens in reading
order with repeats kept, CJK runs as character bigrams, & read as the word it stands for), a numbers gate
(_NUMBER_TOKEN, :273, bare digits or a one-to-two-letter prefix over digits so q1, v2, h1 count),
a polarity gate (_POLARITY_MARKERS, :263), and Dice at 0.7 (:250). Measured with the shipped
functions: 0 false merges over 22 hand-labelled bad pairs (every one the five rounds named, all parametrised
in py/tests/test_core_classify_scoring.py) and 1 miss over 10 hand-labelled duplicates. Over the 675
titles, 23 pairs merge against the pre-branch rule's 46; of the 30 it refuses, 5 are genuine duplicates in
classes the docstring names.

Worker: a half-open circuit breaker no longer spends healthy documents' retries

A document that hit the breaker failed permanently although nothing was wrong with it. The diagnosis is one
layer below where it looks: attempts is spent by ClaimNext at claim time
(internal/store/sqlite/sqlite.go:334) and no path ever gave it back; MarkFailed(retry=true) and
RecoverExpired both leave it. Worker.Process funnelled circuit.ErrOpen into the same Nack path as a
real engine failure, so each refusal cost the document a retry for a call the engine never saw. Only one
task per half-open round passes the breaker's single trial, so raising worker.extract_workers to 12
turned 3 wasted attempts per round into 11. Both layers that were wrong are fixed:

• store.ReleaseTask (internal/store/store.go:130, sqlite.go:427) is the inverse of ClaimNext:
  owner-scoped, back to pending and queued, attempts = MAX(attempts - 1, 0), lease cleared, no error
  recorded. queue.Release wraps it (internal/queue/queue.go:108) and Worker.failEngine
  (internal/worker/worker.go:187) routes circuit.ErrOpen to it at either the extract or the pipeline
  stage. A real engine error still Nacks and still spends the attempt, so a poison document cannot retry
  forever.
• The dispatcher scales its batch to what the breaker admits (internal/worker/dispatcher.go:90-95):
  nothing while cooling down, one task while half-open, the full semaphore when closed. Without it the
  release path would churn a whole batch of claims and releases every poll tick for as long as the probe
  call runs.

Two alternatives were priced and rejected: moving the increment from claim to Nack loses crash-loop
protection (a worker that dies mid-task must still spend an attempt, which is why RecoverExpired keeps
it), and making the breaker block rather than reject holds the lease while waiting and risks lease expiry,
which burns the attempt anyway. A released task records no error, so the fix removed the
extract: circuit: breaker open rows that used to be the only sign of an outage; the dispatcher now logs
each breaker transition once, on change rather than per tick (dispatcher.go:73-75), since a long probe
holds half-open for many ticks.

Also carried: the break-less script range spelled as escapes

Two commits unrelated to the scale defects, on the tokeniser module this branch introduces.
_SCRIPTIO_CONTINUA (_text.py:34) was written as the eight literal characters bounding its four blocks.
Two of them, U+3040 and U+FAFF, are unassigned codepoints that render as an empty box, so the range was
unreadable next to a docstring that describes it numerically. It was also fragile: an NFKC pass over the file
rewrites U+30FF into two characters and U+F900 into U+8C48, leaving a character class that still compiles and
now runs unbroken from U+4E00 to U+FAFF. Nothing raises and nothing shows in a diff, but Yi, every Hangul
syllable and the private use area silently become one token per character, which moves every lexical score
in the retrieval and dedup paths. The constant is now non-raw escapes, resolved at parse time, and
_SCRIPTIO_CONTINUA with all three compiled patterns was verified equal to the values captured before the
rewrite. Thirteen new cases in test_text.py pin both ends of all four blocks and five word characters just
outside them; seven mutants on the range were killed, each reverted afterwards to confirm the new
assertions are what killed them.

Deliberately accepted costs and behaviour changes

• A refused task flaps in the task list. An operator sees it move pending <-> running with attempts
  oscillating and updated_at moving, and if the engine is permanently down tasks requeue indefinitely
  instead of failing. That is the intended semantics (the work really is still pending), and the transition
  log is where the outage shows.
• ReleaseTask does not clear error. If an earlier delivery failed for real and a later one was merely
  refused, that message is the last thing actually known about the document. Expect this in the UI:
  attempts can fall while an older error stays on display, which could not happen before, because every
  requeue went through MarkFailed.
• Two gaps in the transition log come from State()'s semantics rather than the loop. A probe that fails and
  re-opens between two poll ticks logs nothing. And once the cooldown elapses with an empty queue, State()
  answers half-open with no probe running, so the last line reads half-open for the rest of a total outage.
• drain is capped per tick (dispatcher.go:106): it no longer reuses slots freed mid-drain, so a
  backlog of fast-failing tasks drains at maxConc per poll interval (12/s at defaults) instead of all at
  once. Irrelevant for LLM-bound work; about 42s for 500 unreadable files.
• A probe in flight still costs one claim/release round-trip per tick, 2 SQLite writes. Removing it needs new
  breaker state ("is a trial slot free"), more surface than the churn is worth.
• The dedup rule has three limits, each in the docstring and each pinned by a test: an addition that narrows
  the subject still merges (Gateway Migration Plan Deprecation); a respelling that splits a token counts as
  a substitution and is refused, which costs one real duplicate (Global Architecture Bi-Weekly against
  Biweekly, likewise May 6 / May6); and the relation is not transitive, so a generic title acts as a
  hub (the corpus's 15 hub triples are all the one 发言复盘 archive).
• The dedup backstop is now three gates, two tokenisers and one threshold, and its failure mode is a
  duplicate article. Review judged it mergeable as is. The simpler alternative nobody has priced is dropping
  the lexical backstop and leaving deduplication to the classifier's own output. The classify prompt itself
  is untouched by all of this.
• Two attempt-burning paths remain, and both are judged correct: ErrOpen on an already-cancelled context
  (the lease is gone and RecoverExpired owns the row, and this is tested), and a transient SetStage error
  (pre-existing, and that document did do work).
• Two mutation survivors are left in place on purpose. Dropping status = ? from ReleaseTask's WHERE
  clause is unobservable, because every exit from running clears lease_owner, so the owner predicate
  already excludes those rows: defence-in-depth identical to SetStage's and Heartbeat's. And restoring an
  unbounded drain for the closed arm survives, because catching the per-tick cap needs slots to free during
  a drain and whether they do is a scheduler race, so no non-flaky test holds it. The cost of that one is
  throughput, and the loop header states the bound.

Testing

Python, re-measured at the branch head: uv run pytest -q --ignore=tests/test_distill.py gives 1763 passed /
1 xfailed, against 1619 on the same exclusion at the fork point eddefb9. The whole suite there gives 1639,
which is not the same measurement: it counts the 20 distill tests that pass. That is not the whole suite, and
the caveat matters:
tests/test_distill.py holds 21 tests and is excluded because one of them,
test_distill_end_to_end_produces_article, makes a live gateway call and fails here. It is not a branch
failure. Its skip guard checks LLM_API_KEY or OPENAI_API_KEY, the second of which is set in this
environment, so the test runs, but LLM_BASE_URL is unset and it has no endpoint to reach. The count is
lower than the 1770 measured at fdbb1ad for the same reason: 1770 counted the 20 distill tests that do
pass, and 13 text cases have been added since. Coverage, measured on that run: core/classify.py, _text.py
and retrieval/retrieve.py all 100%.

Go, re-measured at the branch head after the second merge: go test -count=1 ./... passes across all 15 packages,
go vet ./... and go build ./... clean, gofmt clean on every file this branch touched. 178 tests across
the three packages the worker work touched,
with internal/worker green under -race and under -shuffle=on. Coverage: internal/worker 98.7%,
internal/queue 100%, internal/store/sqlite 93.0%, and ReleaseTask, failEngine, release and drain
each 100%. Five files elsewhere (internal/api/{api_test,chat,session}.go, internal/bridge/daemon.go,
internal/worker/worker_test.go) were already unformatted on origin/main and were left alone.

Mutation testing, always with the repository untouched (Python: a /tmp copy plus PYTHONPATH; Go:
throwaway git worktrees): 11 mutants in the config/retry follow-up, 39 of 39 in the classify work, 34 in the
worker work, 7 on the tokeniser range. Every mutant that changes production behaviour was killed, with the
two deliberate exceptions listed above, and the reverse control was green each time. The classify set
includes the subsequence test relaxed back to set containment, tested one way only, or walking a fresh
iterator per token (plain set containment in disguise); _NUMBER_TOKEN widened and narrowed; both halves of
the polarity list deleted.
The retry set includes _TIMEOUT_BACKOFF_BASE 10 -> 7, which killed 9 tests once the assertions stopped
reading the constant out of the module under test, and 0 of 44 before. Two worker mutants survived until the
tests were strengthened, both for the same reason: a bound asserted where an exact count was available. Two
further worker experiments were invalid (one hit a compile error, one hung on the pre-existing unguarded
<-started in TestDispatcherRespectsConcurrencyCap, which hangs rather than fails under mutation).

Every review round found the same pattern, never in the production code: an assertion narrower than the
behaviour it stood in for, fixed by widening the test. Swapping the "extract: %v" / "pipeline: %v" stage
literals left the whole suite green, and TestDispatcherPausesWhenBreakerOpen stays green when a task is
claimed and handed straight back, so case StateOpen: continue had no test at all.

Base

origin/main is merged into this branch twice, at 48b4db3 and 36c2f8c, rather than rebased onto it. The
branch forked at eddefb9. The first merge brought in 0d76328, which edits Worker.Process's
PipelineRequest (Model: w.cfg.Model), the same function the worker work changed, plus 253e25d and
05f51ea, both CodeQL workflow changes. Merging keeps main's fix alongside this branch's, and keeps the
reviewed commits and the SHAs the session log references intact.

0d76328 landed with no test, so one commit here adds one (process_test.go, inside the existing
TestProcessPassesConfigToEngine, which already records both bridge requests) and corrects Config.Model's
comment, which listed only ExtractRequest.Model. Two mutants confirm the assertion bites: dropping the
field, and forwarding SummarizeModel instead.

The second merge, 36c2f8c, brought in #55, where this branch's two pure-translation commits a791320 and
e4295f6 landed, plus #54 and #50. #55 landed as a squash, so main's copies of those files share no
ancestry with the branch's and all seven conflicted. Five were pure translation and were taken whole from
main, whose copies carry review fixes made after this branch forked; they are byte-identical to
origin/main now. etc/kaas.toml and py/README.md carry both translation and this branch's config
content, so they were merged by hand, keeping main's comment corrections on top of this branch's
extract_workers = 12, concurrency = 16 and the KB_WORKERS / KAAS_DAEMON_MAX_WORKERS rows.

The two translation commits still show in the commit list because they are ancestors, but their content is
main's now, so this pull request contains no translation file. git diff --name-only origin/main...HEAD is
25 files, +2472 / -154; the only prose in it is py/README.md's two environment-variable rows, which
document this branch's own concurrency change.

🤖 Generated with Claude Code