feat: derive knowledge reorganization with entity-driven topic aggregation

[tybot02]

Problem

The derive pipeline produces a flat, per-source-document wiki: one topic per
input file, with topic names echoing filenames. When a knowledge base is built
from hundreds of overlapping documents the resulting wiki contains redundant
articles, inconsistent naming, and no cross-document structure — users must
mentally reassemble the knowledge graph themselves.

Separately, the write phase hit reliability and UX issues at scale:

• Oversized merges: articles that accumulate material from many sources
  exceed the LLM context window, causing silent truncation or timeouts.
• No batch parallelism: the merge/split pipeline ran sequentially, making
  large KBs unnecessarily slow.
• Poor derive quality for CJK corpora: the filter stage used a
  whitespace-only tokenizer that collapsed Chinese/Japanese text into single
  mega-tokens, producing near-random relevance scores; multi-round voting was
  unavailable, so borderline articles were decided by a single noisy call.
• No visibility into builds: compile and derive jobs were fire-and-forget
  with no status tracking, detail inspection, or progress feedback in the UI.
• Chat not KB-scoped: sessions were global — switching knowledge bases
  mixed conversation history, making multi-KB workflows confusing.

Change

120 commits across Python AI engine, Go backend, and React frontend. The
changes group into seven areas:

1. Knowledge reorganization (py/src/kb_ai/derive/_reorganize.py, new)

An optional post-derive phase that restructures the flat topic list into a
coherent, cross-document knowledge graph:

• Entity index builder — scans every article for named entities (people,
  orgs, products, concepts) and builds an inverted index mapping entities to
  the topics that reference them.
• Entity name resolution — LLM-powered canonicalization that merges
  variant spellings, abbreviations, and aliases into canonical forms (e.g.
  "AWS", "Amazon Web Services", "亚马逊云" → one canonical entry).
• Topic-aware classification — a new classify-topical prompt that
  clusters topics by semantic similarity using the entity co-occurrence graph
  as a signal, replacing the filename-echo naming.
• Aggregation planning — an LLM planner proposes how to merge/split/rename
  topics into a coherent wiki structure, reviewed by a two-round feedback loop
  (aggregate-plan-feedback prompt) before execution.
• Orchestrator — wires the above into derive_kb / compile_kb behind a
  reorganize flag (default off, configurable per derive job via TOML config
  and API payload).

2. Oversized merge handling (py/src/kb_ai/commands/compile.py)

• Batch splitting — when accumulated source material exceeds the LLM
  budget, items are packed into right-sized batches with iterative
  merge-reduce, so no single LLM call receives an oversized prompt.
• Sub-article splitting — articles that grow past a configurable threshold
  are split into coherent sub-articles at section boundaries.
• Budget cap — _MAX_BATCH_BUDGET (20K tokens) prevents merge-path
  timeouts by capping per-batch input regardless of model context size.

3. Batch-parallel write (py/src/kb_ai/commands/compile.py)

• Three-phase parallel structure — _merge_batch_split is refactored into
  Phase A (classify, parallel), Phase B (merge, parallel per article), Phase C
  (split check, sequential) with a configurable parallelism threshold
  (batch_parallel_threshold) and batch-count limit
  (batch_parallel_batch_limit).
• _run_chain — a reusable async chain runner that fans out independent
  batches with bounded concurrency via semaphore, with early-break on batch
  count to avoid runaway parallelism on small items.
• Serial fallback — batches below the threshold run the legacy sequential
  path, preserving determinism for small KBs.

4. Derive quality improvements

• Multi-round voting (_filter.py) — select_by_topic now runs
  configurable voting rounds (filter_rounds, default 1) where each round
  independently scores article relevance; the final decision uses
  majority-vote aggregation with a configurable acceptance threshold
  (filter_threshold).
• CJK-aware tokenizer (classify.py) — the relevance scorer now uses
  character bigrams for CJK text instead of whitespace splitting, fixing
  near-random scores on Chinese/Japanese corpora.
• Language directives — six prompts gain explicit language-output
  directives so the LLM responds in the corpus language, not its default.
• Auto-deduplicate slugs — derive no longer errors on slug conflict;
  it appends a numeric suffix.

5. Builds page & build jobs (web/, internal/)

A full-stack Builds page replacing the old Tasks page:

• Backend: BuildJob model, SQLite store, API handlers
  (build_jobs.go), automatic BuildJob creation on submit. Paged
  ListDerivedJobsPaged and DeleteDerivedJob endpoints. Generic paged
  query builder extracted to sqlite/paged.go.
• Frontend: three-tab layout (Build Jobs / Compile Tasks / Derive Jobs)
  with PagedJobTable shared component, StatsBar summary,
  StatusStageBadge, master-detail Sheet panels
  (BuildJobDetailSheet, TaskDetailSheet, DeriveJobDetailSheet),
  DeriveWikiPreviewSheet for inline wiki preview, article table in
  derive detail, auto-polling via useAutoPolling hook.
• UX polish: WCAG-compliant status badge colors, details button
  replacing full-row click, unified 160px column widths, immediate
  thinking indicator in chat.

6. Chat KB switcher (web/src/features/chat/)

• Session model gains kb_slug; SQLite schema updated with migration.
• ChatKBSelector component lets users switch KB context mid-conversation.
• Session list and chat page filter by selected KB.

7. Bug fixes & code review hardening

• fix(upload): skip unsupported files in ZIP instead of rejecting the
  entire archive.
• fix(worker): don't burn retry attempts on circuit-breaker rejection.
• fix(derive): hash-based fallback slug for pure CJK topics.
• fix: enable TOC sidebar scrolling on overflow.
• fix: Windows-native compatibility (cherry-picked from fix: Windows-native compatibility for the derive-reorganization branch #63).
• refactor: structured slog logging, shared formatDate utility,
  extracted PagedJobTable, generic paged query builder, import grouping
  cleanup.

Config surface

Key	Layer	Default	Purpose
derive.reorganize	TOML / API	false	Enable knowledge reorganization
derive.filter_rounds	TOML / API	1	Multi-round voting rounds
derive.filter_threshold	TOML / API	0.5	Voting acceptance threshold
batch_parallel_threshold	Python	3	Min batches for parallel mode
batch_parallel_batch_limit	Python	8	Max concurrent batch chains

Test plan

• Python AI engine (35 new/updated test files, ~5,700 lines):

  • Reorganize: entity index, entity resolution, aggregation planning,
    orchestrator integration — test_reorganize_*.py (3 files, ~1,900 lines)
  • Compile: batch splitting, sub-article splitting, batch-parallel
    infrastructure — test_compile_batch_split.py (2,100+ lines)
  • Derive filter: multi-round voting, threshold behavior, CJK tokenizer
  • Classify: topic-aware classification prompt coverage
  • Daemon: reorganize parameter threading
  • Merge: _create_system language directive regression test

• Go backend (8 new/updated test files, ~2,800 lines):

  • internal/config: DeriveConf parsing, validation, env overrides
  • internal/store/sqlite: build_jobs CRUD, derived paged queries,
    session kb_slug migration
  • internal/api: build job handlers, derive endpoints, session
    kb_slug filtering, submit BuildJob creation
  • internal/derive: runner reorganize config threading
  • internal/bridge: daemon client reorganize parameters
  • internal/worker: circuit-breaker attempt-burn fix

• Web frontend (18 new/updated test files, ~3,100 lines):

  • Builds page: all three tabs, detail sheets, stats bar, status badge,
    auto-polling hook, wiki preview sheet
  • Chat: KB selector, session list filtering
  • Regression: AppLayout nav, existing Chat/Wiki page tests updated

• go test ./... — all pass

• cd py && uv run pytest tests/ -v — all pass

• cd web && pnpm test — all pass

• Windows compatibility verified via fix: Windows-native compatibility for the derive-reorganization branch #63 cherry-pick

🤖 Generated with Claude Code