Replace HBC LSM with native WAL-backed generations

[ajroetker]

Summary

• replace the dedicated HBC LSM with a crash-safe native v2 index: tokenized source transactions, native WAL/deltas, mmap-friendly immutable generations, atomic CURRENT, covered source sequences, and query generation leases
• store topology, quantized posting-local scan planes, postings, vector-to-leaf assignments, and exact rerank artifacts in the native format; only primary documents/source embeddings remain in the primary LSM
• preserve exact public scoring with SIMD float16/RaBitQ intervals and authoritative lossless float32 completion only where error bounds can change top-k membership or order
• stream publication with bounded memory, cold descriptor-based projection reads, reusable resource-governed query scratch, and node-wide work-weighted search admission
• migrate existing v1 indexes online: the old generation keeps serving while a durable v2 shadow builds, catches up, validates, and atomically promotes
• include public-API VectorDBBench lifecycle tooling and detailed experiment history in zig/VECTORDBBENCH_FINDINGS.md

Production invariants

Transaction and recovery authority

• every native mutation carries an opaque lease bound to index incarnation, capture epoch, ownership rights, and source sequence
• borrowers may append but cannot publish/cancel; stale callbacks fail closed and cannot consume a newer transaction
• WAL append/fsync precedes generation publication; CURRENT is sole durable authority and recovery replays only the uncovered committed suffix
• valid empty authority is an O(1) native generation, distinct from missing or corrupt state

Query correctness and resource governance

• selected leaves expose contiguous posting-local candidate planes and portable Zig SIMD scoring
• conservative error intervals bound approximate scores; centralized lossless residuals produce authoritative float32 public scores/order
• exact completion is limited to candidates whose intervals overlap the top-k boundary, with safe fallback when bounds or optional acceleration are unavailable
• query scratch is pooled and reclaimable; dense scan bandwidth is admitted FIFO by estimated work across the node before transaction/scratch acquisition
• authenticated scan costs are generation-local: an admitted query pins the exact immutable generation it was charged for, while a compact replacement can lower future permits without a process-lifetime high-water
• generation, positioned-I/O, scratch, and admission lifetimes are joined and cancellation-safe

Publication, backup, and cleanup

• complete projection publication uses one source-sequence-pinned vector generation and private cold random readers, avoiding shared page-cache pollution and mixed-generation artifacts
• cold projection reads coalesce physical neighbors per shard, run at most eight shard groups concurrently, release bounded bounce extents immediately, and preserve a direct buffered singleton path
• vector directories and immutable replacement flattening stream into staged files rather than materializing corpus-sized heap maps
• backup/restore authenticates native install paths; publication is pointer-last and crash-safe
• cleanup revalidates canonical active pointers before deletion and preserves unrelated files

Rolling upgrades

• logical index names/configs stay stable while the physical format advances from legacy v1 to native v2
• runtime status retains released V12 and existing positional V15 decoding; framed V16 atomically introduces native projection/authority fields and the extensible record envelope
• metadata activates the native capability floor only after every table-serving voter supports it; native readiness debt is never downgraded to a false-ready state
• legacy v1 remains queryable during migration; correctness repair outranks format migration and remains fail-closed
• new indexes use direct O(1) native-v2 admission after capability activation; older indexes migrate through the shadow lifecycle
• the AFQD file has one strict initial schema version; earlier numeric revisions existed only within this unreleased PR and are intentionally unsupported

Latest qualified public-API results

Same 1M VectorDBBench corpus, public HTTP API, batch 100, four load workers, default unrelated full-text index removed through the public table contract, native HBC, and exact score semantics:

Metric	Result
Ready time	724.25 s / 12.07 min (634.62 s insert + 89.63 s readiness)
Recall	0.9924
QPS c1 / c10 / c20 / c30	70.74 / 626.82 / 701.63 / 649.57
Reversed warm QPS c30 / c20 / c10 / c1	665.97 / 699.61 / 624.66 / 97.64
p95 c1 / c10 / c20 / c30	13.85 / 24.68 / 66.73 / 87.48 ms
Live peak RSS	6.26 GB, down 6.9% from the prior 6.72 GB cache-inclusive run
Restart RSS	2.37 GB
Mean server time	9.97 ms
Approximate / exact completions per query	239,918 / 146.6

Node-wide admission reached 381 MB across 17 active queries, queued 13 queries at peak, and recorded no cancellations. The reversed warm c30 result improves on the uncapped reversed baseline (608.29 QPS, 99.54 ms p95) and is close to r126 throughput (673.72 QPS) with better p95 than r126 (92.88 ms).

The latest fresh 50K qualification, after generation-coupled admission and bounded coalesced cold reads, reached ready in 31.28 s (14.13 s insert + 17.15 s readiness), recall 0.9855, QPS 225.4 / 929.9 / 1,017.2 / 1,029.4 at c1/10/20/30, p95 5.91 / 21.33 / 53.67 / 76.00 ms, 1.10 GB live RSS, and 339 MB restart RSS. The governor held all 30 concurrent queries at a 201.28-MB peak against 402.65-MB capacity with zero queueing, waits, cancellations, or wait time. Its detailed public profile scored 23,707.5 approximate and 137.3 exact vectors/query with 3.492-ms mean server time.

The host retained unrelated repository test load during that run, so its absolute QPS is not an uncontended replacement for the earlier 30.49-second, 0.9862-recall, 274 / 1,109 / 1,198 / 1,229-QPS qualification. The follow-up establishes that the rejected process-monotonic admission prototype's one-query/288.5-QPS serialization is gone without changing recall, candidate work, or exact-boundary reranking.

Full measurements, comparisons, rejected variants, and caveats are in zig/VECTORDBBENCH_FINDINGS.md.

Validation

• merged current origin/main at 20962387bc in 7ba25f9cd9
• make generate
• make zig-generated-check
• zig build -Doptimize=Debug
• zig build -Doptimize=ReleaseFast -j2
• zig build antfly -Doptimize=ReleaseFast -j2
• zig build lib-vectorindex-test -Doptimize=Debug: 114/114
• focused admission, native restart, cold-reader, and positional vector-block storage tests: 4/4
• zig build unit-storage-compile -Doptimize=Debug
• zig build resource-budget-test -Doptimize=Debug: 78/78
• zig build dense-index-lifecycle-regression-test -Doptimize=Debug: 45/45
• isolated metadata unit lanes: 195/195 and 226/226
• zig build lib-metadata-service-test -Doptimize=Debug: 97/97
• fresh ReleaseFast 50K and 1M public-API load/query/restart qualification, plus the post-admission 50K qualification above
• git diff --check

One live-Postgres hosted-round aggregate test timed out locally while its synthetic metadata leader remained unavailable; deterministic metadata, codec, lifecycle, and service suites are green. CI is running on the merged commit.

Deferred artifact ownership boundary

Primary source embeddings and native exact-vector artifacts still have separate ownership. A later catalog feature should assign immutable source-artifact identities and explicit references, then let users promote validated model/dimension/schema/vector-space-compatible artifacts to managed/shared ownership. HBC topology, postings, and tree generations remain index-private.