Skip to content

Simplify Zig build targets and preserve subsystem test coverage - #657

Open
ajroetker wants to merge 10 commits into
mainfrom
codex/consolidate-build-targets
Open

Simplify Zig build targets and preserve subsystem test coverage#657
ajroetker wants to merge 10 commits into
mainfrom
codex/consolidate-build-targets

Conversation

@ajroetker

@ajroetker ajroetker commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Zig's build exposed duplicate and feature-specific test names, used lib-* for Antfly code, and ran benchmarks through build targets. This consolidates the interface around code ownership: lib-* covers zig/lib, Antfly suites follow subsystem paths, and benchmark/tool targets install binaries for callers to execute.

  • Remove 90 public names overall while preserving existing test selections. make zig-unit-test retains the CI gate across library, Antfly, and inference owners. Enrichment keeps the exact 220-case union, including all 62 formerly progress-only selections.
  • Consolidate DB benchmarks and comparisons under antfly-storage-bench and scripts/run_db_query_matrix.py. Remove the DOCID and algebraic matrix scripts, seven algebraic targets, and the archive-evidence checker. One storage_bench executable provides DB/query, analytics, ingest, provisioned-ingest, HBC, and summary subcommands, sharing the compiled storage implementation. Analytics, adaptive coverage, cold/warm reads, graph traversal, and public schema comparisons remain available through the shared matrix. Planner ownership becomes a normal DB test; dynamic-template checks remain in the unit aggregate.
  • Build/install benchmark artifacts without executing them. HBC and dense-ingest workloads join antfly-storage-bench; API workloads build under antfly-api-bench. -Dapi-bench-standalone=true builds only the lightweight client of an existing production executable. WAL and derived-log retain their storage-owner benchmark targets. Remove the vector-write wrapper and document its exact arguments alongside the migrated smoke/stress presets in zig/BENCHMARKS.md.
  • Make lib-<library>-conformance fetch missing fixtures, reuse cached corpora, and run verification. Support offline runs with -Dconformance-fetch=false and custom caches with -Dconformance-fixtures. Remove separate fixture-setup aliases and scripts.
  • Update GitHub Actions, callers, documentation, and tooling regression checks. The audit markdown remains outside the PR.

Restored validation exposed existing defects fixed here:

  • Native ordinal filters were applied before text paging and reapplied afterward, replacing the total match count with the page length (96 versus 48 across six searches). Execution now consumes enforced native filters once across text, dense, sparse, and primary-store paths. Dense hydration also avoids repeated JSON lookups while retaining residual predicates, identity constraints, and visibility checks.
  • A metadata test assumed adjacent JSON name and type keys. It now checks canonical names and preserved configuration semantically across reordered inputs and both response forms.
  • The generated unsupported WebP animation fixture was malformed. It now contains a valid ANIM/ANMF container and lossless frame; the original bytes remain an invalid-input regression. The generator verifies expected errors before writing rejected fixtures.

CI also exposed a streaming connection-policy bug: the E2E client requests Connection: close, but HTTP/1 streaming overwrote it with keep-alive before closing the socket. A following table-cleanup request could race socket retirement and receive a reset. Streaming now shares the buffered-response connection policy, honors client/handler closure and server request limits, and advertises closure before committing headers. Socket-level regressions exercise each policy and reuse before the request limit; the failing E2E test also asserts its response policy.

The retained benchmark drivers also used stale ownership and server APIs. Analytics now creates manager-owned indexes and uses public HLL reads with a dedicated root sketch. Public-query handler measurements call the typed query API; local HTTP measurements use the production registrar/listener instead of the removed executor. Standalone benchmarking reuses the production executable and its compiled runtime kernels. The sparse result-count oracle now includes sparse coordinate overlap and checks underfilled result sets even below k. HBC I/O counters preserve durability and atomic-rename capabilities; table reporting uses the version-aware offset accessor. The static provisioned catalog supplies routing projections. Concurrent query workers use standard thread stacks (LLDB confirmed the old 512 KiB reservation overflowed) and join all started workers on failure. Graph reporting takes a complete statistics snapshot. Churn comparisons match backend, profile, scale, and run identity, avoiding cross-workload baseline ratios.

Validation:

  • Original consolidation preserved all 3,899 DB cases and the default 108 test run nodes. Added query and planner-ownership regressions extend coverage.
  • 181 DB filtering/paging/post-processing regressions and 685 root checks passed. The earlier restored owner suites passed 778 checks; focused storage/enrichment regressions and library JSON/Yacc/SQL tests passed.
  • Storage query smoke and bounded profiles passed all 18 shape/case comparisons with unchanged correctness, resolution-counter, and performance thresholds.
  • Conformance setup regressions cover fetch/cache/offline behavior, retry, failure propagation, and fixture overrides. All 88 WebP/fixture-invariant checks, 15 WebP corpus cases, and six checked-in image verification commands passed. Regeneration reproduces the checked-in bytes; external WebP tools accept and decode the new frame.
  • Unified storage and both API build modes passed. All 17 shared smoke matrix steps and eight matrix/summary regression tests passed. Separate ingest, provisioned-ingest, HBC read/write/split/search, and HLL smokes passed; API local full-text and composed-hybrid HTTP smokes passed.
  • The new HTTP streaming regression failed before the fix; all 116 HTTP server tests passed afterward. The complete Antfly E2E job awaits CI.
  • Zig/Python formatting, shell syntax, workflow lint, and diff checks passed.
  • The full repository runtime suite and downloaded external corpora were not run locally. Offline external conformance runners correctly reject missing caches.

@ajroetker ajroetker added this to the v0.3.0 milestone Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant