Skip to content

feat(relational): add authoritative packed base-row storage - #502

Draft
ajroetker wants to merge 118 commits into
mainfrom
feature/relational-core
Draft

feat(relational): add authoritative packed base-row storage#502
ajroetker wants to merge 118 commits into
mainfrom
feature/relational-core

Conversation

@ajroetker

@ajroetker ajroetker commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What changed

  • carries relational storage mode and the typed column catalog through the durable runtime schema
  • keeps the hardened schema validation, exact document-to-cell projection, and versioned packed-row codec from the current mega-branch tree
  • adds a dedicated relational base-row keyspace and narrow relational_store facade
  • makes packed rows authoritative for writes, point reads, scans, filtering, identity, resolution, enrichment, and existing derived-index/rebuild consumers
  • supports normal transaction commit, one-shot/orphaned-intent recovery, replay, TTL deletion, split/merge cleanup, and portable backup/restore without creating generic JSON primary rows
  • rejects storage-mode changes and first-schema reinterpretation of existing physical rows
  • validates every final post-transform relational write at the DB boundary against one generation-scoped compiled public validator, covering direct/embedded batches, transactions, replicated apply, and recovery without reparsing schema JSON or rebuilding physical-field constraints on the write hot path
  • commits runtime schema, version history, and public validator JSON in one durable transaction; all fallible validator and recovery-schema preparation happens before publication
  • owns and synchronizes transaction-recovery column generations so the worker cannot race schema replacement or retain freed column slices
  • replicates the complete runtime/public schema pair in HA metadata v2, verifies the pair before standby publication, and remains compatible with v1 records
  • preflights portable archives with a bounded-memory two-pass validator that binds format version, row mode, runtime schema, and public schema before destination mutation; structurally valid but schema-invalid packed rows are rejected
  • stages embedded and Lite imports outside the live DB, rebinds identity in staging, rejects non-empty targets, and publishes every portable block in one atomic store transaction under a short apply-lock window
  • reloads durable schema ownership and every schema-dependent runtime context before an already-open embedded, Lite, or server restore becomes visible or starts rebuilding indexes

Scope

This folds the former R2 authoritative-base-row slice into #502 so the PR lands one minimal, usable relational storage path instead of an admitted schema and unused codec. The code was ported surgically from the current final tree of combine-pr-141-143-144, not reconstructed by cherry-picking its historical commits.

Still excluded:

  • relational primary/unique/covering/expression index lifecycle
  • checks, defaults, generated values, collations, arrays, temporal constraints, and distributed foreign keys
  • typed row HTTP APIs
  • relational SQL binding and lowering
  • catalog/namespace routing

Validation

  • complete post-merge bounded storage aggregate: 3,128 passed, 8 skipped, 0 failed, 0 leaked (16/16 build steps)
  • complete HA suite: 371 passed, 0 failed, 0 leaked, including v1 metadata compatibility and promoted-standby constraint enforcement
  • API HTTP/runtime suite: 39 passed, 0 failed, 0 leaked
  • C ABI suite: 11 passed, 0 failed, 0 leaked
  • embedded and standalone embedded package checks: 4 passed, 0 failed, 0 leaked
  • focused direct-batch constraint, reopen, portable schema-binding, atomic import, and immediate post-restore read/write regressions passed with no leaks
  • unit-storage-test-audit, storage compile, formatting, E2E collection, and git diff --check passed after the final main merge
  • exact-head Zig Tests run 32546016284 passed: x86_64 hermetic/TLA+, ARM64 codec, Antfly E2E, inference E2E, and low-FD regressions
  • current origin/main (46b702a8e) is merged cleanly

@ajroetker
ajroetker force-pushed the feature/relational-core branch from 430ad08 to 2bc3374 Compare August 19, 2026 16:51
ajroetker and others added 27 commits August 19, 2026 10:21
# Conflicts:
#	zig/pkg/antfly/src/storage/db/algebraic/schema_capability.zig
# Conflicts:
#	zig/pkg/antfly/src/runtime_error_abi.zig
Share bounded pinned schema plans across column blocks and primary overlays; support nested and hyphenated selections with costed late materialization. Add differential tests and benchmarks, repair schema OOM cleanup and LMDB free-list convergence, and preserve the merged main error ABI.
# Conflicts:
#	zig/pkg/antfly/src/runtime_error_abi.zig
Share off-lock, single-flight read projections; account and retire metadata epochs safely. Persist bounded tombstone GC objectives and decouple admission retries from age triggers. Add cursor, concurrency, fault, scaling, and physical churn coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant