Skip to content

feat!: archive old outbox events to object storage - #105

Draft
nicolasburtey wants to merge 2 commits into
mainfrom
feat-event-archive
Draft

nicolasburtey wants to merge 2 commits into
mainfrom
feat-event-archive

Conversation

@nicolasburtey

@nicolasburtey nicolasburtey commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

First-class cold-storage archiving for persistent outbox events: settled spans of history are swept from postgres to object storage as JSONL, with a transparent read fallback so old history stays consumable through the existing listener API.

obix core is calendar-agnostic: an ArchiveBoundaryProvider certifies contiguous history as final at a stream position and supplies an opaque label used only to group files in storage. The calendar/day notion lives only in concrete policies downstream (or in the provided default).

Write side

  • EventArchiveStorage — minimal put/get trait; obix ships no backend (GCS impl lands downstream in lana-bank). InMemoryArchiveStorage provided for tests.
  • ArchiveBoundaryProvider — pluggable "which history is settled" decision, returning ArchiveBoundary { label, up_to_sequence } oldest-first. Default DailyRetentionBoundary certifies purely by age (UTC-date buckets, ISO-date labels); lana-bank will plug in an EndOfDaySettled-based provider (GaloyMoney/lana-bank#7706) so a span is only swept once settlement certifies it — a stalled settlement then correctly stalls archiving.
  • EventArchiver sweeps up to boundaries_per_run (default 1) settled spans per run, oldest first — first-run catch-up naturally produces one labeled group per span. Sequence gaps are materialized as placeholder lines so every file is contiguous; files roll over at target_file_bytes (16MB default, never mid-event); paths are deterministic (<prefix><label>/events-<min>-<max>.jsonl).
  • Crash-safe/idempotent at file granularity: storage write first, then manifest insert + event deletion in a single statement (WITH deleted AS (DELETE …) INSERT … ON CONFLICT DO NOTHING); reruns overwrite the same path and no-op.

Read side (transparent)

  • Listeners resuming below the archive watermark are streamed from object storage first — whole files, one-ahead prefetch, bypassing the hot broadcast cache — then cross into postgres mid-stream. Public stream item type unchanged (Result<Arc<PersistentOutboxEvent<P>>, UndecodableEventError>); undecodable old payloads surface through the existing error path.
  • Guards the placeholder-masking hazard: pg must never be asked for pre-watermark sequences (load_next_page would synthesize placeholders for deleted rows, disguising archived events as rolled-back transactions). Watermark checked once per backfill request; archiving-without-reader misconfiguration logs an error instead of silently degrading.

Wiring

  • MailboxConfig.archive: Option<ArchiveConfig> (additive); Outbox::register_event_archiver(jobs, …) — reschedules immediately while catching up, polls hourly when idle.
  • persistent_outbox_archive_chunks manifest folded into the existing setup migration (pre-production); manifest is pure sequence bookkeeping (path, min/max_sequence).

BREAKING CHANGE: MailboxTables gains five required methods (archive_watermark, list_archive_chunks_from, list_archivable_boundaries, load_raw_export_page, record_archive_chunk) — non-derive implementors must add them (all known users go through the derive). Prefixed deployments must create the <prefix>persistent_outbox_archive_chunks table.

Test plan

  • cargo nextest run — 64 tests pass, incl. 6 new: export+prune per settled span (paths/manifest/pg assertions), replay across the archive→pg seam from a fresh outbox incl. live publish after catch-up, real sequence-gap materialization (rolled-back INSERT), rerun idempotency, end-to-end job sweep, NotConfigured guard
  • cargo clippy --workspace --all-targets clean, cargo fmt --check clean
  • SQLX_OFFLINE=true cargo check --workspace --all-targets (offline cache regenerated from scratch)

Note

High Risk
Changes the persistent outbox read/write path (delete-after-archive, watermark routing) and breaks manual MailboxTables implementors; incorrect boundary certification or ops mistakes can cause silent gaps or stuck backfills below the watermark.

Overview
Adds cold-storage archiving for persistent outbox events: settled history is exported as JSONL (optional gzip), recorded in a new persistent_outbox_archive_chunks manifest, and pruned from Postgres in one atomic statement per chunk (contiguity guard rejects overlapping manifests).

Write path: pluggable EventArchiveStorage and ArchiveBoundaryProvider (default DailyRetentionBoundary by UTC date + retention); EventArchiver with advisory locking, gap placeholders, file rollover, and Outbox::register_event_archiver job wiring (catch-up reschedules immediately).

Read path: listeners resuming below the watermark are served from object storage first (ArchiveReader, prefetch), then Postgres; backfill re-checks the watermark per page so mid-walk pruning does not wedge or mask archived rows. Misconfiguration (archive without reader) degrades to SELECT-only placeholders with loud tracing instead of load_next_page gap-fill INSERTs.

Breaking: MailboxTables gains five archive methods (generated by MailboxTables derive); MailboxConfig::archive is the new optional hook.

Reviewed by Cursor Bugbot for commit 0ec87d8. Bugbot is set up for automated code reviews on this repo. Configure here.

@bodymindarts bodymindarts left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLAUDE generated review:

Deep review of the archive feature. The overall design is strong: pluggable EventArchiveStorage + ArchiveBoundaryProvider keeps obix backend- and calendar-agnostic, the WITH deleted AS (DELETE ...) INSERT ... ON CONFLICT DO NOTHING single-statement manifest+prune is a clean crash-safety story, path-encoded compression is a nice rollback-friendly touch, and the placeholder-materialization on export preserves the contiguity contract. Tests cover the seam, gaps, idempotency, and the job. The Option<ArchiveConfig> surface is genuinely opt-in — existing users are unaffected at runtime.

I'm requesting changes for one correctness race and a few hardening asks:

Blocker

  1. Archive-prune vs. backfill pg-walk race (src/out/persistent/cache.rs:288) — the watermark is checked once per backfill request, but load_next_page writes placeholder rows for gaps. A concurrent archiver prune mid-walk makes a consumer silently receive placeholders for real events (indistinguishable from rolled-back txns — its cursor advances past them forever) and pollutes the live table with placeholder rows below the watermark. Most likely during the first catch-up sweep racing a full-replay consumer — exactly the deployment moment. Details + suggested fix inline.

Concerns (inline)
2. No mutual exclusion on run_once — concurrent archivers can produce overlapping, disagreeing chunks in the manifest (src/archive/archiver.rs).
3. Reader output is not monotonic by construction if the manifest ever holds overlapping chunks; a late placeholder can replace an unconsumed real event in the listener's local cache (src/archive/reader.rs:91).
4. DailyRetentionBoundary age floor is retention − 1 day; retention = 1 day archives seconds-old history and can race in-flight transactions into silent event loss (src/archive/boundary.rs).
5. One corrupt line wedges every below-watermark consumer forever; no integrity metadata in the manifest to detect it (src/archive/reader.rs:86).
6. The manifest table was folded into the already-released setup migration — checksum mismatch for any DB that applied 0.5.0 (migrations/20251204130225_obix_setup.sql).

Test asks

  • A test interleaving an archiver run with an in-progress pg-walking backfill (the blocker's scenario).
  • A test where put fails mid-span (storage outage) asserting no manifest entry / no prune / clean retry.
  • A test for the archive_reader_missing misconfiguration path (watermark present, no reader).

Confirmed non-issues while reviewing (for the record): archival fires no NOTIFY (notifications ride only the persist INSERT statement, and neither fill_gaps nor record_archive_chunk notify), so slim-NOTIFY (#101) is unaffected; archive reads route through decode_persistent_event, so the #104 Err-arm semantics are preserved end-to-end; old consumer cursors below the watermark (lana-bank cold restarts, cala EC rollup replay) transparently resume from the archive; live traffic is unaffected by a storage outage (archiver job retries; backfill degrades to a 5s retry loop only for pre-watermark reads).

Operational note for downstream rollout: every pod that might serve pre-watermark reads must be configured with the same ArchiveConfig (storage creds included) — a pod without it serves placeholders for archived history (loudly logged, but still degraded). Worth a line in the lana-bank rollout notes.

Comment thread src/out/persistent/cache.rs Outdated
// placeholder rows and mask the archived events as rolled-back
// transactions. Serve it from the archive first, then continue
// from the live tables.
match Tables::archive_watermark(&pool).await {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker — watermark is checked once per backfill request, but the pg-walk below races the archiver's prune.

Scenario:

  1. Backfill starts at current_sequence >= watermark (or watermark None) → archive path skipped.
  2. Mid-walk, the archiver sweeps a span covering (current_sequence, B]record_archive_chunk deletes those rows.
  3. The next Tables::load_next_page call (line 329) sees the deleted sequences as gaps and — per its contract — runs fill_gaps_query, which INSERTs placeholder rows below the new watermark into the live table and delivers them to the consumer as placeholders.

Net effect: the consumer's cursor advances past real events it never saw (placeholders are indistinguishable from rolled-back transactions — this is precisely the hazard the module doc warns about), and the live table is permanently polluted with sub-watermark placeholder rows that nothing ever deletes (list_archivable_boundaries filters sequence > $1).

The widest window is the first catch-up deploy: the archiver sweeping months of history span-by-span while a full-replay consumer (e.g. cala's EC rollup cold start) pg-walks the same range. But steady-state is exposed too — any backfill in flight when the hourly sweep lands.

Ask: re-check the watermark before each load_next_page iteration (it's an indexed MAX(max_sequence) — cheap relative to the page load) and loop back into stream_archived when it has advanced past current_sequence. A leaner variant: only re-verify when a page comes back containing gap-filled placeholders. Plus a test that interleaves an archiver run with an in-progress backfill.

Related: the archive_reader_missing branch below has the same side effect — beyond serving placeholders it also writes them below the watermark via fill_gaps. If you keep that degraded path, consider at least making the pg-walk use the SELECT-only load_events_in_range when below a known watermark, so misconfiguration doesn't mutate the table.

Comment thread src/archive/archiver.rs

/// Archive up to [`ArchiveConfig::boundaries_per_run`](super::ArchiveConfig)
/// settled spans, oldest first.
pub async fn run_once(&self) -> Result<ArchiveRunReport, ArchiveError> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern — nothing enforces mutual exclusion of run_once.

register_event_archiverspawn_unique serializes across pods for one job type, but (a) EventArchiver::run_once is pub and invocable directly, and (b) two pods registering with different JobTypes (easy misconfiguration) run concurrently. Two interleaved runs is not just wasted work:

  • Run B can read an export page after run A already recorded a chunk and deleted part of the span → B materializes A's archived events as placeholder lines in its own chunk files.
  • Roll boundaries can differ between the runs (placeholder recorded_at differs → byte sizes differ → different rollover points → different paths), so ON CONFLICT (path) never fires and the manifest ends up with overlapping chunks whose contents disagree.

Suggest SELECT pg_advisory_xact_lock(hashtext('<prefix>persistent_outbox_archive_chunks')) at the top of the run (or a try_ variant that no-ops), plus a defense-in-depth contiguity guard in record_archive_chunk: only insert when min_sequence = COALESCE((SELECT MAX(max_sequence) FROM ...), 0) + 1, so a lost race cannot commit an overlapping chunk.

Comment thread src/archive/reader.rs Outdated
line.decode::<P>()?;
let delivery = PersistentDelivery::from(item);
let sequence = delivery.sequence();
if sequence <= start_after || sequence > watermark {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern — output is not monotonic by construction. This filter dedupes against start_after but not against last_sent. If the manifest ever contains overlapping chunks (see the run_once concurrency comment), a line re-covering an already-sent sequence is sent again, last_sent rewinds (line 102), and the chunk-tail guard can then emit placeholders for sequences already delivered as real events. PersistentOutboxListener::maybe_add_to_cache inserts by sequence with replace semantics — so a late placeholder can overwrite a not-yet-consumed real event in the listener's local cache.

One-line hardening: if sequence <= last_sent || sequence > watermark { continue; } makes the reader's stream monotonic regardless of manifest state. Cheap insurance for an invariant that otherwise depends on writer-side discipline.

Comment thread src/archive/reader.rs Outdated
if line.is_empty() {
continue;
}
let line = ArchiveEventLine::parse(line)?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern — one unparsable line permanently wedges every consumer below the watermark: parse error → the whole backfill aborts → 5s delay → the listener re-requests → same line again. Real backends' whole-object PUT makes torn writes unlikely, but corruption/manual edits happen, and the manifest carries no integrity metadata (path + range only) to even detect it.

Suggestions, in increasing order of ambition: (a) document the operator playbook for a corrupt chunk (symptom: obix.persistent_cache.archive_backfill_failed every ~5s with a Codec error; fix: repair/replace the object at that path — the path is the format of record); (b) store byte length (ideally a checksum) per chunk in the manifest and verify on read, which also gives operators a bucket-audit tool; (c) degrade a torn final line to the existing chunk_short bridge instead of failing the stream.

Comment thread src/archive/boundary.rs Outdated
after: EventSequence,
) -> Result<Vec<ArchiveBoundary>, ArchiveError> {
let today = self.clock.today();
let Some(last_eligible_date) = today.checked_sub_signed(self.retention) else {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern — the effective age floor is retention − 1 day, not retention. Eligible dates are <= today − retention and recorded_before is the start of the next date, so with retention = 1 day an event recorded at 23:59:59 becomes archivable at 00:00:00 — seconds old. At that age it can race a transaction that allocated a sequence inside the day but hasn't committed: the archiver materializes that sequence as a placeholder, prunes the range, and when the transaction later commits its row lands below the watermark — never served from the archive (placeholder is there) and never served from pg (backfills start above the watermark). Silent event loss.

With retention ≥ 2 whole days the race needs a ≥24h-open transaction, so this is mostly about guarding the config edge: enforce a minimum (reject retention < 2 days?), or compute recorded_before from now − retention rather than date arithmetic, and state the no-in-flight-transactions assumption explicitly in the ArchiveBoundaryProvider contract docs.

Also worth noting in the doc: NaiveDate::checked_sub_signed truncates sub-day components, so retention = 36h silently behaves as 1 day — compounding the edge above.

Comment thread migrations/20251204130225_obix_setup.sql
Comment thread obix-macros/src/tables.rs
// History is bucketed by UTC date of recorded_at (the AT TIME
// ZONE makes the bucketing independent of the session TimeZone);
// the date becomes the boundary label.
let archivable_boundaries_query = format!(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit / efficiencyWHERE sequence > $1 AND recorded_at < $2 GROUP BY date has no usable index for the recorded_at filter, so each call scans every live row above the watermark (the whole retention window). With the default boundaries_per_run = 1 + RescheduleNow, a first catch-up runs this full scan once per historical day — and discards all but one boundary of the list it just computed each time. Cheap fix: sweep everything pending_boundaries returned within one run_once (the list is already in hand), or raise the default. An index on recorded_at is probably overkill for steady state.

@bodymindarts

Copy link
Copy Markdown
Member

IMO we should first partition the table in PG and then write the to-cold-storage code based on the partitioned architecture.

Partitions are then natural boundaries that can be DROPed or exported without as much boilerplate.

Some research on it: https://github.com/GaloyMoney/drua-library/blob/main/spaces/obix-dev/impl-order-partition-then-offload.md

I will start coding the partitioning.

@bodymindarts

Copy link
Copy Markdown
Member

Ready: #106

@nicolasburtey
nicolasburtey force-pushed the feat-event-archive branch 3 times, most recently from 2feb9c1 to 94ba5c3 Compare August 5, 2026 16:33
…pruning

Pluggable event archiver that exports settled spans of
persistent_outbox_events to object storage (JSONL, optional gzip) and
records them in a manifest table. A configurable ArchiveBoundaryProvider
certifies which history is final; the default is age-based (per-UTC-day
retention). The archive reader transparently serves pre-watermark history
during backfill, bridging coverage holes with placeholders to preserve
the contiguity contract.

Pruning is partition-level, not row-level: once the archive watermark
passes a partition's upper bound, the partition maintainer DETACH+DROPs
the whole partition (O(1) metadata op vs O(n) DELETE that would generate
WAL proportional to the span). Archived rows remain in the live table
until their partition is dropped, but are invisible to readers (the
watermark gates all reads below it) and bounded to at most one
partially-archived partition of transient overlap.

Design:
- EventArchiveStorage trait: in-memory (tests) or plug a real backend
  (S3, GCS). Payload-type-agnostic: export works on raw stored JSON,
  decoding happens on read.
- ArchiveBoundaryProvider trait: default DailyRetentionBoundary draws
  boundaries at the highest sequence of each UTC date older than
  retention. Deployments with a settlement process should plug their
  own marker-based provider.
- EventArchiver::run_once is mutually excluded by a postgres advisory
  lock; record_archive_chunk has a contiguity guard
  (COALESCE(MAX(max_sequence),0)+1 = min_sequence) that rejects
  overlapping chunks.
- The backfill pg-walk re-checks the watermark before every page load
  and routes through the archive reader when it has advanced past the
  current sequence. A degraded SELECT-only path handles the
  misconfigured case (watermark present, no archive reader).
- Reader output is monotonic by construction: sequences at or below
  last_sent or above the watermark are skipped, so overlapping manifest
  chunks cannot rewind the stream or replace a delivered real event
  with a placeholder.
@nicolasburtey

Copy link
Copy Markdown
Member Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0ec87d8. Configure here.

}
continue;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Archive prune races placeholder inserts

High Severity

The per-page watermark check is not atomic with the following load_next_page call. load_next_page still gap-fills by INSERTing placeholders, so a concurrent archiver prune — or an archive stream that stops early with current_sequence still below the watermark (listener drop, watermark lookup fail-open) — can write placeholders for archived sequences into the live table and deliver them as rolled-back events.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 0ec87d8. Configure here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The backfill can no longer call load_next_page below a known archive watermark, because that path gap-fills by INSERTing placeholder rows for missing sequences — which below the watermark would be written for archived (partition-dropped) events and delivered as rolled-back transactions.

Changes in handle_backfill_request / serve_archived_up_to_watermark:

  1. Early-exit archive stream: if stream_archived returns Ok(last_sent) with last_sent < watermark (listener dropped mid-stream, or chunks not fully served), the continuation is now degraded_below: Some(watermark) — the pg walk below the watermark is SELECT-only (load_events_in_range, in-memory placeholders, zero DB writes), never load_next_page.
  2. Fail-open watermark lookup: on archive_watermark error we cannot tell where the archived range ends, so the whole remaining walk degrades to SELECT-only (bounded at the walk head) instead of falling through to a gap-filling page load.
  3. Concurrent advance: the per-page re-check is retained, and with partition-level pruning (no row-level DELETE) the residual not-atomic window requires a partition drop in the microseconds between check and load — documented.

Also added backfill_listener_drop_below_watermark_writes_no_placeholders: a cold outbox streams from an archive whose partition was dropped, the listener drops mid-stream, and the assertion is that zero placeholder rows land below the watermark. Verified the test fails without the fix (placeholders written) and passes with it.

Note: this now sits on the current branch state (rebased on #107's pg_notify hardening).

…pruning

Pluggable event archiver that exports settled spans of
persistent_outbox_events to object storage (JSONL, optional gzip) and
records them in a manifest table. A configurable ArchiveBoundaryProvider
certifies which history is final; the default is age-based (per-UTC-day
retention). The archive reader transparently serves pre-watermark history
during backfill, bridging coverage holes with placeholders to preserve
the contiguity contract.

Pruning is partition-level, not row-level: once the archive watermark
passes a partition's upper bound, the partition maintainer DETACH+DROPs
the whole partition (O(1) metadata op vs O(n) DELETE that would generate
WAL proportional to the span). Archived rows remain in the live table
until their partition is dropped, but are invisible to readers (the
watermark gates all reads below it) and bounded to at most one
partially-archived partition of transient overlap.

Design:
- EventArchiveStorage trait: in-memory (tests) or plug a real backend
  (S3, GCS). Payload-type-agnostic: export works on raw stored JSON,
  decoding happens on read.
- ArchiveBoundaryProvider trait: default DailyRetentionBoundary draws
  boundaries at the highest sequence of each UTC date older than
  retention. Deployments with a settlement process should plug their
  own marker-based provider.
- EventArchiver::run_once is mutually excluded by a postgres advisory
  lock; record_archive_chunk has a contiguity guard
  (COALESCE(MAX(max_sequence),0)+1 = min_sequence) that rejects
  overlapping chunks.
- The backfill pg-walk re-checks the watermark before every page load
  and routes through the archive reader when it has advanced past the
  current sequence. A degraded SELECT-only path handles the
  misconfigured case (watermark present, no archive reader).
- Reader output is monotonic by construction: sequences at or below
  last_sent or above the watermark are skipped, so overlapping manifest
  chunks cannot rewind the stream or replace a delivered real event
  with a placeholder.

@bodymindarts bodymindarts left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLAUDE re-review (head 36f8715) — re-check of the 2026-08-03 CHANGES_REQUESTED (blocker + 5 concerns) plus a fresh pass on the updated diff.

Short version: the blocker is genuinely fixed, and the design got materially stronger by pivoting to partition-level pruning (rebased onto #106) — archived rows now stay in the live table until their whole partition drops, which removes the row-DELETE that made the original race easy to hit. All three requested tests landed, plus two more. 4 of 5 concerns are fully/largely resolved with the exact mechanisms asked for. Not hard-blocking; I'd approve once the migration-checksum question (concern e) is answered, and I'd like N1 considered before merge since it guards an irreversible DROP.


1) Prior-review re-check

🔴 BLOCKER — archive-prune vs backfill pg-walk race → FIXED

handle_backfill_request (src/out/persistent/cache.rs:381-514) now re-checks the watermark before every page via serve_archived_up_to_watermark (:282-356), which reads archive_watermark on each call. Sub-watermark reads are routed to the archive reader (stream_archived); the placeholder-writing load_next_page (:487) is reached only when degraded_below == None, i.e. current_sequence >= a freshly-read watermark. The two dangerous early-exit paths both degrade to SELECT-only (load_events_in_range, in-memory placeholders, zero writes): listener-drop mid-stream (:316-329) and watermark-lookup failure (:296-302). The partition-level pruning pivot reinforces this — the row-level DELETE is gone.
Tests: backfill_rechecks_watermark_when_archiver_prunes_mid_walk (tests/archive.rs:563) is exactly the interleave I asked for (sweep prunes a middle span while a cold outbox is stalled mid-page; the just-pruned seq 4 arrives real, not a placeholder); backfill_listener_drop_below_watermark_writes_no_placeholders (:647) asserts zero sub-watermark rows written. 👍
Residual (LOW, documented): the per-page re-check and load_next_page aren't atomic. But for a placeholder to be written, the watermark must advance ~a full partition width (2M) past current_sequence and that partition must drop, all in the check→load window — practically unreachable. Fine to accept as documented.

🟡 (a) No mutual exclusion on run_onceFIXED

run_once (src/archive/archiver.rs:56-86) takes pg_try_advisory_lock(hashtext($1)) keyed by the outbox channel, releases after the run, and reports zero progress if it can't acquire (record_archive_run_skipped_locked). Plus the defense-in-depth contiguity guard in record_archive_chunk (obix-macros/src/tables.rs:286-307): inserts only when COALESCE(MAX(max_sequence),0)+1 = $2, else recorded=falseArchiveError::OverlappingChunk. Both mechanisms I suggested. Minor test gap: no direct two-concurrent-archivers test asserting OverlappingChunk; the guard is only exercised indirectly (idempotency + storage-failure retry-off-watermark).

🟡 (b) Reader not monotonic-by-construction → FIXED

src/archive/reader.rs:116: if sequence <= last_sent || sequence > watermark { continue; } — the exact one-liner, with the intent documented at :111-115.

🟡 (c) DailyRetentionBoundary age floor = retention − 1 dayFIXED

src/archive/boundary.rs:95-99 now computes recorded_before = floor_to_utc_date(now − retention), which is ≤ now − retention, so every archived event is ≥ retention old (flooring only makes it more conservative; sub-day retentions round up to a whole day rather than truncating down). The no-in-flight-transactions contract is now spelled out on the trait (:27-33) and impl (:85-94). The hard min-retention reject wasn't added, but the floor correction + documented contract resolve the silent-loss core.

🟡 (d) One corrupt line wedges all consumers; no integrity metadata → LARGELY ADDRESSED (2 of 3)

  • Torn final line degradation (my suggestion c): reader.rs:101-104 degrades a torn tail to the placeholder bridge (record_archive_line_torn / chunk_short); mid-file corruption still fails loud (:105) by design. 👍
  • Operator runbook (suggestion a): src/archive/mod.rs:25-42 "Operator notes" documents symptom + repair (delete the manifest row, re-sweep). 👍
  • Checksum/length in manifest (suggestion b): not done — manifest is path + min + max + created_at only; no integrity verification on read. Acceptable given whole-object PUT, but it remains the one open sub-item, and there's no test covering the torn-tail / corrupt-line paths.

🟡 (e) Manifest folded into the already-released setup migration → STILL OPEN (by-design, unconfirmed)

The diff vs main appends only persistent_outbox_archive_chunks (+ index) to the same dated migrations/20251204130225_obix_setup.sql. That file was already re-released as part of 0.5.0 and 0.6.0 (both cut releases per the ci(release) commits), so any DB that applied a prior release will fail sqlx::migrate! with a VersionMismatch. #106 established this same mutable-migration pattern, so this is clearly a deliberate stance — but 0.6.0 is a real release, and the original thread was never answered. Please confirm explicitly that no environment (incl. downstream staging that runs obix's migrations) has applied a prior version, or split this into a new dated migration (near-zero cost). This is the weakest-resolved item and the one thing I'd want answered before merge.

🔵 Nit — list_archivable_boundaries full scan + boundaries_per_run=1 discarding the list → ADDRESSED

Default boundaries_per_run raised 1 → 100 (src/archive/mod.rs:200); run_locked now sweeps everything a single pending_boundaries returned up to the cap (archiver.rs:101-113) instead of re-listing per span; idx_..._max_sequence added (migration :48). The boundaries query still lacks a recorded_at index (full retention-window scan per run), but at 100/run the catch-up cost is bounded. (Note the PR body still says "default 1" — see N5.)


2) New findings on current head

N1 🔵 (should-consider before merge) — prune_archived derives the drop decision from the partition name, not its catalog bounds. src/out/partition/mod.rs:267-270: it drops {table}_p{k} when (k+1)*DEFAULT_PARTITION_WIDTH <= watermark, parsing k from the relname. Every creation path uses that fixed-width formula today, so it's internally consistent — but the safety of an irreversible DROP TABLE rests on a naming convention rather than the actual relpartbound. A partition ever created with non-standard bounds (manual op, or future variable-width work) would be dropped with live, unarchived rows → silent data loss. Cheap hardening: gate the drop on the real upper bound read from pg_class / pg_get_expr(relpartbound, oid) rather than on k.

N2 🔵 (ops) — pruning requires both the archiver job and the partition maintainer to be registered. prune_archived only runs inside PartitionMaintainerJobRunner (src/out/partition/job.rs:135). A deployment that calls register_event_archiver but not register_partition_maintainer archives correctly and advances the watermark, but never drops partitions — the live table grows unbounded, silently defeating the disk-reclamation half of the feature. Worth an explicit rollout note (and maybe a startup warning when archive is configured but no maintainer is registered).

N3 🔵 — archived rows stranded in DEFAULT are never reclaimed by prune. If the maintainer fell behind and rows spilled into {table}_default, then got archived, prune_archived (which only drops {table}_p{k}) can't reclaim them until recover_default moves them into explicit partitions. Pure disk leak, not correctness (sub-watermark reads go to the archive). Worth a line in the recover_default runbook.

N4 🔵 (nit) — placeholder recorded_at uses the wall clock, not the config clock. cache.rs:358-366 (placeholder_delivery) uses chrono::Utc::now(), whereas the archive reader's bridge_hole uses self.clock.now() (reader.rs:184). Cosmetic (placeholder payload is None), but inconsistent with the clock injection the rest of the crate/tests rely on.

N5 🔵 (doc) — PR description drift. The body says boundaries_per_run "default 1" (it's 100) and describes the write path as "manifest insert + event deletion in a single statement (WITH deleted AS (DELETE …) INSERT)". Rows are no longer deleted there — pruning is partition-level and record_archive_chunk is manifest-insert-only with a contiguity guard (obix-macros/src/tables.rs:273-307; confirmed no DELETE FROM in the archive path). The "Write side" / BREAKING bullets should be updated so downstream readers don't expect row-level deletion.


3) Composition with recently-landed work (2026-08-03 → 08-06)

  • #106 (range-partition Stage 1, merged 08-04): this PR is now built on it — partition-level pruning (DETACH/DROP) is the archive's prune complement, prune_archived gated on the watermark. Clean fit.
  • #107 (pg_notify trust boundary, merged 08-05): branch is rebased on it (merge-base == origin/main == 1da6a95); the forged-notification clamp (cache.rs:717-780) is preserved intact and archival fires no NOTIFY, so no interaction.
  • #111 (default ids to uuidv7, merged 08-04): no conflict — the migration change here is only the appended archive_chunks table; the gen_random_uuid() column defaults match main (the uuidv7 change is code-side), so merging does not revert it.

Verdict: 💬 comment (blocker cleared; not hard-blocking)

Strong, well-tested fix. The blocker and 4/5 concerns are resolved with the mechanisms requested. Before merge I'd like: (1) an explicit answer on concern (e) — the migration-checksum assumption; and (2) N1 considered, since it guards an irreversible DROP. N2–N5 are low/doc. Nothing here is a correctness blocker on its own — happy to flip to approve once (e) is confirmed.

Comment thread src/out/partition/mod.rs
let should_drop = relname
.strip_prefix(&prefix)
.and_then(|k| k.parse::<u64>().ok())
.is_some_and(|k| (k + 1) * DEFAULT_PARTITION_WIDTH <= watermark);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

N1 🔵 (should-consider before merge) — the drop decision is derived from the partition name (k parsed from {table}_p{k}), not the partition’s actual relpartbound. Every creation path uses the fixed-width formula today so this is internally consistent, but the safety of an irreversible DROP TABLE rests on a naming convention. A partition ever created with non-standard bounds (manual op / future variable-width work) would be dropped here with live, unarchived rows → silent loss. Cheap hardening: gate the drop on the real upper bound from pg_class/pg_get_expr(relpartbound, oid) instead of (k+1)*WIDTH.

-- watermark: everything at or below it must be read from the archive.
-- Any grouping label (e.g. a calendar date) is encoded in a chunk's
-- path; grouping semantics belong to the deployment, not to obix.
CREATE TABLE persistent_outbox_archive_chunks (

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern (e) — still open. This appends the manifest table to a dated, already-released migration: 20251204130225_obix_setup.sql shipped in 0.5.0 and 0.6.0 (both cut releases). Any DB that applied a prior release fails sqlx::migrate! with a VersionMismatch on the changed checksum. #106 set the same mutable-migration precedent, so this is a deliberate stance — but please confirm explicitly that no environment (incl. downstream staging that runs obix migrations) has applied a prior version, or split this into a new dated migration (near-zero cost). This is the one item I’d want answered before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants