feat(outbox): archive old outbox events to cold storage - #795
Draft
nicolasburtey wants to merge 1 commit into
Draft
nicolasburtey wants to merge 1 commit into
nicolasburtey wants to merge 1 commit into
Conversation
Port of GaloyMoney/lana-bank#7757 (obix event-archive wiring, GaloyMoney/obix#105) to the cala outbox, using the obix branch as a git dependency until the archive feature is released on crates.io: - cala-ledger: OutboxArchiveConfig { storage, retention_days = 3, path_prefix = "outbox-archive/cala/" } on CalaLedgerConfig; the consumer supplies an obix::EventArchiveStorage backend (GCS, S3, local fs) and settled days are swept out of postgres to JSONL chunks, with transparent read fallback for pre-watermark history - CalaLedger::register_outbox_archiver registers the "cala.outbox.archiver" job (one settled day per run, rescheduling while catching up) on the app's job executor - boundary: obix's age-based DailyRetentionBoundary over CalaMailboxTables - migration: cala_persistent_outbox_archive_chunks manifest folded into the cala obix setup migration (in-place, matching the recent pre-production edits of that migration) - .sqlx offline cache regenerated (5 new archive queries) Not yet: a released obix version (revert to crates.io once GaloyMoney/obix#105 ships); lana's dw-outbox-relay can then drop its obix-cala pin.
Contributor
📊 Performance ReportCommit: b265562 Cala Performance Benchmark Results (non-representative)Criterion Benchmark Results (single-threaded)
Load Testing Results (parallel-execution)
Note: Performance results may vary based on system resources and database state. Last updated by commit b265562 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of GaloyMoney/lana-bank#7757 to the cala outbox (the cala side was explicitly out of scope there: "cala outbox archiving — needs cala on the new obix"). Wires the obix event-archive feature (GaloyMoney/obix#105, git branch dependency until it is released) into
cala-ledger.Summary
cala-ledger/src/outbox/config.rs:OutboxArchiveConfig { storage, retention_days = 3, path_prefix = "outbox-archive/cala/" }. cala is a library without a cloud-storage crate, so the consumer supplies the backend viaobix::EventArchiveStorage(GCS, S3, local fs;InMemoryArchiveStoragefor tests). The boundary is obix's age-basedDailyRetentionBoundaryoverCalaMailboxTables.CalaLedgerConfig: optional.outbox_archive(...)— when set, settled spans of history are swept from postgres to object storage as JSONL chunks and pre-watermark reads transparently fall back to the archive.CalaLedger::register_outbox_archiver(&mut job::Jobs): registers thecala.outbox.archiverjob on the app's job executor (cala owns no executor of its own) — sweeps one settled day per run, rescheduling while catching up.cala_persistent_outbox_archive_chunksmanifest folded into the existing cala obix setup migration (in-place edit, matching the recent pre-production edits of that file)..sqlx: offline cache regenerated (the 5 new derive-generated archive queries).Not in scope / follow-ups
cala-ledger.dw-outbox-relaycan then drop itsobix-cala =0.5.0pin and unify on one obix version.EndOfDaySettledfollow-up.Test plan
Wrote AccountEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/account_event_schema.json
Wrote AccountSetEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/account_set_event_schema.json
Wrote JournalEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/journal_event_schema.json
Wrote TransactionEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/transaction_event_schema.json
Wrote EntryEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/entry_event_schema.json
Wrote TxTemplateEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/tx_template_event_schema.json
Wrote VelocityLimitEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/velocity_limit_event_schema.json
Wrote VelocityControlEvent schema to /Users/n/code/galoy/cala/cala-ledger/schemas/velocity_control_event_schema.json
git diff --exit-code cala-ledger/schemas clean
EOF
)