chore: add coverage-guided fuzzing for decode_persistent_event - #115
Conversation
Mirrors the fuzz setup in es-entity: a standalone cargo-fuzz crate, a shared ci/fuzz.sh (single source of truth for `make fuzz`, `nix run .#fuzz`, and the CI job), and a nightly Concourse job with a GCS-persisted evolving corpus. The single target fuzzes obix::decode_persistent_event — the chokepoint that decodes every persistent outbox row read back from Postgres, and the most untrusted data obix reads back. Its contract is that a poison row must never panic (a single poison row once wedged the pipeline in a hot panic/retry loop) and an undecodable row becomes an honest Err(UndecodableEventError). The harness asserts: never panics; on Err the carried failure.raw is the exact input value (honest copy); the None-placeholder path is a total function (always Ok, no payload); and row metadata round-trips on both arms. This is the obix analog of es-entity's fuzz_event_hydration. Verified on main (0.7.2-dev): cargo fuzz build --sanitizer=none compiles; ytt -f ci renders; a 1h run did ~311M executions with zero crashes. The CI job reuses es-entity's staging-gcp-creds bucket (namespaced under obix-artifacts/fuzz-corpus/) and zenduty webhook; swap if unavailable to this team. A second commit on an earlier draft added NOTIFY-payload fuzz targets, but main's #107 ("stop trusting pg_notify payloads — notifications are hints, not transport") already removed the .expect() that motivated them, leaving both NOTIFY parses as plain 2-field serde_json calls — serde_json is already exhaustively fuzzed, so those targets carried near-zero marginal value and are dropped. decode_persistent_event remains the one genuinely valuable target.
2f8786c to
4b47f00
Compare
Mirrors es-entity#192: migrate obix's hand-rolled inline fuzz job to the shared fuzz_job() / fuzz_time_resource() / zenduty_* helpers now in galoy-concourse-shared (PR #20, vendir ref 8653c10 -> 9184832). - ci/vendir.yml + vendir.lock.yml: bump the shared ref. - ci/pipeline.yml: the ~110-line inline fuzz job/resources/types -> a single `#@ fuzz_job()` call + shared resource/type helpers. Drops the no-longer- needed gcr_resource_type / public_docker_registry imports (the shared job uses the google/cloud-sdk image + gsutil directly, not a Concourse gcs-resource type). - ci/fuzz.sh DELETED -> use the vendored ci/vendor/tasks/fuzz.sh, which auto-discovers targets via `cargo fuzz list` (no hardcoded names). Makefile + flake.nix (`nix run .#fuzz`) point at it. Behavior change from the inline job: cadence is now the shared default — weekly Saturday 06:00 UTC for 24h (was daily / 3600s) — and the corpus is self-bootstrapping (first run fuzzes from scratch, no manual seed). GCS access moved off the (broken) Concourse gcs-resource type onto gsutil in a google/cloud-sdk step, same as es-entity. The vendir refresh for obix touches only pipeline-fragments.lib.yml (fuzz helpers) + tasks/fuzz.sh, plus the unrelated helpers.sh -> rust-helpers.sh rename — which affects only test-integration/test-bats/check-code.sh, none of which obix's pipeline invokes, so it's harmless here. Verified: `ytt -f ci` renders (shared 3-step job: restore-corpus -> fuzz -> store-corpus, GCS prefix obix-artifacts/fuzz-corpus); `nix eval .#fuzz` resolves; `bash ci/vendor/tasks/fuzz.sh` auto-discovers 1 target and runs it. Credentials still reuse es-entity's shared ((staging-gcp-creds.*)) and ((zenduty.webhook_url)); confirm the obix Concourse team can read those.
Pulls in PR #21 (fix/fuzz-script-path): the shared fuzz_job ran `bash pipeline-tasks/ci/vendor/tasks/fuzz.sh`, but the fuzz task does `cd repo` with no `pipeline-tasks` input, so it failed at runtime on Concourse with "No such file or directory". The fix makes the path repo-relative (`ci/vendor/tasks/fuzz.sh`), which the repo input already contains. Invisible to local `ytt` render tests — only surfaces on the CI runner. Refresh touches only ci/vendor/pipeline-fragments.lib.yml (the one-line path fix); no obix-side changes needed. ytt -f ci renders; rendered fuzz task now reads `bash ci/vendor/tasks/fuzz.sh`.
Mirrors lana-bank's concourseFly derivation: fetches the fly binary from ci.galoy.io (per-platform SRI hashes) so `fly` / `repipe` work in `nix develop` without a manual install. Needed to repipe the new fuzz job from this repo's dev shell.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 990b28f. Configure here.
| process-compose | ||
| ytt | ||
| curl | ||
| concourseFly |
There was a problem hiding this comment.
CI fuzz job lacks cargo-fuzz
High Severity
The Concourse fuzz job runs ci/vendor/tasks/fuzz.sh inside nix develop, but the dev shell’s nativeBuildInputs never includes cargo-fuzz. The script then falls through to cargo install cargo-fuzz --locked on every CI run. That path is network- and lockfile-dependent, while nix run .#fuzz already puts pkgs.cargo-fuzz on PATH — so local and CI setups disagree, and the weekly job can fail before fuzzing starts.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 990b28f. Configure here.


Adds coverage-guided fuzzing to obix, using the shared
fuzz_job()from galoy-concourse-shared (same pattern as es-entity#192).The target:
fuzz_decode_persistent_eventobix::decode_persistent_eventis the single chokepoint that decodes every persistent outbox row read back from Postgres into the consumer's event type — the most untrusted data obix reads back (rows from a different/future event enum, hand-edits, bad migrations). This is the obix analog of es-entity'sfuzz_event_hydration.Its doc spells out the fuzzer-checkable invariant:
The harness asserts:
Err, the carriedfailure.rawis the exact input value (honest copy, not truncated) and the serdeerroris non-empty;None-placeholder path is a total function (alwaysOk, no payload, regardless of surrounding bytes);id/sequence/recorded_at/tracing_context) round-trips on both arms.Everything else in obix's parse surface (
InboxEvent::payload,InboxEventStatus::from_str, the NOTIFY parses — which main's #107 already hardened) is thin serde wrapping or already safe — not worth dedicated targets.CI: shared
fuzz_job()(commit 2)Rather than a hand-rolled inline job, this consumes the shared helpers from galoy-concourse-shared (vendir ref bumped
8653c10 → 9184832):ci/pipeline.yml: ~110-line inline fuzz job/resources/types → a single#@ fuzz_job()+ sharedfuzz_time_resource()/zenduty_notification_resource()/zenduty_resource_type().ci/fuzz.shdeleted → uses vendoredci/vendor/tasks/fuzz.sh, which auto-discovers targets viacargo fuzz list(adding a target later is just a newfuzz_targets/*.rs, no list-editing).Makefile+flake.nix(nix run .#fuzz) point at it.Cadence is now the shared default: weekly Saturday 06:00 UTC, 24h per run, Zenduty on crash. Corpus self-bootstraps (first run fuzzes from scratch) and persists to GCS (
obix-artifacts/fuzz-corpus/) viagsutilin agoogle/cloud-sdkstep (the Concoursegcs-resourcetype is dropped — nixpkgsgsutilis broken).The vendir refresh for obix touches only
pipeline-fragments.lib.yml(fuzz helpers) +tasks/fuzz.sh, plus an unrelatedhelpers.sh → rust-helpers.shrename that only affectstest-integration/test-bats/check-code.sh— none of which obix's pipeline invokes.Files
fuzz/Cargo.toml,fuzz/fuzz_targets/fuzz_decode_persistent_event.rsci/pipeline.yml#@ fuzz_job()+ shared resource/type helpersci/vendir.yml/vendir.lock.yml8653c10 → 9184832ci/vendor/pipeline-fragments.lib.yml,ci/vendor/tasks/fuzz.shci/fuzz.shMakefile,flake.nixmake fuzz/nix run .#fuzz→ci/vendor/tasks/fuzz.sh.gitignorefuzz/corpus/,fuzz/artifacts/ci/values.ymlzenduty_webhook_urlVerified
cargo fuzz build --sanitizer=nonecompiles.ytt -f cirenders the shared 3-step job (restore-corpus → fuzz@86400s → store-corpus; GCS prefixobix-artifacts/fuzz-corpus).bash ci/vendor/tasks/fuzz.shauto-discovers 1 target and runs it.nix eval .#fuzzresolves.Before merging /
repipeThe CI job reuses vault paths es-entity already uses:
((staging-gcp-creds.*))and((zenduty.webhook_url)). Confirm the obix Concourse team can read those, or swap per the shared lib's note. If unavailable, the local /nix runstory still works fully — only CI cross-run corpus accumulation needs them.Draft until that credential question is confirmed and CI is green.
Note
Low Risk
Mostly CI/dev tooling and a fuzz harness over existing decode logic; no production runtime behavior changes beyond stronger regression coverage for poison-row handling.
Overview
Introduces coverage-guided fuzzing for the outbox decode path, wired for local runs and weekly Concourse CI.
A new
fuzz/crate addsfuzz_decode_persistent_event, which feeds arbitrary bytes throughdecode_persistent_eventand checks it never panics on poison rows, returns honestUndecodableEventErrorcopies on failure, and treats invalid JSON as theNone-placeholder path.CI adopts shared
fuzz_job()from vendored galoy-concourse-shared (ref bumped to4c9071e): weekly Saturday run (default 24h), corpus restore/store to GCS viagsutil, and Zenduty on failure. The vendoredci/vendor/tasks/fuzz.shauto-discovers targets;make fuzzandnix run .#fuzzuse the same script (default 60s locally).Also gitignores
fuzz/corpus/andfuzz/artifacts/, addszenduty_webhook_urlin CI values, and puts the ConcourseflyCLI on the dev shell PATH.Reviewed by Cursor Bugbot for commit 990b28f. Configure here.