Skip to content

chore: add coverage-guided fuzzing for decode_persistent_event - #115

Merged
nicolasburtey merged 4 commits into
mainfrom
chore/fuzz-decode-persistent-event
Aug 9, 2026
Merged

nicolasburtey merged 4 commits into
mainfrom
chore/fuzz-decode-persistent-event

Conversation

@nicolasburtey

@nicolasburtey nicolasburtey commented Aug 8, 2026 •

Copy link
Copy Markdown
Member

Adds coverage-guided fuzzing to obix, using the shared fuzz_job() from galoy-concourse-shared (same pattern as es-entity#192).

The target: fuzz_decode_persistent_event

obix::decode_persistent_event is the single chokepoint that decodes every persistent outbox row read back from Postgres into the consumer's event type — the most untrusted data obix reads back (rows from a different/future event enum, hand-edits, bad migrations). This is the obix analog of es-entity's fuzz_event_hydration.

Its doc spells out the fuzzer-checkable invariant:

a single poison row previously wedged the whole pipeline in a hot panic/retry loop — it must never panic; an undecodable row becomes an honest Err(UndecodableEventError).

The harness asserts:

  1. never panics (libFuzzer reports any panic as a crash);
  2. on Err, the carried failure.raw is the exact input value (honest copy, not truncated) and the serde error is non-empty;
  3. the None-placeholder path is a total function (always Ok, no payload, regardless of surrounding bytes);
  4. row metadata (id / sequence / recorded_at / tracing_context) round-trips on both arms.

Everything else in obix's parse surface (InboxEvent::payload, InboxEventStatus::from_str, the NOTIFY parses — which main's #107 already hardened) is thin serde wrapping or already safe — not worth dedicated targets.

CI: shared fuzz_job() (commit 2)

Rather than a hand-rolled inline job, this consumes the shared helpers from galoy-concourse-shared (vendir ref bumped 8653c10 → 9184832):

  • ci/pipeline.yml: ~110-line inline fuzz job/resources/types → a single #@ fuzz_job() + shared fuzz_time_resource() / zenduty_notification_resource() / zenduty_resource_type().
  • ci/fuzz.sh deleted → uses vendored ci/vendor/tasks/fuzz.sh, which auto-discovers targets via cargo fuzz list (adding a target later is just a new fuzz_targets/*.rs, no list-editing). Makefile + flake.nix (nix run .#fuzz) point at it.

Cadence is now the shared default: weekly Saturday 06:00 UTC, 24h per run, Zenduty on crash. Corpus self-bootstraps (first run fuzzes from scratch) and persists to GCS (obix-artifacts/fuzz-corpus/) via gsutil in a google/cloud-sdk step (the Concourse gcs-resource type is dropped — nixpkgs gsutil is broken).

The vendir refresh for obix touches only pipeline-fragments.lib.yml (fuzz helpers) + tasks/fuzz.sh, plus an unrelated helpers.sh → rust-helpers.sh rename that only affects test-integration/test-bats/check-code.sh — none of which obix's pipeline invokes.

Files

File Change
fuzz/Cargo.toml, fuzz/fuzz_targets/fuzz_decode_persistent_event.rs new — the cargo-fuzz crate + target
ci/pipeline.yml #@ fuzz_job() + shared resource/type helpers
ci/vendir.yml / vendir.lock.yml shared ref 8653c10 → 9184832
ci/vendor/pipeline-fragments.lib.yml, ci/vendor/tasks/fuzz.sh vendored (fuzz helpers + runner)
ci/fuzz.sh deleted (replaced by vendored runner)
Makefile, flake.nix make fuzz / nix run .#fuzz → ci/vendor/tasks/fuzz.sh
.gitignore fuzz/corpus/, fuzz/artifacts/
ci/values.yml zenduty_webhook_url

Verified

  • cargo fuzz build --sanitizer=none compiles.
  • ytt -f ci renders the shared 3-step job (restore-corpus → fuzz@86400s → store-corpus; GCS prefix obix-artifacts/fuzz-corpus).
  • bash ci/vendor/tasks/fuzz.sh auto-discovers 1 target and runs it.
  • nix eval .#fuzz resolves.
  • A 1-hour run did ~311M executions, zero crashes.

Before merging / repipe

The CI job reuses vault paths es-entity already uses: ((staging-gcp-creds.*)) and ((zenduty.webhook_url)). Confirm the obix Concourse team can read those, or swap per the shared lib's note. If unavailable, the local / nix run story still works fully — only CI cross-run corpus accumulation needs them.

Draft until that credential question is confirmed and CI is green.


Note

Low Risk
Mostly CI/dev tooling and a fuzz harness over existing decode logic; no production runtime behavior changes beyond stronger regression coverage for poison-row handling.

Overview
Introduces coverage-guided fuzzing for the outbox decode path, wired for local runs and weekly Concourse CI.

A new fuzz/ crate adds fuzz_decode_persistent_event, which feeds arbitrary bytes through decode_persistent_event and checks it never panics on poison rows, returns honest UndecodableEventError copies on failure, and treats invalid JSON as the None-placeholder path.

CI adopts shared fuzz_job() from vendored galoy-concourse-shared (ref bumped to 4c9071e): weekly Saturday run (default 24h), corpus restore/store to GCS via gsutil, and Zenduty on failure. The vendored ci/vendor/tasks/fuzz.sh auto-discovers targets; make fuzz and nix run .#fuzz use the same script (default 60s locally).

Also gitignores fuzz/corpus/ and fuzz/artifacts/, adds zenduty_webhook_url in CI values, and puts the Concourse fly CLI on the dev shell PATH.

Reviewed by Cursor Bugbot for commit 990b28f. Configure here.

Mirrors the fuzz setup in es-entity: a standalone cargo-fuzz crate, a shared
ci/fuzz.sh (single source of truth for `make fuzz`, `nix run .#fuzz`, and the
CI job), and a nightly Concourse job with a GCS-persisted evolving corpus.

The single target fuzzes obix::decode_persistent_event — the chokepoint that
decodes every persistent outbox row read back from Postgres, and the most
untrusted data obix reads back. Its contract is that a poison row must never
panic (a single poison row once wedged the pipeline in a hot panic/retry loop)
and an undecodable row becomes an honest Err(UndecodableEventError). The
harness asserts: never panics; on Err the carried failure.raw is the exact
input value (honest copy); the None-placeholder path is a total function
(always Ok, no payload); and row metadata round-trips on both arms. This is
the obix analog of es-entity's fuzz_event_hydration.

Verified on main (0.7.2-dev): cargo fuzz build --sanitizer=none compiles;
ytt -f ci renders; a 1h run did ~311M executions with zero crashes.

The CI job reuses es-entity's staging-gcp-creds bucket (namespaced under
obix-artifacts/fuzz-corpus/) and zenduty webhook; swap if unavailable to this
team.

A second commit on an earlier draft added NOTIFY-payload fuzz targets, but
main's #107 ("stop trusting pg_notify payloads — notifications are hints, not
transport") already removed the .expect() that motivated them, leaving both
NOTIFY parses as plain 2-field serde_json calls — serde_json is already
exhaustively fuzzed, so those targets carried near-zero marginal value and are
dropped. decode_persistent_event remains the one genuinely valuable target.
@nicolasburtey
nicolasburtey force-pushed the chore/fuzz-decode-persistent-event branch from 2f8786c to 4b47f00 Compare August 9, 2026 01:25
Mirrors es-entity#192: migrate obix's hand-rolled inline fuzz job to the
shared fuzz_job() / fuzz_time_resource() / zenduty_* helpers now in
galoy-concourse-shared (PR #20, vendir ref 8653c10 -> 9184832).

- ci/vendir.yml + vendir.lock.yml: bump the shared ref.
- ci/pipeline.yml: the ~110-line inline fuzz job/resources/types -> a single
  `#@ fuzz_job()` call + shared resource/type helpers. Drops the no-longer-
  needed gcr_resource_type / public_docker_registry imports (the shared job
  uses the google/cloud-sdk image + gsutil directly, not a Concourse
  gcs-resource type).
- ci/fuzz.sh DELETED -> use the vendored ci/vendor/tasks/fuzz.sh, which
  auto-discovers targets via `cargo fuzz list` (no hardcoded names). Makefile
  + flake.nix (`nix run .#fuzz`) point at it.

Behavior change from the inline job: cadence is now the shared default —
weekly Saturday 06:00 UTC for 24h (was daily / 3600s) — and the corpus is
self-bootstrapping (first run fuzzes from scratch, no manual seed). GCS
access moved off the (broken) Concourse gcs-resource type onto gsutil in a
google/cloud-sdk step, same as es-entity.

The vendir refresh for obix touches only pipeline-fragments.lib.yml (fuzz
helpers) + tasks/fuzz.sh, plus the unrelated helpers.sh -> rust-helpers.sh
rename — which affects only test-integration/test-bats/check-code.sh, none
of which obix's pipeline invokes, so it's harmless here.

Verified: `ytt -f ci` renders (shared 3-step job: restore-corpus -> fuzz ->
store-corpus, GCS prefix obix-artifacts/fuzz-corpus); `nix eval .#fuzz`
resolves; `bash ci/vendor/tasks/fuzz.sh` auto-discovers 1 target and runs it.

Credentials still reuse es-entity's shared ((staging-gcp-creds.*)) and
((zenduty.webhook_url)); confirm the obix Concourse team can read those.
Pulls in PR #21 (fix/fuzz-script-path): the shared fuzz_job ran
`bash pipeline-tasks/ci/vendor/tasks/fuzz.sh`, but the fuzz task does
`cd repo` with no `pipeline-tasks` input, so it failed at runtime on
Concourse with "No such file or directory". The fix makes the path
repo-relative (`ci/vendor/tasks/fuzz.sh`), which the repo input already
contains. Invisible to local `ytt` render tests — only surfaces on the CI
runner.

Refresh touches only ci/vendor/pipeline-fragments.lib.yml (the one-line path
fix); no obix-side changes needed. ytt -f ci renders; rendered fuzz task now
reads `bash ci/vendor/tasks/fuzz.sh`.
Mirrors lana-bank's concourseFly derivation: fetches the fly binary from
ci.galoy.io (per-platform SRI hashes) so `fly` / `repipe` work in `nix
develop` without a manual install. Needed to repipe the new fuzz job from
this repo's dev shell.
@nicolasburtey
nicolasburtey marked this pull request as ready for review August 9, 2026 13:50
@nicolasburtey
nicolasburtey merged commit db06356 into main Aug 9, 2026
5 checks passed
@nicolasburtey
nicolasburtey deleted the chore/fuzz-decode-persistent-event branch August 9, 2026 13:50

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 990b28f. Configure here.

Comment thread flake.nix
process-compose
ytt
curl
concourseFly

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI fuzz job lacks cargo-fuzz

High Severity

The Concourse fuzz job runs ci/vendor/tasks/fuzz.sh inside nix develop, but the dev shell’s nativeBuildInputs never includes cargo-fuzz. The script then falls through to cargo install cargo-fuzz --locked on every CI run. That path is network- and lockfile-dependent, while nix run .#fuzz already puts pkgs.cargo-fuzz on PATH — so local and CI setups disagree, and the weekly job can fail before fuzzing starts.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 990b28f. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant