feat(audit): org-wide audit log with actor attribution and durable queue delivery - #683
Open
Makisuo wants to merge 21 commits into
Open
feat(audit): org-wide audit log with actor attribution and durable queue delivery#683Makisuo wants to merge 21 commits into
Makisuo wants to merge 21 commits into
Conversation
…eue delivery Adds an append-only audit trail distinguishing users, API keys, agents, and system automation, following the auditlog.dev spec: - audit_log_entries table (migration 0050): actor snapshot + credential refs, outcome (allowed/denied) + denial_reason, before/after change diffs with queryable changed_fields, affected_user, request forensics (request id, origin IP/country), and occurred_at/recorded_at. - Durable delivery through a Cloudflare Queue (audit-events): producers enqueue, the api worker consumes and inserts idempotently; direct DB write as fallback when the binding is absent or the send fails. - CurrentAuditActor context from all three auth layers distinguishes session vs API-key requests; denied attempts (scope/org/surface rejections) are recorded from inside the auth layers. - Recording wired into every v2 mutation handler (with diffs and secret redaction), the issue-workflow choke point (agent vs user attribution with on-behalf-of), and register_agent. - GET /v2/audit_log: cursor-paginated, filterable by actor, outcome, action, resource, changed field, request id, and time window; new audit_log:read scope and alog_ public IDs. - Settings > Audit Log tab with actor/outcome filters, denied badges, change summaries, and load-more pagination. - Hourly retention sweep (AUDIT_LOG_RETENTION_DAYS, default 400) in the api worker's existing retention cron.
- Tagged AuditQueueSendError instead of a global Error in the queue send failure channel. - compactAuditChanges takes static placeholder strings instead of unknown-typed summarizer functions; null still survives as null. - destinationObservableValue returns a concrete union, not unknown. - iOS OpenAPI spec regenerated for the new /v2/audit_log path.
…, indexes From four review passes over the audit log: - Admin-gate GET /v2/audit_log: entries carry every member's activity, denial history and origin IP for the retention window. The settings tab hides for non-admins to match. - Coalesce denied api.request records per (org, key, method+path, reason) in a 60s isolate-local window. The v1 auth layer has no rate limiter, so a client looping mis-scoped requests could otherwise amplify into unbounded queue messages, rows and warn logs. Both auth layers now share one helper, so v1 records the same forensics as v2. - Audit the two v2 denial branches that returned early (MCP-only key, invalid device credential) — the credential-probing case the feature exists to surface. - Bound queue.send with a 2s timeout: a stalling broker must not hang the mutation's response before the direct-write fallback. - Replace blanket catchCause with catchTag/catchDefect so interruption propagates instead of spawning a Postgres insert mid-teardown. - Structural (key-order insensitive) diff comparison; redact userinfo and query strings from audited scrape-target URLs; carry the cause on AuditLogPersistenceError. - Index occurred_at for the retention sweep, the actor-identity columns for the primary 'what did this credential do' query, and a GIN index for changed-field lookups.
…ields
Adding an audited action meant restating what the action already implied:
a free-string `resourceType` echoing the action's own prefix, an inline
`encodePublicId(PublicIdPrefixes.x, id)`, and — for updates — a
hand-assembled diff pipeline. Across 28 call sites the resource pair was
mechanically derivable every time, and nothing checked it: the service's
own test recorded `alert_rule.delete`, a verb that does not exist.
`AuditResources` now declares each resource with its public-ID prefix and
verbs, and `AuditAction` is the derived `${resource}.${verb}` union.
`record`/`recordHttpAudit` take the internal ID and derive `resourceType`
plus the public encoding themselves, so a typo fails the build, a
`resourceId` on an org-singleton resource fails the build, and the
prefix can no longer disagree with the resource. `error_issue` verbs come
from `ErrorIssueEventType.literals` so a new issue event type cannot
produce an undeclared action.
`auditDiff({ fields, summarize, redact, writeOnly })` replaces the
per-handler diff assembly; scrape-targets' update handler goes from ~40
lines of object surgery to one call. Keying `summarize`/`redact` by
`fields` makes the old "remember to `satisfies`" rule structural.
Durability. The audit-events consumer had no dead letter queue and no final-attempt branch, so after five retries Cloudflare dropped the entry with nothing in the logs at the moment it happened. There is now an `audit-events-dlq` queue with no consumer — an entry landing there is a lost record and the point is that it survives — and the consumer logs the hand-off at Error, with the org and action read defensively off the body. It keeps retrying on the final attempt, because acking is what would discard the message instead of routing it. Attribution. The actors row knows who acted, never how, so every mutation reached through an API key or over MCP was recorded as a dashboard session — the MCP middleware set no audit reference at all. `CurrentAuditActor` now carries the surface alongside the credential, all four auth layers stamp it, and the issue-workflow mirror consults it instead of assuming. Maple's own sweeps run as an agent actor, which made auto-close and lease expiry read as a third-party agent over MCP; they are now recorded as `system`, which until today had no writer at all. Coverage. Audited org deletion, warehouse settings (updated, deleted, schema applied), the Slack and PlanetScale integration lifecycles including the metrics-token install, widget credential mint/revoke, investigations, and issue comments — which wrote their event row directly and so bypassed the audit mirror entirely. Secrets stay out: the entries record which credential was installed, never its value. Membership. Members are changed in Clerk, never through Maple's API, which is why `affected_user` had no writers. The Clerk receiver now audits `organizationMembership.*` against the member. Clerk's payload does not name the admin who acted, so the entry is attributed to `system` rather than guessing a user. Enabling the three events in the Clerk dashboard is what turns this on. UI. The list paginates by offset over a newest-first append-only table, so an entry written mid-scroll shifted later pages and made them repeat one row and skip another. The first Load more now pins `until` to the newest entry on screen, freezing the window, and pages are deduped by id on append. The header no longer claims to record "every change". The retention sweep's ctid-addressed delete already landed with the review fixes; verified rather than changed.
HIPAA audit controls cover access, not only change, and an audit trail that grows with every telemetry read belongs in the warehouse, not in the application database. Storage moves from Postgres (`audit_log_entries`, migration 0050, the hourly retention sweep and `AUDIT_LOG_RETENTION_DAYS` are removed) to the Tinybird datasource `audit_log` — ClickHouse migration 0025, ReplacingMergeTree on the entry id so queue redelivery collapses at merge, monthly partitions, six-year TTL (§164.316(b)(2)). Writes go through `WarehouseQueryService.ingest`, which is pinned to the managed pipeline and never a BYO ClickHouse; reads through the new `auditLogEntriesQuery` builder, routed the same way and listed on `INGEST_PINNED_TABLES`. The datasource is hidden from raw SQL and the per-org read JWT so the admin gate on `GET /v2/audit_log` is the only way in. The queue consumer builds the warehouse layer and ingests one batch per org; ack/retry/DLQ semantics are unchanged. Reads are now recorded on three surfaces: - HTTP endpoints annotated `AuditedRead` in the domain contracts — `telemetry.read` on the internal query-engine group, v2 traces/logs/ metrics/error_issues and the v1 error-issue GETs, `session_replay.read` on every replay group. The three auth layers wrap the handler with `withAuditedRead`, recording endpoint, method, path, status and a bounded body snapshot with the request's forensics. - Every MCP tool invocation from any surface (`mcp_tool.called`, with the tool and its parameters), from the executor. - Every raw SQL statement (`telemetry.sql_executed`) from `run_sql`, `inspect_chart_data` and the dashboard's raw-SQL route — a statement the safety pass refuses is recorded as `denied`. Tests use `AuditLogService.layerMemory`, an in-memory implementation with the query's filter and ordering semantics. The local chDB schema moves to v15 with an additive migration edge.
… interrupts `HttpApiGroup.annotate` writes to the group only (endpoint propagation is `annotateEndpoints`), so `withAuditedRead` never saw the action on any group-annotated surface and recorded reads only for the five per-endpoint annotations on the v1 errors group. Both annotation sets are consulted now, endpoint first, and a test drives a group-annotated endpoint, a v2 replay endpoint, an unannotated endpoint, and a failing handler through the wrapper. `ChangedFields` needed the `[:]` JSONPath suffix every other Events-API array column declares; without it the datasource rejects or quarantines rows. The endpoint label used `endpoint.name` (the class name) instead of `identifier`. The comments claimed `LIMIT … BY Id` dedupe the query never had; the service now drops a repeated id within a page and the comments say what actually happens. `recordEventAudit` used `catchCause`, which also swallows interrupts; it catches failures and defects like `record` does. The DateTime64 decoder accepts an ISO rendering, `occurredAtMs` is `Schema.Finite`, and two stale comments about the Postgres table and self-hosted routing are corrected. Regenerated schema artifacts re-pin the local v15 identity.
…jobs main retired lib/effect-cloudflare and took ClickHouse 0025/0026 plus local schema v15/v16 while this branch was open, so all three failures were the same kind of collision: the branch claimed identities main had already spent. - ClickHouse migration renumbered 0025 -> 0027 (`audit_log`); local store bump redone as v16 -> v17 with `scripts/bump-local-schema.ts`, which did not exist when the original v14 -> v15 edge was written by hand. That hand bump is what pinned the wrong active identity in the native probe. - `WorkerEnvironment` repointed from the retired `@maple/effect-cloudflare` to `@maple/infra/worker-runtime`, and `WorkerEnvironment.layer` to the standalone `workerEnvironmentLayer`, in AuditLogService and the audit-events consumer. - Alerting's audit wiring moved into main's new single-module `scheduled.ts` graph, so issue transitions on the cron path still resolve AuditLogService rather than defecting on a missing service. - `run-raw-sql.test.ts` provides `AuditLogService.layerMemory`: runRawSql records `telemetry.sql_executed` on every path, so the service belongs in the harness. - SQL catalog baseline regenerated; the diff is exactly `auditLogEntriesQuery`. Verified: typecheck 40/40, lint, apps/api 214 files, query-engine 62, domain/clickhouse 7, apps/cli 538 tests. The native migration probe needs a built bundle and is left to CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-shake The web bundle budget went red on this branch at 651.0 KB against a 650.0 KB ceiling — main sits at 648.3 KB, so the audit log's +2.7 KB is what tipped it. None of that 2.7 KB is the settings tab: no audit module is in the static graph. It is the contract. `@maple/domain/http/v2`'s barrel re-exports `audit-log`, and startup modules (error-issues, anomalies, alert form-utils) import that barrel, so the entry schema and the V2 group definition landed in two startup chunks. Without `sideEffects: false` the bundler must assume every module behind a barrel matters and keeps all of it — so any new export taxes startup for everyone, whoever adds it. domain is schemas, contracts and branded types: no import-time global mutation, no prototype patching, nothing registered on load. Declaring that lets rolldown drop what a given entry does not reach. 651.0 -> 648.9 KB gzip. domain 693 tests, web 2391 pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every user row rendered as `user_3BfcmIS3bUNV6BfA…` and every API-key row as `key_jCZgpwAYQ3…`, because nothing on the HTTP path ever set `actorLabel` — only the MCP client name and the system surface did. The two actor kinds want opposite treatments, so they get them: - An API key already has its name on the row auth just resolved, so the name is frozen onto the entry at write time, denials included. That is the property an audit trail wants: a key that is revoked an hour later still reads as "Grafana exporter" on the request that was refused. Free — no extra lookup. - A dashboard session has no name to freeze. Clerk's claims carry none, and resolving one per write would put a directory call on every telemetry read — the hottest audited path there is. Those rows are labelled when the log is read, one directory call per page, on an admin-only screen. Reading the log never fails because the directory does: an unconfigured (self-hosted, no Clerk) or unavailable directory logs a warning and the entries keep their ids. A member who has since left the org is unnameable by construction and renders as an id — deliberately, since that is often exactly the actor being looked for. `listMembers` is promoted onto OrgMembersService for this: `resolveMembers` fails the whole call when any id is not a current member, which is the wrong shape for labelling historical records. Verified against the local stack: a refused key now reads "Grafana exporter". apps/api 215 files / 2623 tests, typecheck 40/40, lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… response time A failed `queue.send` fell back to writing the row straight to the warehouse, inline, before the response went out. That put a second network round trip on the response path at exactly the wrong moment: a Queues brown-out already costs the 2s send timeout, and the fallback then added a Tinybird write on top — every audited read, which is every dashboard telemetry query, turned slow while the platform was degraded. The queue already is the durability story: retries, then the DLQ. A send that cannot be made now logs and drops the entry rather than charging the caller for it. `writeDirect` stays for runtimes with no queue binding at all — local dev, crons, the consumer itself — none of which are serving a response. The test that pinned the old behaviour now pins the new one: a failed send writes nothing to the warehouse and still does not fail the caller. apps/api 215 files / 2623 tests, typecheck clean, lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The actor column named users but still showed them as strangers: a display name only when Clerk had first/last set, no face, and an opaque `user_…` otherwise. - `OrgMember` carries `imageUrl`, which the membership list already returns — no extra call — and the v2 entry gains `actor_avatar_url`, resolved when the log is read like the name is. - A member with no name set falls back to their email rather than an id. That is what the rest of the product shows and what an admin actually recognises; `user_3Bfcm…` is the last resort, not the second one. - The row renders a 16px avatar with initials behind it, for people only. Naming is now gated on the actor being a person, which fixes a real misattribution: every API-key and agent entry also carries a `userId` — whoever minted the credential — so the directory lookup was printing that person's name next to an "API key" badge on actions they did not take. Caught by looking at the page: a denial from a key read "API key · David Ambrus". Keys keep the name frozen at write time; older entries without one show their id, which is honest. apps/api 215 files / 2623 tests, domain 693, typecheck 40/40, lint clean. Web bundle 649.0 KB against the 650.0 KB budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The actor column read `[User] David Ambrus` on every human row — a badge whose only job was to say "this is a person", next to a face and a name that already said it. The avatar now replaces the badge for user actors; keys, agents and system entries keep theirs, because they have no face to identify them by. Two details the change turns up: - The avatar is Clerk's own `publicUserData.imageUrl`, which Clerk serves for every member — a real `img.clerk.com` URL rendering their initials when they uploaded no picture, so a row is never a blank circle. Verified in the page: the img loads (naturalWidth 128), it is not the local fallback. - A row we could not name falls back to the raw `user_…` id, and taking initials off that produced a confident "U". It shows "?" instead — an unknown member, not someone whose name begins with U. Web bundle 649.0 KB against the 650.0 KB budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A sweep of what this branch added, against what the repo already has.
**A third copy of a shared formatter.** `msToWarehouseDateTime64` in
apps/api hand-rolled `YYYY-MM-DD HH:mm:ss.SSS` with manual UTC padding —
byte-identical to `formatWarehouseDateTimeMs`, which query-engine has exported
all along for exactly this ("a minority of callers deliberately keep the
fractional part (DateTime64 columns…)"). Deleted, and the three call sites now
go through the shared one.
**No brand on the millisecond shape.** `WarehouseDateTime` is branded so it
"cannot be produced by string manipulation"; its `DateTime64(3)` sibling had
nothing, so a hand-built whole-second or ISO `T`/`Z` string type-checked into an
ingest row — the first collapses the sub-second ordering a DateTime64 sort key
exists to keep, the second is rejected by the Events API's JSONPath parser and
dropped by the warehouse rather than by anything in front of it. Added
`WarehouseDateTime64` + `warehouseDateTime64(epochMs)` beside them, mirroring
the existing pair.
Worth noting why the builder's `dateTime64` codec is not the answer here: its
encoder is `.slice(0, 19)`, which truncates the milliseconds. The audit query
sidesteps it already by binding `param.dateTimeString`, so the precision does
survive into the SQL — but nothing said so.
**`Effect.catchCause` in the queue consumer.** The one this branch had not yet
converted. v4's catchCause catches interruption too, so a batch interrupted by
a deploy would be counted as a failed attempt and pushed toward the DLQ for
something that never failed. Now `catch` + `catchDefect`, leaving the interrupt
to unwind — the batch stays unacked and the platform redelivers it. A defect
now retries as well, which is new behaviour and has a test that fails when the
`catchDefect` is removed.
**Duplicated request forensics.** `audit-denial` rebuilt the
cf-ray/cf-connecting-ip/cf-ipcountry block that `httpRequestForensics` already
provides three lines away.
Left alone deliberately: `try/finally` at the worker queue boundary (outside
Effect, and identical to its sibling consumers); `Date.parse` on `since`/`until`
in the v2 route, whose input is already constrained by the `Timestamp` schema so
it cannot be NaN.
apps/api 215 files / 2629 tests, query-engine 62, typecheck 40/40, lint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`buildDestinationChanges` was the last update handler still assembling its diff by hand — 30 lines of loop plus two lookup helpers, doing what `auditDiff` was built to do for every other resource. The bespoke part it needed and the shared helper lacked: a field the response does not echo but which is no secret — a Slack channel id, a Telegram chat id, the Hazel org handles. `writeOnly` withholds a credential's value as `<redacted>`; these are simply not knowable from the document, and read better as `<updated>`. So `auditDiff` grows an `opaque` bucket alongside `writeOnly`, and destinations become a declaration: three diffed fields, five credentials, eight handles. That kills a quiet hazard. The old loop derived audit field names by regex from the *internal* camelCase request (`channelId` -> `channel_id`), so a wire key rename would silently change what the log recorded, and the secret set was keyed on internal names too — a rename there would have silently un-redacted a credential. The spec is keyed on the wire names the payload actually carries. Equivalence checked rather than assumed: old and new agree on all nine representative payloads (every destination type, an unchanged field, a rename alongside a rotation, and a missing pre-update document), modulo field order, which moves from request-key order to a stable declaration order. The old code had no tests. This adds nine — three for `opaque`, six for the destination spec, including that a webhook URL is withheld rather than diffed. apps/api 215 files / 2637 tests, typecheck 40/40, lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The manifest is generated, so both sides' edits were regenerated from source rather than hand-merged: main's `deploymentMethod: "alter"` on `ai_trace_index_mv` and this branch's `audit_log` datasource. The ClickHouse schema, local DDL and insert mappings follow the same regeneration. The local structural schema is unchanged — a deployment method is not DDL — so v17 stands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ardown `Effect.catch` + `Effect.catchDefect` as a pair was mine, and it is not what this repo does. `Effect.catchCause` guarded by `Cause.hasInterruptsOnly` is — ten call sites already, in turn-runner, agent-pass, DigestService, AnomalyDetectionService, EscalationService and PlanetScaleService. One combinator instead of two, and the interrupt exemption is stated in the code rather than in a comment explaining why catchCause was avoided. Both sites this branch introduced now follow it, and both re-raise rather than recover: a request being torn down has no audit page left to label, and an interrupted queue batch must not spend a retry — unacked is enough, the platform redelivers. Causes are logged through `summarizeCause`, which the repo uses in seventy places, instead of dumping the cause object into the log annotation. New test: an interrupted batch retries nothing and exits as a failure. The suite is checked against a mutant — forcing the guard to `true` fails the failure, defect and DLQ cases and leaves only the interrupt case passing. apps/api 215 files / 2638 tests, typecheck 40/40, lint clean (the effect-lint `unnecessary-pipe-chain` rule caught a chained pipe here first). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ound it Every queue consumer and cron in worker.ts wrapped `runScheduledEffect` in a `try/finally` for one reason: to register the telemetry flush. Six copies of the same hand-rolled boundary around a helper that already owns the runtime, the scheduler drain, the dispose and the `waitUntil`. `runScheduledEffect` takes `onSettled` now and runs it after dispose, inside the `waitUntil` it already registered. That is also strictly safer than what the callers did: a `finally` that fires after the awaited promise settles was calling `ctx.waitUntil` late, sometimes after the handler had already rejected; chaining it into the existing registration cannot miss the window. And the flush still runs after dispose, where the last spans have been emitted. Converted all six — the two other queue consumers and the three crons alongside the audit one, since leaving worker.ts half-migrated would be worse than either end state. `onSettled` is typed `() => Promise<void>`, not `Promise<unknown>`: the repo's anti-slop lint rejects handing `unknown` back to a caller, and the SDK's flush is `Promise<void>` anyway. apps/api 215 files / 2638 tests, infra 57, typecheck 40/40, lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An org-wide, append-only audit log stored in ClickHouse: every allowed or denied action performed against Maple — from the dashboard, the public API, or MCP — and, for HIPAA audit controls, every read of telemetry and session replays, attributed to the user, API key, or agent that performed it.
How it works
Storage — the Tinybird datasource
audit_log(ClickHouse migration0025,requiredForIngest: false).ReplacingMergeTreekeyed on(OrgId, OccurredAt, Id)so queue redelivery collapses at merge time; monthly partitions; six-year TTL (HIPAA §164.316(b)(2)). Absent values are'', diffs and metadata are JSON documents, touched field names live inChangedFields Array(String).WarehouseQueryService.ingest, which is pinned to the managed pipeline and never routed to a BYO ClickHouse — the log is Maple's record, not the customer warehouse's.auditLogEntriesQuerybuilder with.route("ingest")(+INGEST_PINNED_TABLES), so a BYO-CH org still reads the managed table.audit_logis hidden fromrun_sql, the per-org raw-SQL JWT, anddescribe_warehouse_tables; the admin-gatedGET /v2/audit_logis the only way in.AUDIT_LOG_RETENTION_DAYSare removed (the PG migration was never applied to prd).Entries — actor snapshot at write time (
user/api_key/agent/system, credential refs, frozen label, on-behalf-of user),outcome+denial_reason, before/afterchangeswith queryable changed fields (secrets redacted), request forensics (request_id, origin IP + country,affected_user),occurred_at(producer) vsrecorded_at(consumer).Durability — the
audit-eventsCloudflare Queue is unchanged in shape: producers enqueue (2s send timeout, direct-write fallback), the consumer writes one ingest batch per org; write failures retry through the queue's policy and exhaust intoaudit-events-dlq.Recording surfaces
register_agent, membership changes from the Clerk webhook, and auth-layer denials.AuditedReadin the domain contracts —telemetry.readon the internal query-engine group, v2 traces / logs / metrics / error_issues and the v1 error-issue GETs;session_replay.readon every replay group. The auth layers record endpoint, method, path, status and a bounded body snapshot.mcp_tool.called, with the tool and its parameters.run_sql,inspect_chart_dataand the dashboard raw-SQL route astelemetry.sql_executed; a statement the safety pass refuses is recorded asdenied.Read side —
GET /v2/audit_log: cursor-paginated, filterable byactor_type,actor_id,affected_user,action,outcome,resource_type,resource_id,changed,request_id, and time window.audit_log:readscope,alog_public IDs, admin-only.UI — Settings → Audit Log: actor + outcome filter pills, denied badges with reason, change summaries with before→after tooltips, origin/request detail, load-more pagination.
Notes for deploy
tinybird:deploy) before the API ships; self-hosted ClickHouse gets the table from migration0025(clickhouse-cli apply). The local chDB schema moves to v15 with an additive migration edge.Testing
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.