diff --git a/adrs/005-authorization.md b/adrs/005-authorization.md index 33a4d6be..ebcc0229 100644 --- a/adrs/005-authorization.md +++ b/adrs/005-authorization.md @@ -7,6 +7,9 @@ **Implementation:** `src/authz.rs`, `src/backend_auth.rs`, `src/source_api/registry.rs`, `src/source_api/auth.rs`; `source.coop:src/lib/api/oidc.ts` **Implemented by:** #116 (registry + API resolution), #149 (product visibility model), #162 (authorize and enable writes), #170 (extract `decide_backend_auth` + CI ordering test), #183 (hermetic API stub, contract and failure-mode tests) · source.coop#283 (OIDC auth), source.coop#284 (require auth for restricted products) +> [!NOTE] +> **Amended by ADR-013 (revised 2026-09-25).** One route, `POST /api/v1/service-account-keys/exchanges`, is called before the proxy knows which account is calling, so the proxy authenticates it as itself: a proxy-signed assertion whose subject is the sentinel `urn:source:data-proxy`, accepted only on that route and resolving to no account anywhere else. Every other lookup stays on behalf of the caller, as below. + --- ## Context diff --git a/adrs/013-api-keys.md b/adrs/013-api-keys.md index 80b16ed1..65a479cd 100644 --- a/adrs/013-api-keys.md +++ b/adrs/013-api-keys.md @@ -1,15 +1,16 @@ # ADR-013: API Keys for Environments Without OIDC -**Status:** Proposed — not implemented -**Date:** 2026-04-01 -**RFC:** RFC-001 -**Depends on:** ADR-001, ADR-004, ADR-006, ADR-010 +**Status:** Proposed — not implemented (revised 2026-09-25) +**Date:** 2026-04-01 · revised 2026-09-25 +**RFC:** RFC-001 §12 +**Depends on:** ADR-001, ADR-004, ADR-007, ADR-014 +**Amends:** ADR-005 (proxy-to-API authentication), ADR-014 (the amendment of this ADR) > [!NOTE] > The `api-keys` endpoints that exist in `source.coop` today are the **legacy** admin-managed keys used by the pre-Workers proxy. They are unrelated to this design, and the current proxy has no code path that accepts them (ADR-001). This ADR proposes a replacement, not a formalisation of what is there. > [!NOTE] -> **Amended by ADR-014 (Service Accounts).** The `sub` of an API-key JWT is a **service account** id; a key belongs to one service account. There is no per-key Role binding in the first release — the account's memberships are the grant and the hardcoded Roles only subtract — so the dependency on ADR-010 is replaced by one on ADR-014. Expiry is optional and may be changed after issuance; several keys may be active at once. The examples below use ADR-010's `sc::` `RoleArn` grammar; clients send the AWS form ADR-014 describes, `arn:aws:iam:::role/`. +> **Revised 2026-09-25.** The original Decision — a long-lived JWT signed by the data proxy — was implemented (source-cooperative/data.source.coop#233, source-cooperative/source.coop#570) and withdrawn on review before release. The Decision below replaces it; the original design and why it was withdrawn are recorded under Context and Alternatives. ADR-014's amendment of this ADR (the key belongs to one service account; no per-key Role binding; editable expiry, several active keys) carries over. --- @@ -25,132 +26,65 @@ However, a significant class of users has neither: These users have Source Cooperative accounts but operate in compute environments that do not issue OIDC tokens and cannot perform interactive browser authentication at runtime. ADR-001 and ADR-004 both identify this gap as future work. ---- - -## Decision - -### API Keys as Long-Lived JWTs - -Source Cooperative issues API keys as long-lived JWTs signed by the data proxy's own signing key — the same key the proxy uses as an OIDC issuer for outbound storage authentication (ADR-006). The proxy already publishes its JWKS and `/.well-known/openid-configuration`; API key JWTs are verifiable against the same key material. - -An API key JWT contains: - -```json -{ - "iss": "https://data.source.coop", - "sub": "", - "jti": "", - "iat": 1711929600, - "exp": 1743465600, - "type": "api_key" -} -``` - -- `iss` is the proxy's own issuer URL, not `auth.source.coop` (which is Ory Network and outside Source Cooperative's control for token minting) -- `sub` identifies the Source Cooperative account that owns the key -- `jti` is a unique key identifier used for revocation checks -- `exp` is optional — keys without an expiry are valid until explicitly revoked -- `type` distinguishes API key JWTs from other tokens the proxy may issue (e.g. outbound federation tokens) - -### Key Lifecycle +### Why the first Decision was withdrawn -**Creation:** +The first version of this ADR chose to issue API keys as long-lived JWTs signed by the data proxy, on the grounds that such a key could be exchanged at `/.sts` "with no new endpoint and no new validation logic beyond the `jti` check". ADR-014 kept that shape and made the key's subject a service account. -Users create API keys via the Source Cooperative UI or CLI: +Implementing it (source-cooperative/data.source.coop#233, developmentseed/multistore#147, source-cooperative/source.coop#570) showed the premise does not hold: -``` -source keys create --label "ncar-cronjob" --role sc::my-org::role/publisher -``` +- **The lookup was never optional.** Revocation needs a per-key check at the Source API on every exchange, cached for 60 seconds. A lookup keyed by the secret itself returns the account, so the signature's only remaining job, naming the account before the lookup, is one the lookup can do. +- **The existing path could not be used.** A Cloudflare Worker cannot fetch its own JWKS (error 1042), so the proxy verified its own keys in process, ahead of the STS route, with a dedicated role and error mapping, plus a new `POST /.keys` for minting. That is the new endpoint and the new validation logic the JWT was meant to avoid. +- **Minting became a cycle.** source.coop wrote the record, obtained an Ory ID token, called the proxy, and the proxy called source.coop back to confirm the caller manages the account, then signed. `/.keys` would sign any `jti` for a manager, so a manager could re-derive a live key for an existing record with no audit; "shown once" was not a property. +- **Two sources of truth for expiry.** `exp` was baked into the token; the record's `expires_at` was editable. Extending or removing an expiry did not change what the proxy enforced. +- **A rotation cliff.** The proxy's signing key also signs outbound federation assertions (ADR-006) and its own calls to the API (ADR-005). The proxy verified keys against the current and one previous signer. A key with no expiry, which ADR-014 promises, stopped verifying on the second rotation, and any rotation forced by the other two uses invalidated every key at once. -The system: -1. Generates a unique `jti` -2. Stores key metadata in the policy store: `jti`, account ID, label, bound Role (optional), created-at, expires-at (nullable) -3. Mints and signs the JWT -4. Returns the raw JWT to the user — displayed once, never stored by the platform +A design comment on the epic (source-cooperative/source.coop#491, 2026-08-22) had already stated the principle: a long-lived credential "must not depend on a signature staying verifiable for years". That comment drew the two-hop conclusion revisited under Alternatives. -**Revocation:** - -Users revoke keys via the UI or CLI: - -``` -source keys revoke -``` - -Revocation marks the key's `jti` as revoked in the policy store. The revocation takes effect within the `jti` validation cache TTL (see below). - -**Management API:** +--- -``` -POST /api/accounts/{account_id}/keys -GET /api/accounts/{account_id}/keys -DELETE /api/accounts/{account_id}/keys/{key_id} -``` +## Decision -The `GET` endpoint returns key metadata (ID, label, created-at, expires-at, last-used-at) but never the JWT itself. Only account owners and org admins can manage keys. +### The key is an opaque secret -### STS Exchange +An API key is `sck_` followed by 32 random bytes in base64url, a fixed 47 characters matching `^sck_[A-Za-z0-9_-]{43}$`, with no checksum. source.coop generates it, stores `sha256(key)` on the key record, shows it once, and never stores or logs the key. Nothing signs it. There is no key material for API keys anywhere on the platform. The hash needs no salt or key-derivation function: its input is 256 random bits, and the lookup is a key get, so there is no comparison to time. -API key JWTs are exchanged at `/.sts` using the same flow as any other OIDC token (ADR-004) — `AssumeRoleWithWebIdentity` is an action parameter, not a path segment: +The record holds `key_hash` (partition key), `key_id` (random, public, for the UI and management actions), `account_id` (a service account, per ADR-014), `label`, `created_at`, `created_by`, `expires_at` (nullable, editable after issuance), `revoked_at` and `last_used_at`. A service account may hold several active keys; rotation is issue-new, deploy, revoke-old. A disabled service account is refused a key. -``` -Action=AssumeRoleWithWebIdentity -&WebIdentityToken= -&RoleArn=sc::my-org::role/publisher -&RoleSessionName=ncar-daily-sync -``` +### The proxy resolves it by asking source.coop -The STS exchange flow proceeds as defined in ADR-004 with one additional step: +`/.sts` accepts a key as `WebIdentityToken` in an `AssumeRoleWithWebIdentity` request, exactly as it accepts a JWT. The proxy: -1. Parse `RoleArn` → extract `account_id` and `role_name` -2. Load Role definition (cached) -3. Extract `iss` from JWT → matches `https://data.source.coop` -4. Verify JWT signature against the proxy's own JWKS -5. Verify `exp` (if present), `nbf`, `iat` -6. **Validate `jti` against the policy store** — confirm the key has not been revoked (cached, 30–60s TTL) -7. Evaluate claim constraints for the matched IdP binding -8. Validate `DurationSeconds` ≤ Role's `max_session_duration` -9. Generate credentials and return response +1. Accepts a key **only from the form body of a POST**. A key anywhere in the query string is refused before any lookup, with a message that says so, because the platform logs request URLs. +2. Trims surrounding whitespace, then checks the fixed format; a malformed value is refused locally. +3. Hashes the key and looks up its standing at `POST /api/v1/service-account-keys/exchanges` with `{"key_hash"}`, authenticated as the proxy itself (see Amendments). The answer, `{account_id, key_id, active}` with `active: false` for an unknown hash, is cached for 60 seconds. An API failure fails closed and caches nothing. +4. Refuses an inactive or unknown key with one client-visible outcome, `InvalidIdentityToken` "API key was not accepted (request id …)", the id in the message because SDKs surface only the message; the reason is in the log. +5. Otherwise mints session credentials for `account_id` through the same minting, sealing and response code as every other exchange, with the same duration floor and cap. The account segment of `RoleArn` is ignored, as it is for an Ory ID token; the key names the account. The role must be one the proxy serves. -Step 6 is the only addition to the existing STS flow. For non-API-key tokens (those without `"type": "api_key"`), this step is skipped. +source.coop answers `active` only when the key is not revoked, not expired, and its service account is not disabled, and records last use best-effort. -### Platform IdP Registration +Exchange attempts are rate-limited by client IP with the Workers rate-limiting binding, generously: a legitimate client exchanges about once a session, so a cluster behind one NAT stays far under the limit, while a flood of distinct junk keys, each of which costs the API one lookup, is what it bounds. -The proxy's own issuer is registered as a platform IdP: +### Clients -```json -{ - "id": "source-coop-api-key", - "issuer_url": "https://data.source.coop", - "display_name": "Source Cooperative API Key", - "well_known_claims": ["type"], - "audience_hint": "https://data.source.coop" -} -``` +A stock AWS SDK or CLI needs `AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE` pointing at a file containing the key, `AWS_ENDPOINT_URL_STS`, `AWS_ENDPOINT_URL_S3` and `AWS_REGION`. The SDK re-reads the file, exchanges, and refreshes on its own; nothing else runs on the machine. This is the same setup GitHub Actions uses with a real GitHub token. The Source CLI offers the same exchange with the key in the request body and can serve as an AWS `credential_process`, which is how tools that would otherwise send the STS request as a GET, such as GDAL, obtain credentials: the proxy refuses a key in a URL, and GDAL honours `credential_process` in the AWS config. -Roles that should be assumable via API key must include an identity constraint binding for this IdP: +### Revocation -```json -{ - "idp": "source-coop-api-key", - "claim_constraints": [ - {"claim": "type", "operator": "equals", "value": "api_key"} - ] -} -``` +Revoking a key, or its expiry passing, denies new exchanges within the 60-second standing cache. Session credentials already issued cannot be recalled; they live to their cap. **Disabling the service account** is the emergency stop for a leaked key: new exchanges are refused, and existing sessions degrade to anonymous as the proxy's per-request caches expire, writes within 60 seconds and reads of restricted products within five minutes; public reads are unaffected. Re-enabling makes any leaked keys live again. Rotating `SESSION_TOKEN_KEY` (ADR-001) invalidates every session on the platform and is not part of the key runbook. -This reuses the Role and identity constraint model from ADR-010 without modification — and therefore depends on it, since no such model exists today. Account owners explicitly opt in to API key access per Role: a Role without a `source-coop-api-key` binding cannot be assumed with an API key. +A public self-revoke route lets anyone holding a key revoke it (source-cooperative/source.coop#561): the same hash lookup, body-only, a uniform response, the same rate limit. -### Role Binding +--- -API keys can optionally be bound to a specific Role at creation time. A bound key can only be used to assume that Role. An unbound key can assume any Role the account owns that has a `source-coop-api-key` identity constraint. +## Amendments -Bound keys reduce blast radius: if leaked, the key can only access what that specific Role permits. For high-value automated workflows, bound keys are recommended. +### ADR-005 — Authorization delegated to the Source API -### Caching and Revocation Latency +ADR-005 rejected "a service-account identity for proxy-to-API calls" and authenticates every lookup as the caller. One route is the exception: `POST /api/v1/service-account-keys/exchanges` is called before the proxy knows which account is calling, so the proxy authenticates it **as itself**, with a proxy-signed assertion whose subject is the sentinel `urn:source:data-proxy`. That subject fails both account-id grammars, is accepted only on that route, and resolves to no account anywhere else. Every other lookup stays on behalf of the caller. -The `jti` validity check uses the same caching infrastructure as other policy store lookups (ADR-007): the Cloudflare Cache API, with a short TTL in line with the permission lookup. +### ADR-014 — Service accounts -This means revocation takes effect within roughly a minute. For the target use case (long-running cronjobs, batch pipelines), that latency is acceptable. If faster revocation is needed, rotating `SESSION_TOKEN_KEY` (ADR-001) invalidates all active STS sessions immediately — a more disruptive but available emergency response. +The first two bullets of ADR-014's amendment of this ADR are replaced: a key is an opaque secret whose record names the service account; revocation is per key, by record, not by `jti`. The remaining bullets, no per-key Role binding and editable expiry with several active keys, stand. --- @@ -158,30 +92,34 @@ This means revocation takes effect within roughly a minute. For the target use c **Benefits** -- Covers the authentication gap for environments without OIDC or browser access -- No new auth path at the proxy layer — API key JWTs flow through the existing `/.sts` exchange -- Reuses the proxy's existing OIDC issuer infrastructure (signing key, JWKS) from ADR-006 -- Reuses the Role and identity constraint model from ADR-010 -- Revocation is explicit and auditable via `jti` lookup -- Optional Role binding limits blast radius of leaked keys +- One source of truth. Existence, account, expiry, revocation and disablement are all the record's, read by one lookup the proxy already had to make. +- No signing key on the key path. The proxy's key stays reserved for outbound federation and its API calls; rotating it cannot affect an API key. Non-expiring keys need no retained key ring. +- Issuance is one DynamoDB write, with no cross-service call and no compensating delete. +- The proxy never holds a key at rest and forwards only its hash. Enumeration is infeasible at 256 bits. +- The stock-SDK experience the epic promises for GitHub Actions holds for every environment. +- The secret-scanning marker is a fixed-length, all-entropy pattern. **Costs / Risks** -- API key JWTs are bearer tokens — anyone with the raw JWT can use it. Users must treat them like passwords (store in environment variables or secret files, not in source control) -- The `jti` revocation check adds a policy store dependency to the STS exchange path for API key tokens. Cache misses add latency. -- Keys without expiry are valid indefinitely until revoked. If a user loses access to the management UI (e.g. leaves a university), orphaned keys persist unless an org admin revokes them. -- The proxy's signing key is now used for two purposes: outbound federation tokens (ADR-006) and API key JWTs. A signing key compromise affects both. Key rotation must account for both uses. +- `/.sts` gains a second verification branch ahead of the STS route. It reuses the STS crate's minting, sealing and response builders, but restates the duration floor and default in a few lines because the crate has no pre-resolved-subject entry point. +- A long-lived secret rides in `WebIdentityToken`, a parameter defined for signed assertions. It is accepted only in a POST body over TLS and hashed on arrival. Clients that send the STS request as a GET are refused and must go through the CLI. +- The exchanges route is the first Source API route that authenticates the proxy as itself. It is confined to that route and its sentinel subject. +- This is the first `/.sts` path where an unverified caller triggers a Source API call, so the rate limit ships with the branch, not after it. +- Revocation is bounded by the 60-second cache, and outstanding session credentials by their cap and by the per-request caches. Both are true of every credential the proxy issues. +- The public self-revoke route is a second surface that takes a raw key, under the same rules. --- ## Alternatives Considered -**Ory-issued long-lived tokens** — not feasible. `auth.source.coop` is Ory Network, which controls its own signing keys. Source Cooperative cannot mint arbitrary long-lived JWTs from Ory's issuer. +**Proxy-signed JWT keys (this ADR's first Decision)** — withdrawn for the reasons in Context: the lookup makes the signature redundant, the platform prevented reuse of the JWT path, and the shared signing key created an issuance cycle, a re-mint hole, an expiry conflict and a rotation cliff. + +**source.coop-signed JWT keys, verified at the proxy via a source.coop JWKS** — the trust direction is right (the control plane issues, the data plane verifies as it verifies GitHub), and it avoids the self-JWKS problem. It keeps the lookup, the `exp`-versus-record conflict and the obligation to retain every signing key forever, now on Vercel where secrets are baked at build. It buys nothing over an opaque key that the lookup does not already provide. -**OAuth2 client credentials grant** — considered. The client credentials grant authenticates an application, not a user — the resulting token's `sub` is the client ID, not a user identity. Mapping OAuth2 clients back to Source Cooperative accounts would require a bespoke service account system built on top of OAuth2. +**Two-hop: opaque key exchanged at source.coop for a short-lived token, then `/.sts`** — the most conventional shape (OAuth 2.0 client credentials; AWS IAM Roles Anywhere), and the conclusion of the 2026-08-22 design comment. A spike against the staging Ory project on 2026-09-25 (run output in source-cooperative/data.source.coop#234) showed Ory Network mints an ID token for a service-account subject through the headless flow source.coop already uses, so the proxy would need no change. Rejected for the first release because it charges its cost to the users this ADR exists for: every HPC node, instrument and VM would need the Source CLI as a credential helper or a timer rewriting a token file, where this design needs environment variables and a stock SDK. It also widens the revocation window from 60 seconds to the token lifetime and puts Ory's admin API on the machine path. It remains available as an addition, without changing the key format, should a use case need a JWT derived from a key. -**Ory personal access tokens** — investigated. Ory Network's PAT/API key concept (`ory_pat_`) is for project admin API access, not end-user authentication. User-scoped PATs are an [open feature request](https://github.com/ory/kratos/issues/1106) on Ory Kratos but not available. +**Long-lived credentials via the proxy's `get_credential` slot** — rejected in the same design comment: SigV4 is symmetric, so the proxy would need the secret in the clear. -**Opaque API keys with hash-based validation** — considered. The platform generates a random secret, stores a hash, and validates by re-hashing. This works but requires a dedicated validation endpoint or a new auth path at `/.sts`. The JWT approach avoids this by making API keys indistinguishable from other OIDC tokens at the STS layer — no new endpoint, no new validation logic beyond the `jti` check. +**Ory personal access tokens; OAuth 2.0 client credentials at Ory directly** — not available for end users, and a client is not an account (retained from the first version). -**Long-lived Ory refresh tokens** — considered as a near-term workaround. The user performs a one-time `source login` (device flow) and stores the refresh token. Cronjobs silently refresh access tokens. This works without new infrastructure but refresh tokens expire eventually, causing silent failures in unattended workflows. Suitable as an interim measure but not a durable solution for indefinitely recurring workloads. +**Long-lived Ory refresh tokens** — retained from the first version: a one-time `source login` whose refresh token cronjobs use silently. Refresh tokens expire eventually, causing silent failures in unattended workflows; an interim measure, not a durable one. diff --git a/adrs/014-service-accounts.md b/adrs/014-service-accounts.md index 830e33cf..39dcabe6 100644 --- a/adrs/014-service-accounts.md +++ b/adrs/014-service-accounts.md @@ -31,7 +31,7 @@ A service account's id is namespaced under its owner: `{owner_account_id}--{id}` ### How it authenticates: account trusts -An account says which subjects may act as it, the way an AWS role's trust policy does. `source.coop` holds an `account-trusts` table keyed by the account, with one row per issuer and exact subject the account trusts. A subject may be trusted by any number of accounts; nothing about a subject alone chooses an account. An individual's Ory identity is not a trust: it stays on the account row as `identity_id`, which the session, the email lookup and the proxy credentials already read. An API key's subject is the service account's own id (ADR-013), which the API resolves directly, so a key writes no trust either. The table holds what the platform cannot derive from an account: a service account's trust in whichever platform-IdP subjects (ADR-009) it integrates with — for GitHub Actions, one repository pinned to one ref or one environment, never organisation-wide. +An account says which subjects may act as it, the way an AWS role's trust policy does. `source.coop` holds an `account-trusts` table keyed by the account, with one row per issuer and exact subject the account trusts. A subject may be trusted by any number of accounts; nothing about a subject alone chooses an account. An individual's Ory identity is not a trust: it stays on the account row as `identity_id`, which the session, the email lookup and the proxy credentials already read. An API key names its service account on the key record (ADR-013), not through a trust, so a key writes no trust either. The table holds what the platform cannot derive from an account: a service account's trust in whichever platform-IdP subjects (ADR-009) it integrates with — for GitHub Actions, one repository pinned to one ref or one environment, never organisation-wide. **A trust is written when a manager adds it; nothing has to prove control of the subject first.** The workload names the account it wants when it exchanges its token — the account segment of `RoleArn`, `arn:aws:iam:::role/FullAccess` (or `ReadOnly`, or the `_default` alias). The partition stays `aws`: `aws-actions/configure-aws-credentials`, which is how a workflow is meant to obtain credentials, treats any other partition as a bare role name, and SDKs check only the value's length. The action validates the credentials it exports with `GetCallerIdentity`, which the proxy answers once developmentseed/multistore#126 lands — and the exchange succeeds only if that account trusts the token's issuer and subject. Trusting a subject one does not control gains nothing: its workflows never ask for the account. This is AWS's model, and the flow people already know from integrating GitHub Actions with AWS. @@ -58,8 +58,7 @@ The division of labour: **a Role answers "how narrow is this credential"; a serv ### ADR-013 — API keys -- The `sub` of an API-key JWT is a **service account** id, not an arbitrary account id. A key belongs to one service account; an individual or organisation does not hold keys directly. -- Enabling keys on a service account writes no binding: the key's subject is the account id itself, and the API resolves it as a service account by id after trying Ory. Whether an account has keys is the keys table's to say; revocation stays per key, by `jti`. +- A key belongs to one service account; an individual or organisation does not hold keys directly. Enabling keys writes no binding: the proxy resolves the key to its account by asking the API, and revocation is per key, by record. (ADR-013 as revised 2026-09-25: the key is an opaque secret, not a JWT, so it has no `sub` or `jti`.) - **No per-key Role binding in the first release.** The service account's memberships are the grant, and the hardcoded Roles only subtract, so any key may name either. The dependency on ADR-010 is dropped; ADR-013 depends on this ADR instead. - Expiry is optional and may be changed after issuance; a service account may hold several active keys, so rotation is overlap by construction. diff --git a/adrs/rfc-001.md b/adrs/rfc-001.md index d26fb4e0..c5323d1f 100644 --- a/adrs/rfc-001.md +++ b/adrs/rfc-001.md @@ -588,6 +588,7 @@ ADRs are grouped by status: those describing decisions already implemented and r | [ADR-011](011-role-ceiling-authorization.md) | Role-ceiling authorization | | [ADR-012](012-outbound-federation-coverage.md) | Outbound federation coverage — per-connection audience, GCP/Azure, stored credentials | | [ADR-013](013-api-keys.md) | API keys for environments without OIDC | +| [ADR-014](014-service-accounts.md) | Service accounts | The proposed set has a dependency order: ADR-009 unblocks ADR-010, which unblocks ADR-011. ADR-009 should not ship ahead of ADR-010 — see the warning in ADR-009.