Source Cooperative Data Proxy

A read-only data proxy for Source Cooperative, built as a Cloudflare Worker in Rust. It translates Source Cooperative URL paths into requests against cloud storage backends (S3, Azure Blob Storage, GCS) using the multistore S3 gateway.

The proxy supports GET, HEAD, and S3-compatible LIST operations with anonymous access, and resolves storage backends dynamically via the Source Cooperative API.

Getting Started

Prerequisites

• Rust with the wasm32-unknown-unknown target
• wrangler (Cloudflare Workers CLI)

rustup target add wasm32-unknown-unknown
cargo install worker-build@0.7.5
npm install -g wrangler@4

Run Locally

wrangler dev

The proxy will be available at http://localhost:8787.

Running Tests

Unit tests for routing and pagination run on native targets:

cargo test

Integration tests run against a live instance:

wrangler dev &
python -m pytest tests/test_integration.py

Example Requests

# Version info
curl http://localhost:8787/

# List products for an account
curl "http://localhost:8787/cholmes?list-type=2&delimiter=/"

# List files in a product
curl "http://localhost:8787/cholmes/admin-boundaries?list-type=2&prefix=&max-keys=10"

# Get object metadata
curl -I http://localhost:8787/cholmes/admin-boundaries/countries.parquet

# Download with range request
curl -r 0-1023 http://localhost:8787/cholmes/admin-boundaries/countries.parquet -o chunk.bin

Architecture

The proxy rewrites Source Cooperative URL paths (/{account}/{product}/{key}) into multistore's virtual bucket model, resolving storage backends dynamically via the Source Cooperative API.

Client Request: GET /{account}/{product}/{key}
  │
  ├─ [routing]   Parse path, rewrite to bucket={account}:{product}, key={key}
  ├─ [registry]  Resolve backend via Source API (product metadata + data connections)
  ├─ [cache]     Cache API responses (products: 5min, data connections: 30min, listings: 1min)
  ├─ [multistore ProxyGateway]  Generate presigned URL for the resolved storage backend
  └─ Stream response back to client

Modules

Module	Purpose
src/lib.rs	Fetch handler, OIDC discovery endpoints, request routing, CORS
src/routing.rs	Request classification and path rewriting
src/registry.rs	Source Cooperative API client and backend resolution
src/cache.rs	Cloudflare Cache API wrapper with per-datatype TTLs
src/pagination.rs	S3-compatible pagination for prefix listings
src/analytics.rs	Cloudflare Analytics Engine request logging
src/handlers.rs	Custom route handlers (index, account listing)

Supported Operations

Operation	Description
GET /	Version info
GET /{account}?list-type=2	List products for an account
GET /{account}/{product}?list-type=2	List objects in a product (S3-compatible, with pagination)
GET /{account}/{product}/{key}	Download an object (supports range requests)
HEAD /{account}/{product}/{key}	Get object metadata
OPTIONS *	CORS preflight
GET /.well-known/openid-configuration	OIDC discovery document
GET /.well-known/jwks.json	JSON Web Key Set for JWT verification

Write operations (PUT, POST, DELETE, PATCH) return 405 Method Not Allowed.

Configuration

Environment Variables

Set in wrangler.toml or via the Cloudflare dashboard:

Variable	Default	Description
SOURCE_API_URL	https://source.coop	Source Cooperative API base URL
LOG_LEVEL	WARN	Tracing level (TRACE, DEBUG, INFO, WARN, ERROR)
AUTH_ISSUER	https://auth.source.coop	OIDC issuer trusted for /.sts token exchange
AUTH_AUDIENCE	—	Comma-separated OAuth client ID(s) that /.sts subject tokens must be issued to (aud claim); a token is accepted if it matches any. Unset = /.sts token exchange is disabled (returns 501)
OIDC_PROVIDER_ISSUER	https://data.source.coop	Issuer URL for minted JWTs and OIDC discovery
OIDC_PROVIDER_KID	data-proxy-1	Key ID for the active signing key
OIDC_PROVIDER_KID_PREVIOUS	—	Key ID for the previous key (during rotation)

Bindings

Binding	Kind	Description
KEY_EXCHANGE_LIMIT	ratelimit	Per-client-IP limit on API-key exchanges at /.sts (ADR-013). Declared under [[ratelimits]] in every wrangler*.toml; a deployment without it logs an error and exchanges without a limit

API keys

A service account's API key (ADR-013) is an opaque sck_ secret that source.coop stores as a hash. It is presented at /.sts as WebIdentityToken, from a POST form body only — a key in the URL is refused, because the URL is logged. The proxy trims it and checks its shape and checksum (the last six characters are a CRC-32 of the thirty random ones before them, in base62), hashes it, and asks POST {SOURCE_API_URL}/api/v1/service-account-keys/exchanges for its standing as itself (subject urn:source:data-proxy), caching the answer for 60 seconds; then it mints credentials for the account the API names, exactly as it would for an ID token. A key that fails its shape or checksum was cut short or mistyped, and is refused as such without a lookup; every other refusal of the key reads API key was not accepted (request id …), and the reason is in the log under that id.

Roles

Every exchange at /.sts, of an ID token or an API key, names a Role in RoleArn, either bare or as the resource of an ARN of any partition and account (arn:aws:iam::000000000000:role/ReadOnly), since AWS SDKs insist on an ARN. The Roles are hardcoded (ADR-014):

Role	Credentials may
FullAccess	do everything the account's memberships allow
ReadOnly	do the same, except write
_default	do what FullAccess does; the name existing clients use

Any other name is refused with MalformedPolicyDocument, never mapped to a default. A Role only subtracts: its ceiling is sealed into the session token and checked locally before the account's own permissions are looked up (ADR-011), and a request it refuses gets the same AccessDenied as any other refusal.

Secrets

GitHub environment secrets are the source of truth. The deploy workflow (.github/workflows/deploy.yml) re-uploads them via wrangler secret bulk on every deploy, so a value set directly with wrangler secret put is silently overwritten by the next deploy. Set them per GitHub environment (preview, staging, production) with gh secret set <NAME> --env <environment> or in the repository settings UI.

Secret	Description
OIDC_PROVIDER_KEY	PEM-encoded PKCS#8 RSA private key for JWT signing
OIDC_PROVIDER_KEY_PREVIOUS	Previous RSA key (optional, during rotation)
SESSION_TOKEN_KEY	Base64-encoded 32-byte AES key sealing STS credentials
IP_HASH_SALT	Salt for hashing client IPs in analytics (optional; unset = unsalted, brute-forceable)

Authentication Priority

The proxy authenticates to the Source Cooperative API via minting short-lived JWT (60s TTL) signed with its RSA key and sending it as a Bearer token

OIDC Provider

When configured with an RSA key, the proxy acts as its own OpenID Connect identity provider. It serves standard discovery endpoints so that relying parties (such as the Source Cooperative API) can verify its JWTs.

Endpoints

• GET /.well-known/openid-configuration — discovery document containing issuer and JWKS URI
• GET /.well-known/jwks.json — public key(s) for JWT signature verification

These endpoints are only active when OIDC_PROVIDER_KEY is configured.

Setup

Generate an RSA key pair, store it as a GitHub environment secret, and deploy:

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out oidc-key.pem
gh secret set OIDC_PROVIDER_KEY --env production < oidc-key.pem

Verify the endpoints:

curl https://data.source.coop/.well-known/openid-configuration
curl https://data.source.coop/.well-known/jwks.json

Key Rotation

The proxy supports zero-downtime key rotation by serving both active and previous public keys in the JWKS endpoint. Only the active key signs new tokens.

Because CI re-uploads secrets from GitHub on every deploy, rotation happens through GitHub environment secrets and wrangler.toml, not wrangler secret put (which the next deploy would silently revert).

Rotation procedure (repeat per environment — preview, staging, production):

1. Generate a new RSA key with a new key ID:

   openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out new-key.pem

2. Move the current key to the previous slot and stage the new key:

   # Copy the CURRENT key PEM into the previous slot
   gh secret set OIDC_PROVIDER_KEY_PREVIOUS --env production < current-key.pem
   gh secret set OIDC_PROVIDER_KEY --env production < new-key.pem

   In wrangler.toml, set OIDC_PROVIDER_KID_PREVIOUS to the current OIDC_PROVIDER_KID value and set OIDC_PROVIDER_KID to the new key ID. Merge and let CI deploy.

3. The JWKS endpoint now serves both keys. Wait for relying parties to refresh their JWKS cache.

4. Remove the previous key:

   gh secret delete OIDC_PROVIDER_KEY_PREVIOUS --env production
   # The worker-side copy isn't removed by CI, so delete it once manually:
   wrangler secret delete OIDC_PROVIDER_KEY_PREVIOUS

   Remove OIDC_PROVIDER_KID_PREVIOUS from wrangler.toml, merge, and let CI deploy.

Rotating SESSION_TOKEN_KEY

SESSION_TOKEN_KEY seals the temporary credentials issued by /.sts. There is no dual-key window: rotating it immediately invalidates every outstanding sealed credential (e.g. the frontend's sc_proxy_creds cookies), and affected clients must redo the token exchange. To rotate:

openssl rand -base64 32 | gh secret set SESSION_TOKEN_KEY --env production

The next deploy picks it up.

Design Documents

See docs/plans/ for architecture and design documents.