From 48f5f79b2799f17e8116e44d1576f51d7bb589ea Mon Sep 17 00:00:00 2001 From: Winston Date: Fri, 4 Sep 2026 13:04:02 +1000 Subject: [PATCH 1/5] docs(saas-api): specify cache-key path encoding + path-encoding test vectors (LAB-2879) spec/saas-api.md documented /v1/cache/{key} and its /ttl and /lock sub-resources without saying how {key} is placed in the path. That silence cost three SDK tickets (cachekit-py#279 CWE-22; LAB-2877 ts; LAB-2878 rs). New normative section "Cache-Key Path Encoding": one percent-encoded segment with only RFC 3986 unreserved chars raw; the server decodes exactly once and validates the decoded key (no double-encoding, %2F never a boundary, health/ ttl/lock are route tokens); encoders may differ on the sub-delims !*'() because interop is defined on the decoded key, and every server-accepted key is byte-identical on the wire regardless. All-dot keys are stack-dependent and %2E is NOT a universal fix. RFC 3986 stacks (httpx 0.28.1) leave %2E%2E intact; WHATWG stacks (Node 25 URL/undici Request, rust-url 2.5.8 parser.rs:1319-1337) collapse %2e / %2e%2e / .%2e / %2e. in any case. On WHATWG stacks the client MUST reject "." / ".." before building the URL. Found by execution while writing the section. test-vectors/path-encoding.json: 12 key/encoded/decoded rows (canonical key, embedded ../, ?# injection, space, %, both all-dot keys flagged dot_segment, inert a:.. / ..a, ns:key, encoder-variance row with encoded_alternates). tools/path-encoding-verify.py (stdlib): pins encoded to the reference encoder, single-decode round-trip, rejects raw / ? # % and literal dot segments, cross-checks the WHATWG flag; mutation self-test runs first. Wired into verify.yml as an additional check. sdk-feature-matrix.md: Compliance Status row with actual state (py merged f000ba3, unreleased; rs/ts partial, in progress). README + CHANGELOG updated. --- .github/workflows/verify.yml | 3 + CHANGELOG.md | 40 +++++++++++ README.md | 4 +- sdk-feature-matrix.md | 3 +- spec/saas-api.md | 32 +++++++++ test-vectors/path-encoding.json | 86 +++++++++++++++++++++++ tools/path-encoding-verify.py | 116 ++++++++++++++++++++++++++++++++ 7 files changed, 281 insertions(+), 3 deletions(-) create mode 100644 test-vectors/path-encoding.json create mode 100644 tools/path-encoding-verify.py diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index a267e98..8138073 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -58,6 +58,9 @@ jobs: - name: File-backend format verify (stdlib only) run: python3 tools/file-backend-reference.py + - name: Cache-key path-encoding verify (stdlib only; mutation self-test first) + run: python3 tools/path-encoding-verify.py + - name: Python-frame JS cross-check (zero-dep independent reader, full round-trip) run: node tools/frame-crosscheck.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 7954064..397f3d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,46 @@ All notable changes to the CacheKit Protocol Specification. ## [Unreleased] +### SaaS API — cache-key path encoding specified (LAB-2879) + +- [`spec/saas-api.md`](spec/saas-api.md) gains a normative **Cache-Key Path + Encoding** section. Until now the spec documented `/v1/cache/{key}` and its + `/ttl` and `/lock` sub-resources without saying how `{key}` is placed in the + path — a silence that cost three SDK tickets (cachekit-py + [#279](https://github.com/cachekit-io/cachekit-py/pull/279) shipped the raw + key unquoted, CWE-22; cachekit-ts LAB-2877 and cachekit-rs LAB-2878 carry the + same latent all-dot gap). The section states: the key is ONE percent-encoded + path segment with only RFC 3986 unreserved characters raw; the server decodes + exactly once and validates the decoded key (no double-encoding, `%2F` is never + a segment boundary); encoders may differ on `! * ' ( )` because interop is + defined on the **decoded** key — and every server-accepted key is + byte-identical on the wire regardless. +- **All-dot keys (`.` / `..`) are stack-dependent, and `%2E` is not a universal + fix.** Dot-segment removal happens in the client's URL parser before the + request is sent, so the server cannot compensate. RFC 3986 §5.2.4 stacks + (httpx) remove only the literal `.`/`..`, so cachekit-py's `%2E` rewrite is + sound there. WHATWG URL Standard stacks (`fetch`/undici, browsers, Workers, + rust-url/`reqwest`) also treat `%2e`, `%2e%2e`, `.%2e`, `%2e.` (any case) as + dot segments — verified empirically (Node 25 `new URL()`/`Request`) and in + rust-url 2.5.8 `src/parser.rs:1319-1337`. No percent-encoding of an all-dot + key survives such a parser; on those stacks the client MUST reject the key + before building the URL. Found by execution while writing the section; the + filing ticket's premise that `%2E%2E` suffices everywhere was false. +- New [`test-vectors/path-encoding.json`](test-vectors/path-encoding.json): + `key → encoded → decoded` rows for a canonical 7-segment key, embedded `../` + traversals, `?#` injection, space, `%`, both all-dot keys (flagged + `dot_segment: true`), the inert `a:..` / `..a`, `ns:key`, and an + encoder-variance row carrying `encoded_alternates` for the + `encodeURIComponent` form. Verified in `verify.yml` by + [`tools/path-encoding-verify.py`](tools/path-encoding-verify.py) (stdlib; + pins `encoded` to the reference encoder, round-trips a single decode, rejects + raw `/ ? # %` and literal dot segments, and cross-checks the WHATWG + dot-segment flag). A mutation self-test runs first so the guard cannot + degrade to silently passing. +- [`sdk-feature-matrix.md`](sdk-feature-matrix.md) Compliance Status gains a + path-encoding row with actual state at merge time: Python merged (`f000ba3`, + unreleased — latest PyPI 0.17.1); Rust and TypeScript in progress. + ### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) - LZ4 compressed bytes are **not canonical** across conforming block encoders. diff --git a/README.md b/README.md index 58388f6..6aaff23 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ layer's own store/retrieve flows are specified in | [spec/cache-key-format.md](spec/cache-key-format.md) | Cache key generation algorithm — Blake2b-256, argument normalization, cross-SDK key strategy | | [spec/wire-format.md](spec/wire-format.md) | ByteStorage envelope — LZ4 block compression, xxHash3-64 integrity, decompression bomb protection | | [spec/encryption.md](spec/encryption.md) | AES-256-GCM encryption, HKDF-SHA256 key derivation, AAD v0x03, counter-based nonces, key rotation | -| [spec/saas-api.md](spec/saas-api.md) | REST API endpoints, binary wire protocol, error codes, metrics headers | +| [spec/saas-api.md](spec/saas-api.md) | REST API endpoints, cache-key path encoding, binary wire protocol, error codes, metrics headers | | [spec/interop-mode.md](spec/interop-mode.md) | Cross-SDK cache sharing — language-neutral key format, canonical argument normalization *(normative; shipped opt-in in all three SDKs — see the [feature matrix](sdk-feature-matrix.md#compliance-status) for per-SDK version floors)* | | [spec/interop-v2.md](spec/interop-v2.md) | Interop v2 compressed-values profile — opt-in LZ4-block + AES-256-GCM cross-SDK values *(DRAFT; no SDK implements it yet)* | | [spec/file-backend-format.md](spec/file-backend-format.md) | Shared local File backend filename, header, expiry, and fail-closed flag negotiation | @@ -125,7 +125,7 @@ An SDK is protocol-compliant when: 1. [Interop-mode](spec/interop-mode.md) key generation produces identical keys for identical inputs across all languages (auto-mode keys embed language-specific function identity and are not cross-SDK by design) 2. Interop-mode values encode and decode per the canonical vectors; where the SDK implements the ByteStorage envelope, it deserializes any spec-conformant envelope 3. Encrypted interop-mode payloads can be decrypted by any SDK with the same master key and tenant ID -4. SaaS API integration follows the documented endpoint contracts +4. SaaS API integration follows the documented endpoint contracts, including [cache-key path encoding](spec/saas-api.md#cache-key-path-encoding) (`test-vectors/path-encoding.json`) Test vectors are published in [`test-vectors/`](test-vectors/) as JSON files. diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index a1f64e9..681749f 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -285,6 +285,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | +| SaaS API — cache-key path encoding ([spec](spec/saas-api.md#cache-key-path-encoding)) | ✅ Conformant — merged `f000ba3` ([#279](https://github.com/cachekit-io/cachekit-py/pull/279)); in **no published release** as of 2026-09-04 (PyPI latest 0.17.1, `main` still `0.17.1`) — the first release after 0.17.1 carries it. The `%2E` all-dot rewrite is valid for its RFC 3986 stack (`httpx`) | ⚠️ Partial — percent-encodes one segment (`urlencoding::encode`, rules 1/3) but no all-dot guard (rule 2) at six call sites on `main` @ `42697bd`; LAB-2878 in progress. rust-url is WHATWG, so conformance means rejecting `.`/`..` client-side, not `%2E` | ⚠️ Partial — percent-encodes one segment (`encodeURIComponent`, rules 1/3) but no all-dot guard (rule 2) at five call sites on `main` @ `4261f16`; LAB-2877 in progress. undici/Workers `fetch` is WHATWG, so conformance means rejecting `.`/`..` client-side, not `%2E` | ❌ Not implemented | | Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | @@ -293,7 +294,7 @@ its spec: > > ¹⁵ Auto-mode **stored bytes** are SDK-internal and differ per SDK — see [wire-format.md → SDK Storage Containers](spec/wire-format.md#sdk-storage-containers-auto-mode). Python stores the ByteStorage envelope *inside* its CK v3 frame; `cachekit-rs` does not use the envelope for values at all (it uses `cachekit-core` only for encryption). Cross-SDK value compatibility is exclusively an [interop-mode](spec/interop-mode.md) property (protocol#11). > -> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), and — since LAB-423 — `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py)) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). +> ¹⁶ "Test vectors in CI" = vectors the SDK's own default CI executes. Beyond the SDKs, this repo's `verify.yml` CI-verifies `interop-mode.json`, `encryption.json`, `python-frame.json`, `file-backend.json` ([`tools/file-backend-reference.py`](tools/file-backend-reference.py)), `wire-format.json` ([`tools/wire-format-reference.py`](tools/wire-format-reference.py); since LAB-423), and `path-encoding.json` ([`tools/path-encoding-verify.py`](tools/path-encoding-verify.py); since LAB-2879) against reference implementations. `cache-keys.json` (regenerated by cachekit-py v0.12.0, byte-identical to the v0.5.0 originals) is vendored and CI-verified in cachekit-py since [cachekit-py#229](https://github.com/cachekit-io/cachekit-py/pull/229) (LAB-425). > > ¹⁷ Version cells are **floors** (`X+`), not snapshots — they stay true as new versions publish; check the registry for the current release. Python's floor is the first *installable* one: interop merged under the `v0.13.0` tag, but neither `0.12.0` nor `0.13.0` was ever published to PyPI, so `0.14.0` is the earliest PyPI release containing interop mode. Do not "correct" this to 0.13.0 from the cachekit-py changelog alone. diff --git a/spec/saas-api.md b/spec/saas-api.md index 386c39f..41bcc9d 100644 --- a/spec/saas-api.md +++ b/spec/saas-api.md @@ -17,6 +17,7 @@ - [Overview](#overview) - [Authentication](#authentication) - [Content Type](#content-type) +- [Cache-Key Path Encoding](#cache-key-path-encoding) - [Cache Endpoints](#cache-endpoints) - [Stale-While-Revalidate](#stale-while-revalidate) - [Lock Endpoints](#lock-endpoints) @@ -66,6 +67,37 @@ Content-Type: application/octet-stream --- +## Cache-Key Path Encoding + +Every endpoint below carries the cache key as a path segment — `/v1/cache/{key}`, `/v1/cache/{key}/ttl`, `/v1/cache/{key}/lock`. The key is caller-controlled (each SDK's `key=` escape hatch accepts an arbitrary string), so how it is placed in the path is a security boundary, not a formatting detail: an unencoded key can escape `/v1/cache/` and deliver the bearer token to a different route (CWE-22 — cachekit-py shipped exactly that until [cachekit-py#279](https://github.com/cachekit-io/cachekit-py/pull/279)). MUST, MUST NOT, SHOULD and MAY are used as in RFC 2119. + +### Encoding rules + +**1. One segment, percent-encoded.** `{key}` MUST be exactly one path segment. Clients MUST percent-encode the key's UTF-8 bytes (RFC 3986 §2.1) so that only unreserved characters — `ALPHA / DIGIT / "-" / "." / "_" / "~"` — appear raw. Every other byte MUST be sent as `%HH`: the delimiters `/ ? # %`, every reserved character (`:` in particular — a canonical key carries six), space (`%20`, never `+`), and every byte ≥ `0x80`. Hex digits SHOULD be uppercase (RFC 3986 §2.1); the server decodes either case. Reference encoders: Python `urllib.parse.quote(key, safe="")`, Rust `urlencoding::encode`, JavaScript `encodeURIComponent` (see rule 4 for its one permitted divergence). + +**2. All-dot keys are dot segments, and the server cannot help.** A key of exactly `.` or `..` survives rule 1 unchanged (`.` is unreserved), and a path segment that is `.` or `..` is removed by the client's own URL parser *before the request is sent*: `/v1/cache/..` becomes `/v1/`, `/v1/cache/../ttl` becomes `/v1/ttl` — a different route, still carrying `Authorization`, never reaching the key validator. Servers see only the collapsed path and MUST NOT be relied on to compensate. Clients MUST NOT emit a request whose key segment their URL layer treats as a dot segment. What that takes depends on the URL layer: + +- **RFC 3986 §5.2.4 `remove_dot_segments`** (e.g. `httpx`) removes only the literal `.` and `..`. On such a stack the client MAY percent-encode the dots — `%2E`, `%2E%2E` — which travel intact; the server decodes them once and applies its normal validation (`..` → `400`). +- **WHATWG URL Standard** (`fetch`/undici, browsers, the Workers runtime, rust-url and therefore `reqwest`) treats an ASCII-case-insensitive `%2e` as a single-dot segment and `%2e%2e`, `.%2e`, `%2e.` as double-dot segments (URL Standard §4.1; rust-url `src/parser.rs` path state). **No percent-encoding of an all-dot key survives a WHATWG parser.** On such a stack the client MUST reject a key of exactly `.` or `..` before building the URL and surface a client-side error. + +Only an *entirely*-dot segment is affected: `a:..`, `..a`, `x..y` are inert under both models and MUST be encoded per rule 1 with their dots left raw. Conformance tests MUST assert on the *parsed* request path (`httpx.Request.url.raw_path`, `new URL(u).pathname`, `Url::parse(u)?.path()`), not on the un-parsed template string — a template-string test passes while the traversal ships. + +> **Evidence (2026-09-04):** `httpx` 0.28.1 sends `/v1/cache/%2E%2E/ttl` unchanged; Node 25 `new URL()` and undici `Request` resolve the same string to `/v1/ttl`; rust-url 2.5.8 `src/parser.rs:1319-1337` matches `%2e%2e` / `.%2e` / `%2e.` / `%2e` in any case. The two models really do disagree, so a fix proven on one stack is not proof for the other. + +**3. The server decodes exactly once.** The server percent-decodes the key segment once (`decodeURIComponent`-equivalent; a malformed escape is `400`), then validates the *decoded* key: non-empty, within an implementation-defined maximum length (the deployed cap exceeds the 250-character SDK key limit in [cache-key-format.md](cache-key-format.md#key-length-limits)), drawn from `[A-Za-z0-9_.:-]`, free of the substring `..`, and — for `ns:` / `nsapi:` keys — carrying a well-formed namespace. Anything else is `400 Bad Request`. Consequences clients MUST honour: + +- Clients MUST NOT double-encode. A literal `%` in a key is sent as `%25` once; `%2525` decodes to `%25`, a different key. +- An encoded `%2F` never becomes a segment boundary: the router splits the path on raw `/` *before* decoding, so `a%2Fb` reaches the validator as `a/b` and is rejected by the charset rule. A conformant client can neither traverse nor store a key containing `/`. +- The literal segments `health`, `ttl` and `lock` are route tokens at this level (`/v1/cache/health`; a final `ttl` or `lock` segment selects the sub-resource). A key whose encoded form is exactly one of those words is therefore not addressable at `/v1/cache/{key}` — it is routed as the health check or as a sub-resource with an empty key. Canonical and interop keys always contain `:` and are unaffected. + +**4. Interop is defined on the decoded key.** Encoders MAY differ on the five sub-delims `! * ' ( )`: `encodeURIComponent` leaves them raw (they are legal `pchar` in a path segment and decode to themselves); `quote(safe="")` and `urlencoding::encode` emit `%21 %2A %27 %28 %29`. Both forms are conformant because the server-side key is identical after the single decode. Cross-SDK key equality is therefore a property of the **decoded** key, not of the wire bytes in general — but every key the server accepts is drawn from `[A-Za-z0-9_.:-]`, on which all three reference encoders agree (`:` → `%3A`, the rest raw). Every canonical auto-mode key and every [interop-mode](interop-mode.md) key is thus byte-identical on the wire across SDKs; the variance set only ever appears in keys the server rejects. + +### Test vectors + +[`test-vectors/path-encoding.json`](../test-vectors/path-encoding.json) pins `key → encoded → decoded` for the cases above: a canonical key, embedded `../` traversals, `?#` injection, space, `%`, both all-dot keys, the inert `a:..` / `..a`, and an encoder-variance row whose `encoded_alternates` carries the `encodeURIComponent` form. `encoded` is the reference form (`quote(safe="")`, all-dot result rewritten to `%2E`); rows flagged `dot_segment: true` are the two all-dot keys, and on a WHATWG stack the conformant outcome for those rows is rejection, not transmission. Verified in this repo's CI by [`tools/path-encoding-verify.py`](../tools/path-encoding-verify.py). + +--- + ## Cache Endpoints All cache endpoints are prefixed with `/v1/cache/`. diff --git a/test-vectors/path-encoding.json b/test-vectors/path-encoding.json new file mode 100644 index 0000000..410a5a3 --- /dev/null +++ b/test-vectors/path-encoding.json @@ -0,0 +1,86 @@ +{ + "version": "1.0.0", + "generator": "cachekit-py v0.17.1+ `CachekitIOBackend._encode_key` (commit f000ba3): `urllib.parse.quote(key, safe=\"\")`, then an all-dot result (`.` / `..`) has each dot rewritten to `%2E`", + "spec": "spec/saas-api.md § Cache-Key Path Encoding", + "ci_verification": "tools/path-encoding-verify.py (stdlib only; runs in this repo's verify.yml)", + "contract": "`encoded` is the single path segment a client sends in place of `{key}` in `/v1/cache/{key}`, `/v1/cache/{key}/ttl`, `/v1/cache/{key}/lock`. Only RFC 3986 unreserved characters (ALPHA DIGIT - . _ ~) appear raw; every other UTF-8 byte is `%HH` (uppercase). `decoded` is what the server sees after its single percent-decode and equals `key` in every row — interop is defined on the decoded key.", + "encoder_variance": "`encodeURIComponent` (cachekit-ts) leaves `! * ' ( )` raw where `quote(safe=\"\")` (cachekit-py) and `urlencoding::encode` (cachekit-rs) emit `%21 %2A %27 %28 %29`. Both forms are conformant: they decode to the same key. Rows carrying `encoded_alternates` list the other conformant wire form; assert `encoded in [encoded] + encoded_alternates`. Every key the server accepts (`[A-Za-z0-9_.:-]`) contains none of these characters, so accepted keys are byte-identical on the wire across all three encoders.", + "dot_segment_rows": "Rows with `dot_segment: true` are keys whose encoded form is a dot segment under the WHATWG URL Standard (`%2e`, `%2e%2e`, `.%2e`, `%2e.`, case-insensitive). `encoded` is the form an RFC 3986 §5.2.4 client (e.g. httpx) sends intact — it collapses only the literal `.` / `..`. A WHATWG client (fetch/undici, browsers, Cloudflare Workers, rust-url/reqwest) collapses `encoded` too, so on that stack the conformant outcome for these rows is a client-side rejection of `key` before the URL is built — there is no percent-encoding of an all-dot key that survives a WHATWG parser.", + "vectors": [ + { + "key": "ns:test:func:__main__.get_user:args:3870b2ea5735ae639ded9450ef117768db676f037bec636503796c5b81095153:1s", + "encoded": "ns%3Atest%3Afunc%3A__main__.get_user%3Aargs%3A3870b2ea5735ae639ded9450ef117768db676f037bec636503796c5b81095153%3A1s", + "decoded": "ns:test:func:__main__.get_user:args:3870b2ea5735ae639ded9450ef117768db676f037bec636503796c5b81095153:1s", + "note": "Canonical 7-segment auto-mode key (test-vectors/cache-keys.json `single_integer`). Only `:` is encoded; identical bytes from all three reference encoders." + }, + { + "key": "default:../../admin", + "encoded": "default%3A..%2F..%2Fadmin", + "decoded": "default:../../admin", + "note": "Embedded traversal: every `/` is `%2F`, so no `../` boundary exists for a URL parser to collapse. Server decodes once, then rejects (`/` outside charset; `..` substring)." + }, + { + "key": "x/../../health", + "encoded": "x%2F..%2F..%2Fhealth", + "decoded": "x/../../health", + "note": "Traversal aimed at the `/v1/cache/health` route token. Inert once `/` is `%2F`; an encoded `%2F` never becomes a segment boundary because the router splits on raw `/` before decoding." + }, + { + "key": "k?x=1#f", + "encoded": "k%3Fx%3D1%23f", + "decoded": "k?x=1#f", + "note": "Query/fragment injection: `?` and `#` MUST be encoded or the client's URL parser truncates the key and emits a query string." + }, + { + "key": "a b", + "encoded": "a%20b", + "decoded": "a b", + "note": "Space is `%20` in a path segment, never `+` (that is form-encoding, which the server does not decode)." + }, + { + "key": "100%", + "encoded": "100%25", + "decoded": "100%", + "note": "A literal `%` is encoded exactly once (`%25`); the server decodes once and sees `100%`. Clients MUST NOT double-encode (`%2525`)." + }, + { + "key": ".", + "encoded": "%2E", + "decoded": ".", + "dot_segment": true, + "note": "All-dot key. `quote`/`encodeURIComponent`/`urlencoding` all leave `.` raw (RFC 3986 unreserved), and a bare `.` segment is removed by every URL parser before the request is sent. RFC 3986 stacks (httpx) send `%2E` intact; WHATWG stacks collapse `%2E` as well and MUST reject the key client-side." + }, + { + "key": "..", + "encoded": "%2E%2E", + "decoded": "..", + "dot_segment": true, + "note": "All-dot key. Unencoded, `/v1/cache/..` collapses to `/v1/` and `/v1/cache/../ttl` to `/v1/ttl` — a different route, carrying the bearer token, never reaching the key validator. RFC 3986 stacks (httpx) send `%2E%2E` intact and the server rejects the decoded `..`; WHATWG stacks collapse `%2E%2E` as well and MUST reject the key client-side." + }, + { + "key": "a:..", + "encoded": "a%3A..", + "decoded": "a:..", + "note": "Trailing dots but NOT an all-dot segment: not a dot segment under either parsing model, so the dots stay raw. Server rejects the decoded `..` substring." + }, + { + "key": "..a", + "encoded": "..a", + "decoded": "..a", + "note": "Leading dots, not an all-dot segment: sent verbatim (all characters unreserved). Server rejects the decoded `..` substring." + }, + { + "key": "ns:key", + "encoded": "ns%3Akey", + "decoded": "ns:key", + "note": "Minimal namespaced key: `:` → `%3A`, decoded once server-side before the namespace-shape check." + }, + { + "key": "f(x)!*'", + "encoded": "f%28x%29%21%2A%27", + "decoded": "f(x)!*'", + "encoded_alternates": ["f(x)!*'"], + "note": "Encoder-variance row: `quote(safe=\"\")` and `urlencoding::encode` produce `encoded`; `encodeURIComponent` produces the alternate with `!*'()` raw. Both decode to `key`, both are conformant. The server rejects this key (charset), so the variance never reaches stored data." + } + ] +} diff --git a/tools/path-encoding-verify.py b/tools/path-encoding-verify.py new file mode 100644 index 0000000..aad25f5 --- /dev/null +++ b/tools/path-encoding-verify.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +"""Validate test-vectors/path-encoding.json with Python stdlib. + +spec/saas-api.md § Cache-Key Path Encoding. Every row's `encoded` must be the +reference form (`quote(key, safe="")`, all-dot result rewritten to `%2E`), decode +once back to `key`, carry no raw `/ ? # %`, and be a WHATWG dot segment only when +the row says so. A mutation self-test runs first so the guard cannot degrade to +silently reporting OK (same doctrine as tools/test_wire_format_reference.py). +""" + +from __future__ import annotations + +import copy +import json +import logging +from pathlib import Path +import re +import sys +from urllib.parse import quote, unquote + +ROOT = Path(__file__).resolve().parents[1] +VECTORS = ROOT / "test-vectors" / "path-encoding.json" + +# Reference form: RFC 3986 unreserved characters and uppercase %HH escapes only. +SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~-]|%[0-9A-F]{2})+$") +# Encoder-variance form: additionally the five sub-delims encodeURIComponent leaves raw +# (legal pchar in a path segment, decode to themselves — spec rule 4). +ALT_SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~!*'()-]|%[0-9A-F]{2})+$") +# WHATWG URL Standard § 4.1: single-/double-dot path segments, ASCII case-insensitive. +WHATWG_DOT_SEGMENTS = {".", "%2e", "..", ".%2e", "%2e.", "%2e%2e"} + + +def check(condition: bool, name: str, detail: str) -> None: + """Fail closed even under ``python -O`` (asserts would be stripped).""" + if not condition: + raise ValueError(f"{name}: {detail}") + + +def reference_encode(key: str) -> str: + """cachekit-py ``CachekitIOBackend._encode_key`` (f000ba3).""" + encoded = quote(key, safe="") + return encoded.replace(".", "%2E") if encoded in (".", "..") else encoded + + +def check_segment(name: str, segment: str, dot_segment: bool, key: str, pattern: re.Pattern[str] = SEGMENT) -> None: + check(pattern.fullmatch(segment) is not None, name, f"{segment!r} has a raw reserved character (or lowercase/incomplete %HH)") + check(segment not in (".", ".."), name, f"{segment!r} is a literal dot segment") + check((segment.lower() in WHATWG_DOT_SEGMENTS) == dot_segment, name, f"{segment!r} WHATWG dot-segment status does not match dot_segment={dot_segment}") + check(unquote(segment) == key, name, f"single decode of {segment!r} != key") + + +def verify(document: dict) -> int: + for vector in document["vectors"]: + key = vector["key"] + name = repr(key) + dot_segment = vector.get("dot_segment", False) + check(vector["decoded"] == key, name, "decoded != key (interop is defined on the decoded key)") + check(vector["encoded"] == reference_encode(key), name, f"encoded {vector['encoded']!r} != reference {reference_encode(key)!r}") + check_segment(name, vector["encoded"], dot_segment, key) + for alt in vector.get("encoded_alternates", []): + check(alt != vector["encoded"], name, "encoded_alternates repeats encoded") + check_segment(name, alt, dot_segment, key, ALT_SEGMENT) + return len(document["vectors"]) + + +def self_test(document: dict) -> None: + """Each poisoned copy must be rejected — otherwise the verify above is toothless.""" + def poisoned(mutate) -> dict: + doc = copy.deepcopy(document) + mutate(doc["vectors"]) + return doc + + def row(vectors: list, key: str) -> dict: + return next(v for v in vectors if v["key"] == key) + + mutations = { + "raw slash": lambda v: row(v, "x/../../health").__setitem__("encoded", "x/..%2F..%2Fhealth"), + "raw percent": lambda v: row(v, "100%").__setitem__("encoded", "100%"), + "double-encoded": lambda v: row(v, "100%").__setitem__("encoded", "100%2525"), + "literal dot segment": lambda v: row(v, "..").__setitem__("encoded", ".."), + "unflagged WHATWG dot segment": lambda v: row(v, "..").__delitem__("dot_segment"), + "flag on inert row": lambda v: row(v, "a:..").__setitem__("dot_segment", True), + "decoded drift": lambda v: row(v, "ns:key").__setitem__("decoded", "ns:kex"), + "lowercase hex": lambda v: row(v, "ns:key").__setitem__("encoded", "ns%3akey"), + "bad alternate": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), + } + for label, mutate in mutations.items(): + try: + verify(poisoned(mutate)) + except ValueError: + continue + raise ValueError(f"self-test: mutation {label!r} was not rejected") + + +def main() -> None: + try: + document = json.loads(VECTORS.read_text(encoding="utf-8")) + except OSError as exc: + sys.exit(f"cannot read {VECTORS}: {exc}") + except json.JSONDecodeError as exc: + sys.exit(f"invalid JSON in {VECTORS}: {exc}") + + try: + self_test(document) + count = verify(document) + except ValueError as exc: + sys.exit(f"invalid vector file: {exc}") + except (KeyError, TypeError, AttributeError, StopIteration) as exc: + sys.exit(f"invalid vector file: malformed structure ({exc!r})") + + logging.info("validated %d path-encoding vectors (self-test passed)", count) + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, format="%(message)s") + main() From d2b89876b8462f81f6a4390fabbc9105dd4c101e Mon Sep 17 00:00:00 2001 From: Winston Date: Fri, 4 Sep 2026 13:20:02 +1000 Subject: [PATCH 2/5] =?UTF-8?q?docs(saas-api):=20reserved=20segments=20mus?= =?UTF-8?q?t=20be=20rejected=20client-side=20=E2=80=94=20server=20WHATWG?= =?UTF-8?q?=20parse=20collapses=20%2E=20(LAB-2879=20panel=20round=201)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expert panel (bug-hunter + security, independently) probed api.cachekit.io: GET /v1/cache/%2E%2E/health returns the /v1/health response and /v1/cache/%2E%2E/ttl routes as /v1/ttl, while /v1/cache/a%3A..%2Fb/ttl reaches cache auth. The saas worker parses with WHATWG new URL(request.url) (index.ts:234), so the stack-dependent MAY-encode-to-%2E rule was false: no wire form of an all-dot key reaches the validator from any client. Reproduced before rewriting. Rule 2 is now uniform: clients MUST reject a key whose encoded form is exactly . .. health ttl lock (the last three are route tokens on the same level, per the security agent). Rule 1 scopes its MUST so the !*'() tolerance of rule 4 is not a contradiction. Rule 3 states the WHATWG parse precedes split and decode, and spells the ns:/nsapi: namespace shape. ns:key row note corrected (server rejects it). Evidence blockquote trimmed (no source line ranges). Vectors: 15 rows — 5 reject rows (encoded/decoded null) replace the dot_segment flag; envelope and row notes trimmed to row-specific facts. Verifier: one regex (alternates only), reserved-segment logic keyed on quote(key, safe=''), 7 mutations each tripping a distinct guard, poison built outside the try. CHANGELOG shortened and corrected (cachekit-py v0.18.0 tagged 2026-09-04, PyPI still 0.17.1). Matrix: Python downgraded to Partial with follow-up LAB-2880; call-site counts dropped. --- CHANGELOG.md | 62 +++++++++++++------------- sdk-feature-matrix.md | 2 +- spec/saas-api.md | 20 ++++----- test-vectors/path-encoding.json | 57 ++++++++++++++++-------- tools/path-encoding-verify.py | 77 +++++++++++++++------------------ 5 files changed, 112 insertions(+), 106 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 397f3d7..c55ce96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,42 +7,40 @@ All notable changes to the CacheKit Protocol Specification. ### SaaS API — cache-key path encoding specified (LAB-2879) - [`spec/saas-api.md`](spec/saas-api.md) gains a normative **Cache-Key Path - Encoding** section. Until now the spec documented `/v1/cache/{key}` and its - `/ttl` and `/lock` sub-resources without saying how `{key}` is placed in the - path — a silence that cost three SDK tickets (cachekit-py + Encoding** section. The spec documented `/v1/cache/{key}` and its `/ttl` and + `/lock` sub-resources without saying how `{key}` is placed in the path — a + silence that cost three SDK tickets (cachekit-py [#279](https://github.com/cachekit-io/cachekit-py/pull/279) shipped the raw key unquoted, CWE-22; cachekit-ts LAB-2877 and cachekit-rs LAB-2878 carry the - same latent all-dot gap). The section states: the key is ONE percent-encoded - path segment with only RFC 3986 unreserved characters raw; the server decodes - exactly once and validates the decoded key (no double-encoding, `%2F` is never - a segment boundary); encoders may differ on `! * ' ( )` because interop is - defined on the **decoded** key — and every server-accepted key is - byte-identical on the wire regardless. -- **All-dot keys (`.` / `..`) are stack-dependent, and `%2E` is not a universal - fix.** Dot-segment removal happens in the client's URL parser before the - request is sent, so the server cannot compensate. RFC 3986 §5.2.4 stacks - (httpx) remove only the literal `.`/`..`, so cachekit-py's `%2E` rewrite is - sound there. WHATWG URL Standard stacks (`fetch`/undici, browsers, Workers, - rust-url/`reqwest`) also treat `%2e`, `%2e%2e`, `.%2e`, `%2e.` (any case) as - dot segments — verified empirically (Node 25 `new URL()`/`Request`) and in - rust-url 2.5.8 `src/parser.rs:1319-1337`. No percent-encoding of an all-dot - key survives such a parser; on those stacks the client MUST reject the key - before building the URL. Found by execution while writing the section; the - filing ticket's premise that `%2E%2E` suffices everywhere was false. -- New [`test-vectors/path-encoding.json`](test-vectors/path-encoding.json): - `key → encoded → decoded` rows for a canonical 7-segment key, embedded `../` - traversals, `?#` injection, space, `%`, both all-dot keys (flagged - `dot_segment: true`), the inert `a:..` / `..a`, `ns:key`, and an - encoder-variance row carrying `encoded_alternates` for the - `encodeURIComponent` form. Verified in `verify.yml` by + same latent all-dot gap). Rules: the key is ONE percent-encoded segment with + only RFC 3986 unreserved characters raw (`! * ' ( )` tolerated); the server + URL-parses under the WHATWG URL Standard, splits on raw `/`, then decodes + exactly once and validates the decoded key; encoders may differ on the + sub-delims because interop is defined on the **decoded** key, and every + server-accepted key is byte-identical on the wire regardless. +- **Reserved segments must be rejected client-side; percent-encoding cannot + save an all-dot key.** The filing premise — encode `.`/`..` as `%2E`/`%2E%2E` + — is false: the server parses the request URL under the WHATWG URL Standard, + which treats `%2e`, `%2e%2e`, `.%2e`, `%2e.` (any case) as dot segments. + Verified live: `GET api.cachekit.io/v1/cache/%2E%2E/health` returns the + `/v1/health` response, `/v1/cache/%2E%2E/ttl` routes as `/v1/ttl`. httpx + 0.28.1 sends `%2E%2E` intact (RFC 3986 §5.2.4 removes only literal dots), so + cachekit-py's f000ba3 rewrite moves the collapse from client to server rather + than preventing it; Node 25 `new URL()` and rust-url 2.5.8 collapse it before + sending. Clients MUST reject a key whose encoded form is exactly `.`, `..`, + or one of the route tokens `health`, `ttl`, `lock`. Found by execution during + the expert-panel round on this change (bug-hunter and security agents + independently probed the live server). +- New [`test-vectors/path-encoding.json`](test-vectors/path-encoding.json) + (15 rows: 10 transmittable with `key → encoded → decoded`, 5 `reject: true` + reserved segments with no wire form, one `encoded_alternates` row for the + `encodeURIComponent` form), CI-verified by [`tools/path-encoding-verify.py`](tools/path-encoding-verify.py) (stdlib; - pins `encoded` to the reference encoder, round-trips a single decode, rejects - raw `/ ? # %` and literal dot segments, and cross-checks the WHATWG - dot-segment flag). A mutation self-test runs first so the guard cannot - degrade to silently passing. + 7-mutation self-test first, each mutation tripping a distinct guard). - [`sdk-feature-matrix.md`](sdk-feature-matrix.md) Compliance Status gains a - path-encoding row with actual state at merge time: Python merged (`f000ba3`, - unreleased — latest PyPI 0.17.1); Rust and TypeScript in progress. + path-encoding row with actual state: all three SDKs percent-encode one + segment but none yet rejects the reserved segments — Python's v0.18.0 `%2E` + rewrite is insufficient (follow-up filed); Rust and TypeScript in progress. ### Wire format — compressed-byte reproducibility scoped per-vector (LAB-1751) diff --git a/sdk-feature-matrix.md b/sdk-feature-matrix.md index 681749f..de8cd99 100644 --- a/sdk-feature-matrix.md +++ b/sdk-feature-matrix.md @@ -285,7 +285,7 @@ its spec: | Encryption (AES-256-GCM) | ✅ Compliant | ✅ Canonical (cachekit-core) | ✅ Compliant | ⚠️ Untested | | AAD v0x03 | ✅ Compliant (5 components — every auto serializer appends `original_type`; interop mode is the sole 4-component path) | ✅ Compliant (4 components) | ✅ Compliant (4 components) | ❌ Not implemented | | SaaS API | ✅ Compliant | ✅ Compliant (CachekitIO backend) | ✅ Compliant | ❌ Not implemented | -| SaaS API — cache-key path encoding ([spec](spec/saas-api.md#cache-key-path-encoding)) | ✅ Conformant — merged `f000ba3` ([#279](https://github.com/cachekit-io/cachekit-py/pull/279)); in **no published release** as of 2026-09-04 (PyPI latest 0.17.1, `main` still `0.17.1`) — the first release after 0.17.1 carries it. The `%2E` all-dot rewrite is valid for its RFC 3986 stack (`httpx`) | ⚠️ Partial — percent-encodes one segment (`urlencoding::encode`, rules 1/3) but no all-dot guard (rule 2) at six call sites on `main` @ `42697bd`; LAB-2878 in progress. rust-url is WHATWG, so conformance means rejecting `.`/`..` client-side, not `%2E` | ⚠️ Partial — percent-encodes one segment (`encodeURIComponent`, rules 1/3) but no all-dot guard (rule 2) at five call sites on `main` @ `4261f16`; LAB-2877 in progress. undici/Workers `fetch` is WHATWG, so conformance means rejecting `.`/`..` client-side, not `%2E` | ❌ Not implemented | +| SaaS API — cache-key path encoding ([spec](spec/saas-api.md#cache-key-path-encoding)) | ⚠️ Partial — percent-encodes one segment (`quote(safe="")`, rules 1/3/4) since `f000ba3` ([#279](https://github.com/cachekit-io/cachekit-py/pull/279), v0.18.0). Its `%2E` all-dot rewrite is collapsed by the server's WHATWG parse, so `.`/`..` still route-escape (rule 2); reserved-segment rejection is LAB-2880 | ⚠️ Partial — percent-encodes one segment (`urlencoding::encode`, rules 1/3/4) on `main` @ `42697bd`; no reserved-segment rejection (rule 2); LAB-2878 in progress | ⚠️ Partial — percent-encodes one segment (`encodeURIComponent`, rules 1/3/4) on `main` @ `4261f16`; no reserved-segment rejection (rule 2); LAB-2877 in progress | ❌ Not implemented | | Test vectors in CI¹⁶ | ✅ interop/v1 (full set, incl. AAD + encryption through the real stack) | ✅ interop/v1 (full set) since [#33](https://github.com/cachekit-io/cachekit-rs/pull/33) | ✅ interop/v1 (full set, incl. its key vectors) + inline Python-generated AAD-construction and encryption (decrypt-Python-ciphertext) vectors | ⚠️ Pending | | Interop mode ([spec](spec/interop-mode.md), opt-in) | ✅ Released — PyPI 0.14.0+¹⁷ ([#220](https://github.com/cachekit-io/cachekit-py/pull/220)) | ✅ Released — crates.io 0.4.0+ ([#33](https://github.com/cachekit-io/cachekit-rs/pull/33)) | ✅ Released — npm 0.1.3+ ([#71](https://github.com/cachekit-io/cachekit-ts/pull/71)) | ❌ Not implemented | diff --git a/spec/saas-api.md b/spec/saas-api.md index 41bcc9d..5bf6a8b 100644 --- a/spec/saas-api.md +++ b/spec/saas-api.md @@ -73,28 +73,24 @@ Every endpoint below carries the cache key as a path segment — `/v1/cache/{key ### Encoding rules -**1. One segment, percent-encoded.** `{key}` MUST be exactly one path segment. Clients MUST percent-encode the key's UTF-8 bytes (RFC 3986 §2.1) so that only unreserved characters — `ALPHA / DIGIT / "-" / "." / "_" / "~"` — appear raw. Every other byte MUST be sent as `%HH`: the delimiters `/ ? # %`, every reserved character (`:` in particular — a canonical key carries six), space (`%20`, never `+`), and every byte ≥ `0x80`. Hex digits SHOULD be uppercase (RFC 3986 §2.1); the server decodes either case. Reference encoders: Python `urllib.parse.quote(key, safe="")`, Rust `urlencoding::encode`, JavaScript `encodeURIComponent` (see rule 4 for its one permitted divergence). +**1. One segment, percent-encoded.** `{key}` MUST be exactly one path segment. Clients MUST percent-encode the key's UTF-8 bytes (RFC 3986 §2.1) so that only unreserved characters — `ALPHA / DIGIT / "-" / "." / "_" / "~"` — appear raw. Every other byte MUST be sent as `%HH` — the delimiters `/ ? # %`, `:` (a canonical key carries six), space (`%20`, never `+`), every byte ≥ `0x80` — with one tolerance: the sub-delims `! * ' ( )` MAY be left raw (rule 4). Hex digits SHOULD be uppercase (RFC 3986 §2.1); the server decodes either case. Reference encoders: Python `urllib.parse.quote(key, safe="")`, Rust `urlencoding::encode`, JavaScript `encodeURIComponent`. -**2. All-dot keys are dot segments, and the server cannot help.** A key of exactly `.` or `..` survives rule 1 unchanged (`.` is unreserved), and a path segment that is `.` or `..` is removed by the client's own URL parser *before the request is sent*: `/v1/cache/..` becomes `/v1/`, `/v1/cache/../ttl` becomes `/v1/ttl` — a different route, still carrying `Authorization`, never reaching the key validator. Servers see only the collapsed path and MUST NOT be relied on to compensate. Clients MUST NOT emit a request whose key segment their URL layer treats as a dot segment. What that takes depends on the URL layer: +**2. Reserved segments MUST be rejected client-side.** A key of exactly `.` or `..` survives rule 1 unchanged (`.` is unreserved) and is a *dot segment*: URL parsers remove it before routing — `/v1/cache/..` becomes `/v1/`, `/v1/cache/../ttl` becomes `/v1/ttl` — so the request lands on a different route, still carrying `Authorization`, and never reaches the key validator. Percent-encoding the dots does not help. The server parses the request URL under the WHATWG URL Standard, which treats an ASCII-case-insensitive `%2e` as a single-dot segment and `%2e%2e`, `.%2e`, `%2e.` as double-dot segments (URL Standard §4.1), so `%2E%2E` is collapsed *server-side* even when the client's own parser (RFC 3986 §5.2.4, e.g. `httpx`) sent it intact; WHATWG clients (`fetch`/undici, browsers, the Workers runtime, rust-url and therefore `reqwest`) collapse it before sending. **No wire form of an all-dot key reaches the validator from any client.** The literal segments `health`, `ttl` and `lock` are route tokens at this level — `/v1/cache/health` is the health endpoint, and a final `ttl` or `lock` segment selects the sub-resource — so a key encoding to one of those words is routed elsewhere or read as an empty key. -- **RFC 3986 §5.2.4 `remove_dot_segments`** (e.g. `httpx`) removes only the literal `.` and `..`. On such a stack the client MAY percent-encode the dots — `%2E`, `%2E%2E` — which travel intact; the server decodes them once and applies its normal validation (`..` → `400`). -- **WHATWG URL Standard** (`fetch`/undici, browsers, the Workers runtime, rust-url and therefore `reqwest`) treats an ASCII-case-insensitive `%2e` as a single-dot segment and `%2e%2e`, `.%2e`, `%2e.` as double-dot segments (URL Standard §4.1; rust-url `src/parser.rs` path state). **No percent-encoding of an all-dot key survives a WHATWG parser.** On such a stack the client MUST reject a key of exactly `.` or `..` before building the URL and surface a client-side error. +Therefore clients MUST reject a key whose encoded form is exactly `.`, `..`, `health`, `ttl` or `lock` before building the URL, surfacing a client-side error; servers MUST NOT be relied on to compensate. Only an *entirely*-dot segment is a dot segment: `a:..`, `..a`, `x..y` are inert and MUST be sent per rule 1 with their dots raw. Canonical and interop keys always contain `:` and never meet this rule. Conformance tests MUST assert on the *parsed* request path (`httpx.Request.url.raw_path`, `new URL(u).pathname`, `Url::parse(u)?.path()`), not on the un-parsed template string — a template-string test passes while the traversal ships. -Only an *entirely*-dot segment is affected: `a:..`, `..a`, `x..y` are inert under both models and MUST be encoded per rule 1 with their dots left raw. Conformance tests MUST assert on the *parsed* request path (`httpx.Request.url.raw_path`, `new URL(u).pathname`, `Url::parse(u)?.path()`), not on the un-parsed template string — a template-string test passes while the traversal ships. +> **Evidence (2026-09-04):** against `api.cachekit.io`, `GET /v1/cache/%2E%2E/health` returns the `/v1/health` response and `/v1/cache/%2E%2E/ttl` is routed as `/v1/ttl`, while `/v1/cache/a%3A..%2Fb/ttl` reaches the cache route. `httpx` 0.28.1 sends `%2E%2E` unchanged; Node 25 `new URL()` and rust-url 2.5.8 collapse it client-side. A fix proven on one parser is not proof for the other, and cachekit-py's `%2E` rewrite ([cachekit-py#279](https://github.com/cachekit-io/cachekit-py/pull/279), v0.18.0) moves the collapse from client to server rather than preventing it (LAB-2880). -> **Evidence (2026-09-04):** `httpx` 0.28.1 sends `/v1/cache/%2E%2E/ttl` unchanged; Node 25 `new URL()` and undici `Request` resolve the same string to `/v1/ttl`; rust-url 2.5.8 `src/parser.rs:1319-1337` matches `%2e%2e` / `.%2e` / `%2e.` / `%2e` in any case. The two models really do disagree, so a fix proven on one stack is not proof for the other. - -**3. The server decodes exactly once.** The server percent-decodes the key segment once (`decodeURIComponent`-equivalent; a malformed escape is `400`), then validates the *decoded* key: non-empty, within an implementation-defined maximum length (the deployed cap exceeds the 250-character SDK key limit in [cache-key-format.md](cache-key-format.md#key-length-limits)), drawn from `[A-Za-z0-9_.:-]`, free of the substring `..`, and — for `ns:` / `nsapi:` keys — carrying a well-formed namespace. Anything else is `400 Bad Request`. Consequences clients MUST honour: +**3. The server decodes exactly once.** After the WHATWG parse of rule 2, the server splits the path on raw `/`, then percent-decodes the key segment once (`decodeURIComponent`-equivalent; a malformed escape is `400`) and validates the *decoded* key: non-empty, within an implementation-defined maximum length (the deployed cap exceeds the 250-character SDK key limit in [cache-key-format.md](cache-key-format.md#key-length-limits)), drawn from `[A-Za-z0-9_.:-]`, free of the substring `..`, and — for `ns:` / `nsapi:` keys — of the shape `{prefix}:{namespace}:{rest}` with a non-empty namespace drawn from `[A-Za-z0-9_-]`. Anything else is `400 Bad Request`. Consequences clients MUST honour: - Clients MUST NOT double-encode. A literal `%` in a key is sent as `%25` once; `%2525` decodes to `%25`, a different key. -- An encoded `%2F` never becomes a segment boundary: the router splits the path on raw `/` *before* decoding, so `a%2Fb` reaches the validator as `a/b` and is rejected by the charset rule. A conformant client can neither traverse nor store a key containing `/`. -- The literal segments `health`, `ttl` and `lock` are route tokens at this level (`/v1/cache/health`; a final `ttl` or `lock` segment selects the sub-resource). A key whose encoded form is exactly one of those words is therefore not addressable at `/v1/cache/{key}` — it is routed as the health check or as a sub-resource with an empty key. Canonical and interop keys always contain `:` and are unaffected. +- An encoded `%2F` never becomes a segment boundary: the split on raw `/` happens *before* decoding, so `a%2Fb` reaches the validator as `a/b` and is rejected by the charset rule. A conformant client can neither traverse nor store a key containing `/`. -**4. Interop is defined on the decoded key.** Encoders MAY differ on the five sub-delims `! * ' ( )`: `encodeURIComponent` leaves them raw (they are legal `pchar` in a path segment and decode to themselves); `quote(safe="")` and `urlencoding::encode` emit `%21 %2A %27 %28 %29`. Both forms are conformant because the server-side key is identical after the single decode. Cross-SDK key equality is therefore a property of the **decoded** key, not of the wire bytes in general — but every key the server accepts is drawn from `[A-Za-z0-9_.:-]`, on which all three reference encoders agree (`:` → `%3A`, the rest raw). Every canonical auto-mode key and every [interop-mode](interop-mode.md) key is thus byte-identical on the wire across SDKs; the variance set only ever appears in keys the server rejects. +**4. Interop is defined on the decoded key.** `encodeURIComponent` leaves the sub-delims `! * ' ( )` raw (legal `pchar` in a path segment; they decode to themselves); `quote(safe="")` and `urlencoding::encode` emit `%21 %2A %27 %28 %29`. Both forms are conformant because the server-side key is identical after the single decode. Cross-SDK key equality is therefore a property of the **decoded** key, not of the wire bytes in general — but every key the server accepts is drawn from `[A-Za-z0-9_.:-]`, on which all three reference encoders agree (`:` → `%3A`, the rest raw). Every canonical auto-mode key and every [interop-mode](interop-mode.md) key is thus byte-identical on the wire across SDKs; the variance set only ever appears in keys the server rejects. ### Test vectors -[`test-vectors/path-encoding.json`](../test-vectors/path-encoding.json) pins `key → encoded → decoded` for the cases above: a canonical key, embedded `../` traversals, `?#` injection, space, `%`, both all-dot keys, the inert `a:..` / `..a`, and an encoder-variance row whose `encoded_alternates` carries the `encodeURIComponent` form. `encoded` is the reference form (`quote(safe="")`, all-dot result rewritten to `%2E`); rows flagged `dot_segment: true` are the two all-dot keys, and on a WHATWG stack the conformant outcome for those rows is rejection, not transmission. Verified in this repo's CI by [`tools/path-encoding-verify.py`](../tools/path-encoding-verify.py). +[`test-vectors/path-encoding.json`](../test-vectors/path-encoding.json) pins `key → encoded → decoded` in the reference form (`quote(safe="")`). Rows with `reject: true` are the reserved segments of rule 2 and carry no wire form; `encoded_alternates` lists the `encodeURIComponent` form where it differs (rule 4). Verified in this repo's CI by [`tools/path-encoding-verify.py`](../tools/path-encoding-verify.py). --- diff --git a/test-vectors/path-encoding.json b/test-vectors/path-encoding.json index 410a5a3..9231f75 100644 --- a/test-vectors/path-encoding.json +++ b/test-vectors/path-encoding.json @@ -1,11 +1,9 @@ { "version": "1.0.0", - "generator": "cachekit-py v0.17.1+ `CachekitIOBackend._encode_key` (commit f000ba3): `urllib.parse.quote(key, safe=\"\")`, then an all-dot result (`.` / `..`) has each dot rewritten to `%2E`", + "generator": "urllib.parse.quote(key, safe=\"\") — cachekit-py v0.18.0 CachekitIOBackend._encode_key (f000ba3) minus its %2E all-dot rewrite, which the server's WHATWG URL parse collapses; all-dot keys are reject rows", "spec": "spec/saas-api.md § Cache-Key Path Encoding", - "ci_verification": "tools/path-encoding-verify.py (stdlib only; runs in this repo's verify.yml)", - "contract": "`encoded` is the single path segment a client sends in place of `{key}` in `/v1/cache/{key}`, `/v1/cache/{key}/ttl`, `/v1/cache/{key}/lock`. Only RFC 3986 unreserved characters (ALPHA DIGIT - . _ ~) appear raw; every other UTF-8 byte is `%HH` (uppercase). `decoded` is what the server sees after its single percent-decode and equals `key` in every row — interop is defined on the decoded key.", - "encoder_variance": "`encodeURIComponent` (cachekit-ts) leaves `! * ' ( )` raw where `quote(safe=\"\")` (cachekit-py) and `urlencoding::encode` (cachekit-rs) emit `%21 %2A %27 %28 %29`. Both forms are conformant: they decode to the same key. Rows carrying `encoded_alternates` list the other conformant wire form; assert `encoded in [encoded] + encoded_alternates`. Every key the server accepts (`[A-Za-z0-9_.:-]`) contains none of these characters, so accepted keys are byte-identical on the wire across all three encoders.", - "dot_segment_rows": "Rows with `dot_segment: true` are keys whose encoded form is a dot segment under the WHATWG URL Standard (`%2e`, `%2e%2e`, `.%2e`, `%2e.`, case-insensitive). `encoded` is the form an RFC 3986 §5.2.4 client (e.g. httpx) sends intact — it collapses only the literal `.` / `..`. A WHATWG client (fetch/undici, browsers, Cloudflare Workers, rust-url/reqwest) collapses `encoded` too, so on that stack the conformant outcome for these rows is a client-side rejection of `key` before the URL is built — there is no percent-encoding of an all-dot key that survives a WHATWG parser.", + "ci_verification": "tools/path-encoding-verify.py (stdlib only; runs in this repo's verify.yml; mutation self-test first)", + "contract": "`encoded` is the single `{key}` path segment in the reference form (spec rule 1); `decoded` is the key the server sees after its single percent-decode and equals `key` in every transmittable row (spec rules 3-4). Rows with `reject: true` are the reserved segments of spec rule 2 (`.`, `..`, `health`, `ttl`, `lock`): a conformant client raises before building the URL, so `encoded` and `decoded` are null. `encoded_alternates` lists the other conformant wire form where `encodeURIComponent` differs (`! * ' ( )` raw); assert `encoded in [encoded] + encoded_alternates`.", "vectors": [ { "key": "ns:test:func:__main__.get_user:args:3870b2ea5735ae639ded9450ef117768db676f037bec636503796c5b81095153:1s", @@ -23,39 +21,60 @@ "key": "x/../../health", "encoded": "x%2F..%2F..%2Fhealth", "decoded": "x/../../health", - "note": "Traversal aimed at the `/v1/cache/health` route token. Inert once `/` is `%2F`; an encoded `%2F` never becomes a segment boundary because the router splits on raw `/` before decoding." + "note": "Traversal aimed at the `/v1/cache/health` route token; inert once `/` is `%2F`. Server rejects the decoded key (charset)." }, { "key": "k?x=1#f", "encoded": "k%3Fx%3D1%23f", "decoded": "k?x=1#f", - "note": "Query/fragment injection: `?` and `#` MUST be encoded or the client's URL parser truncates the key and emits a query string." + "note": "Query/fragment injection: `?` and `#` MUST be encoded or the client's URL parser truncates the key and emits a query string. Server rejects the decoded key (charset)." }, { "key": "a b", "encoded": "a%20b", "decoded": "a b", - "note": "Space is `%20` in a path segment, never `+` (that is form-encoding, which the server does not decode)." + "note": "Space is `%20` in a path segment, never `+` (form-encoding, which the server does not decode). Server rejects the decoded key (charset)." }, { "key": "100%", "encoded": "100%25", "decoded": "100%", - "note": "A literal `%` is encoded exactly once (`%25`); the server decodes once and sees `100%`. Clients MUST NOT double-encode (`%2525`)." + "note": "A literal `%` is encoded exactly once; the server decodes once and sees `100%`, then rejects it (charset). `%2525` would decode to `100%25`, a different key." }, { "key": ".", - "encoded": "%2E", - "decoded": ".", - "dot_segment": true, - "note": "All-dot key. `quote`/`encodeURIComponent`/`urlencoding` all leave `.` raw (RFC 3986 unreserved), and a bare `.` segment is removed by every URL parser before the request is sent. RFC 3986 stacks (httpx) send `%2E` intact; WHATWG stacks collapse `%2E` as well and MUST reject the key client-side." + "encoded": null, + "decoded": null, + "reject": true, + "note": "All-dot key: the plain encoding `.` is a dot segment, and `%2E` is collapsed by the server's WHATWG parse (`/v1/cache/%2E` → `/v1/cache/`). No wire form reaches the validator." }, { "key": "..", - "encoded": "%2E%2E", - "decoded": "..", - "dot_segment": true, - "note": "All-dot key. Unencoded, `/v1/cache/..` collapses to `/v1/` and `/v1/cache/../ttl` to `/v1/ttl` — a different route, carrying the bearer token, never reaching the key validator. RFC 3986 stacks (httpx) send `%2E%2E` intact and the server rejects the decoded `..`; WHATWG stacks collapse `%2E%2E` as well and MUST reject the key client-side." + "encoded": null, + "decoded": null, + "reject": true, + "note": "All-dot key: unencoded, `/v1/cache/..` collapses to `/v1/` and `/v1/cache/../ttl` to `/v1/ttl`; `%2E%2E` is collapsed the same way by the server (`GET /v1/cache/%2E%2E/health` returns the `/v1/health` response). No wire form reaches the validator." + }, + { + "key": "health", + "encoded": null, + "decoded": null, + "reject": true, + "note": "Route token: `/v1/cache/health` is the health endpoint, so a GET for this key would return the health payload as a cache hit." + }, + { + "key": "ttl", + "encoded": null, + "decoded": null, + "reject": true, + "note": "Route token: a final `ttl` segment selects the TTL sub-resource, so `/v1/cache/ttl` is read as an empty key plus `/ttl`." + }, + { + "key": "lock", + "encoded": null, + "decoded": null, + "reject": true, + "note": "Route token: a final `lock` segment selects the lock sub-resource, so `/v1/cache/lock` is read as an empty key plus `/lock`." }, { "key": "a:..", @@ -73,14 +92,14 @@ "key": "ns:key", "encoded": "ns%3Akey", "decoded": "ns:key", - "note": "Minimal namespaced key: `:` → `%3A`, decoded once server-side before the namespace-shape check." + "note": "`:` → `%3A`, decoded once server-side. The server then rejects it: an `ns:` key must be `ns:{namespace}:{rest}` (spec rule 3), and this one has no `{rest}`." }, { "key": "f(x)!*'", "encoded": "f%28x%29%21%2A%27", "decoded": "f(x)!*'", "encoded_alternates": ["f(x)!*'"], - "note": "Encoder-variance row: `quote(safe=\"\")` and `urlencoding::encode` produce `encoded`; `encodeURIComponent` produces the alternate with `!*'()` raw. Both decode to `key`, both are conformant. The server rejects this key (charset), so the variance never reaches stored data." + "note": "Encoder-variance row (spec rule 4): `quote(safe=\"\")` and `urlencoding::encode` produce `encoded`; `encodeURIComponent` produces the alternate. Both decode to `key`. Server rejects the decoded key (charset)." } ] } diff --git a/tools/path-encoding-verify.py b/tools/path-encoding-verify.py index aad25f5..01a332e 100644 --- a/tools/path-encoding-verify.py +++ b/tools/path-encoding-verify.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 """Validate test-vectors/path-encoding.json with Python stdlib. -spec/saas-api.md § Cache-Key Path Encoding. Every row's `encoded` must be the -reference form (`quote(key, safe="")`, all-dot result rewritten to `%2E`), decode -once back to `key`, carry no raw `/ ? # %`, and be a WHATWG dot segment only when -the row says so. A mutation self-test runs first so the guard cannot degrade to -silently reporting OK (same doctrine as tools/test_wire_format_reference.py). +spec/saas-api.md § Cache-Key Path Encoding. A transmittable row's `encoded` must be +the reference form (`quote(key, safe="")`) and decode once back to `key`; a key whose +reference form is a reserved segment (WHATWG dot segment or route token) must be a +`reject` row with no wire form. A mutation self-test runs first so the guard cannot +degrade to silently reporting OK (same doctrine as tools/test_wire_format_reference.py). """ from __future__ import annotations @@ -21,13 +21,13 @@ ROOT = Path(__file__).resolve().parents[1] VECTORS = ROOT / "test-vectors" / "path-encoding.json" -# Reference form: RFC 3986 unreserved characters and uppercase %HH escapes only. -SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~-]|%[0-9A-F]{2})+$") -# Encoder-variance form: additionally the five sub-delims encodeURIComponent leaves raw -# (legal pchar in a path segment, decode to themselves — spec rule 4). -ALT_SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~!*'()-]|%[0-9A-F]{2})+$") # WHATWG URL Standard § 4.1: single-/double-dot path segments, ASCII case-insensitive. WHATWG_DOT_SEGMENTS = {".", "%2e", "..", ".%2e", "%2e.", "%2e%2e"} +# saas router: `/v1/cache/health` is the health endpoint; a final `ttl`/`lock` selects a sub-resource. +ROUTE_TOKENS = {"health", "ttl", "lock"} +# A conformant alternate wire form: RFC 3986 unreserved, the five sub-delims +# encodeURIComponent leaves raw (spec rule 4), and uppercase %HH escapes. +ALT_SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~!*'()-]|%[0-9A-F]{2})+$") def check(condition: bool, name: str, detail: str) -> None: @@ -36,57 +36,50 @@ def check(condition: bool, name: str, detail: str) -> None: raise ValueError(f"{name}: {detail}") -def reference_encode(key: str) -> str: - """cachekit-py ``CachekitIOBackend._encode_key`` (f000ba3).""" - encoded = quote(key, safe="") - return encoded.replace(".", "%2E") if encoded in (".", "..") else encoded - - -def check_segment(name: str, segment: str, dot_segment: bool, key: str, pattern: re.Pattern[str] = SEGMENT) -> None: - check(pattern.fullmatch(segment) is not None, name, f"{segment!r} has a raw reserved character (or lowercase/incomplete %HH)") - check(segment not in (".", ".."), name, f"{segment!r} is a literal dot segment") - check((segment.lower() in WHATWG_DOT_SEGMENTS) == dot_segment, name, f"{segment!r} WHATWG dot-segment status does not match dot_segment={dot_segment}") - check(unquote(segment) == key, name, f"single decode of {segment!r} != key") +def is_reserved(segment: str) -> bool: + return segment.lower() in WHATWG_DOT_SEGMENTS or segment in ROUTE_TOKENS def verify(document: dict) -> int: for vector in document["vectors"]: key = vector["key"] name = repr(key) - dot_segment = vector.get("dot_segment", False) + reference = quote(key, safe="") + if vector.get("reject"): + check(is_reserved(reference), name, "reject flag on a transmittable key") + check(vector["encoded"] is None and vector["decoded"] is None, name, "reject row carries a wire form") + continue + check(not is_reserved(reference), name, "reserved segment must be a reject row") check(vector["decoded"] == key, name, "decoded != key (interop is defined on the decoded key)") - check(vector["encoded"] == reference_encode(key), name, f"encoded {vector['encoded']!r} != reference {reference_encode(key)!r}") - check_segment(name, vector["encoded"], dot_segment, key) + check(vector["encoded"] == reference, name, f"encoded {vector['encoded']!r} != reference {reference!r}") for alt in vector.get("encoded_alternates", []): - check(alt != vector["encoded"], name, "encoded_alternates repeats encoded") - check_segment(name, alt, dot_segment, key, ALT_SEGMENT) + check(ALT_SEGMENT.fullmatch(alt) is not None, name, f"alternate {alt!r} has a raw reserved character or bad %HH") + check(unquote(alt) == key, name, f"alternate {alt!r} does not decode to key") return len(document["vectors"]) def self_test(document: dict) -> None: - """Each poisoned copy must be rejected — otherwise the verify above is toothless.""" - def poisoned(mutate) -> dict: - doc = copy.deepcopy(document) - mutate(doc["vectors"]) - return doc - + """Each poisoned copy must trip a distinct guard — otherwise verify() is toothless.""" def row(vectors: list, key: str) -> dict: return next(v for v in vectors if v["key"] == key) + def set_field(key: str, field: str, value: object): + return lambda v: row(v, key).__setitem__(field, value) + mutations = { - "raw slash": lambda v: row(v, "x/../../health").__setitem__("encoded", "x/..%2F..%2Fhealth"), - "raw percent": lambda v: row(v, "100%").__setitem__("encoded", "100%"), - "double-encoded": lambda v: row(v, "100%").__setitem__("encoded", "100%2525"), - "literal dot segment": lambda v: row(v, "..").__setitem__("encoded", ".."), - "unflagged WHATWG dot segment": lambda v: row(v, "..").__delitem__("dot_segment"), - "flag on inert row": lambda v: row(v, "a:..").__setitem__("dot_segment", True), - "decoded drift": lambda v: row(v, "ns:key").__setitem__("decoded", "ns:kex"), - "lowercase hex": lambda v: row(v, "ns:key").__setitem__("encoded", "ns%3akey"), - "bad alternate": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), + "encoded drift": set_field("x/../../health", "encoded", "x/..%2F..%2Fhealth"), + "decoded drift": set_field("ns:key", "decoded", "ns:kex"), + "reject row with wire form": set_field("..", "encoded", "%2E%2E"), + "reject flag on transmittable key": set_field("a:..", "reject", True), + "reserved key not flagged": lambda v: row(v, "..").update(reject=False, encoded="%2E%2E", decoded=".."), + "alternate raw slash": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), + "alternate decodes elsewhere": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f%28x%29"), } for label, mutate in mutations.items(): + poisoned = copy.deepcopy(document) + mutate(poisoned["vectors"]) try: - verify(poisoned(mutate)) + verify(poisoned) except ValueError: continue raise ValueError(f"self-test: mutation {label!r} was not rejected") From a0ea1a2e41a1432f0c37dd9b34d71c60ff8c1a49 Mon Sep 17 00:00:00 2001 From: Winston Date: Fri, 4 Sep 2026 13:31:33 +1000 Subject: [PATCH 3/5] docs(saas-api): panel round 2 precision fixes (LAB-2879) Spell the ns:/nsapi: namespace grammar (1-64 chars, non-empty rest) as the deployed validator enforces it; say '.'/'..' where 'all-dot' was imprecise ('...' is all-dot yet transmittable); collapse the verifier's unreachable WHATWG %2e set to the five literal reserved segments quote() can actually emit; make each self-test mutation assert the guard it names; CHANGELOG no longer reads as if cachekit-py#279 introduced the bug. --- CHANGELOG.md | 10 ++++----- spec/saas-api.md | 4 ++-- test-vectors/path-encoding.json | 6 +++--- tools/path-encoding-verify.py | 38 ++++++++++++++++----------------- 4 files changed, 28 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c55ce96..937bc45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,17 +9,17 @@ All notable changes to the CacheKit Protocol Specification. - [`spec/saas-api.md`](spec/saas-api.md) gains a normative **Cache-Key Path Encoding** section. The spec documented `/v1/cache/{key}` and its `/ttl` and `/lock` sub-resources without saying how `{key}` is placed in the path — a - silence that cost three SDK tickets (cachekit-py - [#279](https://github.com/cachekit-io/cachekit-py/pull/279) shipped the raw - key unquoted, CWE-22; cachekit-ts LAB-2877 and cachekit-rs LAB-2878 carry the - same latent all-dot gap). Rules: the key is ONE percent-encoded segment with + silence that cost three SDK tickets (cachekit-py shipped the raw key + unquoted, CWE-22, until + [#279](https://github.com/cachekit-io/cachekit-py/pull/279); cachekit-ts + LAB-2877 and cachekit-rs LAB-2878 carry the same latent `.`/`..` gap). Rules: the key is ONE percent-encoded segment with only RFC 3986 unreserved characters raw (`! * ' ( )` tolerated); the server URL-parses under the WHATWG URL Standard, splits on raw `/`, then decodes exactly once and validates the decoded key; encoders may differ on the sub-delims because interop is defined on the **decoded** key, and every server-accepted key is byte-identical on the wire regardless. - **Reserved segments must be rejected client-side; percent-encoding cannot - save an all-dot key.** The filing premise — encode `.`/`..` as `%2E`/`%2E%2E` + save a `.` or `..` key.** The filing premise — encode `.`/`..` as `%2E`/`%2E%2E` — is false: the server parses the request URL under the WHATWG URL Standard, which treats `%2e`, `%2e%2e`, `.%2e`, `%2e.` (any case) as dot segments. Verified live: `GET api.cachekit.io/v1/cache/%2E%2E/health` returns the diff --git a/spec/saas-api.md b/spec/saas-api.md index 5bf6a8b..fe0db02 100644 --- a/spec/saas-api.md +++ b/spec/saas-api.md @@ -75,13 +75,13 @@ Every endpoint below carries the cache key as a path segment — `/v1/cache/{key **1. One segment, percent-encoded.** `{key}` MUST be exactly one path segment. Clients MUST percent-encode the key's UTF-8 bytes (RFC 3986 §2.1) so that only unreserved characters — `ALPHA / DIGIT / "-" / "." / "_" / "~"` — appear raw. Every other byte MUST be sent as `%HH` — the delimiters `/ ? # %`, `:` (a canonical key carries six), space (`%20`, never `+`), every byte ≥ `0x80` — with one tolerance: the sub-delims `! * ' ( )` MAY be left raw (rule 4). Hex digits SHOULD be uppercase (RFC 3986 §2.1); the server decodes either case. Reference encoders: Python `urllib.parse.quote(key, safe="")`, Rust `urlencoding::encode`, JavaScript `encodeURIComponent`. -**2. Reserved segments MUST be rejected client-side.** A key of exactly `.` or `..` survives rule 1 unchanged (`.` is unreserved) and is a *dot segment*: URL parsers remove it before routing — `/v1/cache/..` becomes `/v1/`, `/v1/cache/../ttl` becomes `/v1/ttl` — so the request lands on a different route, still carrying `Authorization`, and never reaches the key validator. Percent-encoding the dots does not help. The server parses the request URL under the WHATWG URL Standard, which treats an ASCII-case-insensitive `%2e` as a single-dot segment and `%2e%2e`, `.%2e`, `%2e.` as double-dot segments (URL Standard §4.1), so `%2E%2E` is collapsed *server-side* even when the client's own parser (RFC 3986 §5.2.4, e.g. `httpx`) sent it intact; WHATWG clients (`fetch`/undici, browsers, the Workers runtime, rust-url and therefore `reqwest`) collapse it before sending. **No wire form of an all-dot key reaches the validator from any client.** The literal segments `health`, `ttl` and `lock` are route tokens at this level — `/v1/cache/health` is the health endpoint, and a final `ttl` or `lock` segment selects the sub-resource — so a key encoding to one of those words is routed elsewhere or read as an empty key. +**2. Reserved segments MUST be rejected client-side.** A key of exactly `.` or `..` survives rule 1 unchanged (`.` is unreserved) and is a *dot segment*: URL parsers remove it before routing — `/v1/cache/..` becomes `/v1/`, `/v1/cache/../ttl` becomes `/v1/ttl` — so the request lands on a different route, still carrying `Authorization`, and never reaches the key validator. Percent-encoding the dots does not help. The server parses the request URL under the WHATWG URL Standard, which treats an ASCII-case-insensitive `%2e` as a single-dot segment and `%2e%2e`, `.%2e`, `%2e.` as double-dot segments (URL Standard §4.1), so `%2E%2E` is collapsed *server-side* even when the client's own parser (RFC 3986 §5.2.4, e.g. `httpx`) sent it intact; WHATWG clients (`fetch`/undici, browsers, the Workers runtime, rust-url and therefore `reqwest`) collapse it before sending. **No wire form of a `.` or `..` key reaches the validator from any client.** The literal segments `health`, `ttl` and `lock` are route tokens at this level — `/v1/cache/health` is the health endpoint, and a final `ttl` or `lock` segment selects the sub-resource — so a key encoding to one of those words is routed elsewhere or read as an empty key. Therefore clients MUST reject a key whose encoded form is exactly `.`, `..`, `health`, `ttl` or `lock` before building the URL, surfacing a client-side error; servers MUST NOT be relied on to compensate. Only an *entirely*-dot segment is a dot segment: `a:..`, `..a`, `x..y` are inert and MUST be sent per rule 1 with their dots raw. Canonical and interop keys always contain `:` and never meet this rule. Conformance tests MUST assert on the *parsed* request path (`httpx.Request.url.raw_path`, `new URL(u).pathname`, `Url::parse(u)?.path()`), not on the un-parsed template string — a template-string test passes while the traversal ships. > **Evidence (2026-09-04):** against `api.cachekit.io`, `GET /v1/cache/%2E%2E/health` returns the `/v1/health` response and `/v1/cache/%2E%2E/ttl` is routed as `/v1/ttl`, while `/v1/cache/a%3A..%2Fb/ttl` reaches the cache route. `httpx` 0.28.1 sends `%2E%2E` unchanged; Node 25 `new URL()` and rust-url 2.5.8 collapse it client-side. A fix proven on one parser is not proof for the other, and cachekit-py's `%2E` rewrite ([cachekit-py#279](https://github.com/cachekit-io/cachekit-py/pull/279), v0.18.0) moves the collapse from client to server rather than preventing it (LAB-2880). -**3. The server decodes exactly once.** After the WHATWG parse of rule 2, the server splits the path on raw `/`, then percent-decodes the key segment once (`decodeURIComponent`-equivalent; a malformed escape is `400`) and validates the *decoded* key: non-empty, within an implementation-defined maximum length (the deployed cap exceeds the 250-character SDK key limit in [cache-key-format.md](cache-key-format.md#key-length-limits)), drawn from `[A-Za-z0-9_.:-]`, free of the substring `..`, and — for `ns:` / `nsapi:` keys — of the shape `{prefix}:{namespace}:{rest}` with a non-empty namespace drawn from `[A-Za-z0-9_-]`. Anything else is `400 Bad Request`. Consequences clients MUST honour: +**3. The server decodes exactly once.** After the WHATWG parse of rule 2, the server splits the path on raw `/`, then percent-decodes the key segment once (`decodeURIComponent`-equivalent; a malformed escape is `400`) and validates the *decoded* key: non-empty, within an implementation-defined maximum length (the deployed cap exceeds the 250-character SDK key limit in [cache-key-format.md](cache-key-format.md#key-length-limits)), drawn from `[A-Za-z0-9_.:-]`, free of the substring `..`, and — for `ns:` / `nsapi:` keys — of the shape `{prefix}:{namespace}:{rest}` with a 1–64-character namespace drawn from `[A-Za-z0-9_-]` and a non-empty `{rest}`. Anything else is `400 Bad Request`. Consequences clients MUST honour: - Clients MUST NOT double-encode. A literal `%` in a key is sent as `%25` once; `%2525` decodes to `%25`, a different key. - An encoded `%2F` never becomes a segment boundary: the split on raw `/` happens *before* decoding, so `a%2Fb` reaches the validator as `a/b` and is rejected by the charset rule. A conformant client can neither traverse nor store a key containing `/`. diff --git a/test-vectors/path-encoding.json b/test-vectors/path-encoding.json index 9231f75..3837a80 100644 --- a/test-vectors/path-encoding.json +++ b/test-vectors/path-encoding.json @@ -1,6 +1,6 @@ { "version": "1.0.0", - "generator": "urllib.parse.quote(key, safe=\"\") — cachekit-py v0.18.0 CachekitIOBackend._encode_key (f000ba3) minus its %2E all-dot rewrite, which the server's WHATWG URL parse collapses; all-dot keys are reject rows", + "generator": "urllib.parse.quote(key, safe=\"\") — cachekit-py v0.18.0 CachekitIOBackend._encode_key (f000ba3) minus its `.`/`..` → %2E rewrite, which the server's WHATWG URL parse collapses; `.` and `..` are reject rows", "spec": "spec/saas-api.md § Cache-Key Path Encoding", "ci_verification": "tools/path-encoding-verify.py (stdlib only; runs in this repo's verify.yml; mutation self-test first)", "contract": "`encoded` is the single `{key}` path segment in the reference form (spec rule 1); `decoded` is the key the server sees after its single percent-decode and equals `key` in every transmittable row (spec rules 3-4). Rows with `reject: true` are the reserved segments of spec rule 2 (`.`, `..`, `health`, `ttl`, `lock`): a conformant client raises before building the URL, so `encoded` and `decoded` are null. `encoded_alternates` lists the other conformant wire form where `encodeURIComponent` differs (`! * ' ( )` raw); assert `encoded in [encoded] + encoded_alternates`.", @@ -46,14 +46,14 @@ "encoded": null, "decoded": null, "reject": true, - "note": "All-dot key: the plain encoding `.` is a dot segment, and `%2E` is collapsed by the server's WHATWG parse (`/v1/cache/%2E` → `/v1/cache/`). No wire form reaches the validator." + "note": "Dot segment: the plain encoding `.` is removed by every URL parser, and `%2E` is collapsed by the server's WHATWG parse (`/v1/cache/%2E` → `/v1/cache/`). No wire form reaches the validator." }, { "key": "..", "encoded": null, "decoded": null, "reject": true, - "note": "All-dot key: unencoded, `/v1/cache/..` collapses to `/v1/` and `/v1/cache/../ttl` to `/v1/ttl`; `%2E%2E` is collapsed the same way by the server (`GET /v1/cache/%2E%2E/health` returns the `/v1/health` response). No wire form reaches the validator." + "note": "Dot segment: unencoded, `/v1/cache/..` collapses to `/v1/` and `/v1/cache/../ttl` to `/v1/ttl`; `%2E%2E` is collapsed the same way by the server (`GET /v1/cache/%2E%2E/health` returns the `/v1/health` response). No wire form reaches the validator." }, { "key": "health", diff --git a/tools/path-encoding-verify.py b/tools/path-encoding-verify.py index 01a332e..79700ba 100644 --- a/tools/path-encoding-verify.py +++ b/tools/path-encoding-verify.py @@ -5,7 +5,7 @@ the reference form (`quote(key, safe="")`) and decode once back to `key`; a key whose reference form is a reserved segment (WHATWG dot segment or route token) must be a `reject` row with no wire form. A mutation self-test runs first so the guard cannot -degrade to silently reporting OK (same doctrine as tools/test_wire_format_reference.py). +degrade to silently reporting OK, and each mutation must trip the guard it names (same doctrine as tools/test_wire_format_reference.py). """ from __future__ import annotations @@ -21,10 +21,10 @@ ROOT = Path(__file__).resolve().parents[1] VECTORS = ROOT / "test-vectors" / "path-encoding.json" -# WHATWG URL Standard § 4.1: single-/double-dot path segments, ASCII case-insensitive. -WHATWG_DOT_SEGMENTS = {".", "%2e", "..", ".%2e", "%2e.", "%2e%2e"} -# saas router: `/v1/cache/health` is the health endpoint; a final `ttl`/`lock` selects a sub-resource. -ROUTE_TOKENS = {"health", "ttl", "lock"} +# spec rule 2. `.`/`..` are dot segments (the server's WHATWG parse also collapses the +# `%2e` forms, but quote() never emits those for `.`, so only the literals can appear here); +# `health`/`ttl`/`lock` are saas route tokens at the `/v1/cache/` level. +RESERVED_SEGMENTS = {".", "..", "health", "ttl", "lock"} # A conformant alternate wire form: RFC 3986 unreserved, the five sub-delims # encodeURIComponent leaves raw (spec rule 4), and uppercase %HH escapes. ALT_SEGMENT = re.compile(r"^(?:[A-Za-z0-9._~!*'()-]|%[0-9A-F]{2})+$") @@ -36,20 +36,16 @@ def check(condition: bool, name: str, detail: str) -> None: raise ValueError(f"{name}: {detail}") -def is_reserved(segment: str) -> bool: - return segment.lower() in WHATWG_DOT_SEGMENTS or segment in ROUTE_TOKENS - - def verify(document: dict) -> int: for vector in document["vectors"]: key = vector["key"] name = repr(key) reference = quote(key, safe="") if vector.get("reject"): - check(is_reserved(reference), name, "reject flag on a transmittable key") + check(reference in RESERVED_SEGMENTS, name, "reject flag on a transmittable key") check(vector["encoded"] is None and vector["decoded"] is None, name, "reject row carries a wire form") continue - check(not is_reserved(reference), name, "reserved segment must be a reject row") + check(reference not in RESERVED_SEGMENTS, name, "reserved segment must be a reject row") check(vector["decoded"] == key, name, "decoded != key (interop is defined on the decoded key)") check(vector["encoded"] == reference, name, f"encoded {vector['encoded']!r} != reference {reference!r}") for alt in vector.get("encoded_alternates", []): @@ -66,21 +62,23 @@ def row(vectors: list, key: str) -> dict: def set_field(key: str, field: str, value: object): return lambda v: row(v, key).__setitem__(field, value) + # label: (poison, substring the tripped guard's message must contain) mutations = { - "encoded drift": set_field("x/../../health", "encoded", "x/..%2F..%2Fhealth"), - "decoded drift": set_field("ns:key", "decoded", "ns:kex"), - "reject row with wire form": set_field("..", "encoded", "%2E%2E"), - "reject flag on transmittable key": set_field("a:..", "reject", True), - "reserved key not flagged": lambda v: row(v, "..").update(reject=False, encoded="%2E%2E", decoded=".."), - "alternate raw slash": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), - "alternate decodes elsewhere": lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f%28x%29"), + "encoded drift": (set_field("x/../../health", "encoded", "x/..%2F..%2Fhealth"), "!= reference"), + "decoded drift": (set_field("ns:key", "decoded", "ns:kex"), "decoded != key"), + "reject row with wire form": (set_field("..", "encoded", "%2E%2E"), "carries a wire form"), + "reject flag on transmittable key": (set_field("a:..", "reject", True), "reject flag on a transmittable"), + "reserved key not flagged": (lambda v: row(v, "..").update(reject=False, encoded="%2E%2E", decoded=".."), "must be a reject row"), + "alternate raw slash": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), "raw reserved character"), + "alternate decodes elsewhere": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f%28x%29"), "does not decode to key"), } - for label, mutate in mutations.items(): + for label, (mutate, expected) in mutations.items(): poisoned = copy.deepcopy(document) mutate(poisoned["vectors"]) try: verify(poisoned) - except ValueError: + except ValueError as exc: + check(expected in str(exc), "self-test", f"mutation {label!r} tripped the wrong guard: {exc}") continue raise ValueError(f"self-test: mutation {label!r} was not rejected") From 6abf38d6bdf4d7be27cf56a7df6f89395350f7e0 Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Fri, 4 Sep 2026 16:11:33 +1000 Subject: [PATCH 4/5] =?UTF-8?q?docs(saas-api):=20harden=20path-encoding=20?= =?UTF-8?q?verifier=20=E2=80=94=20reject=20non-UTF-8=20escapes=20and=20dup?= =?UTF-8?q?licate=20alternates=20(LAB-2879)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round on head a0ea1a2 surfaced two real gaps in the alternates loop of tools/path-encoding-verify.py, both fixed here: - Non-UTF-8 escape accepted (CodeRabbit, functional correctness). unquote() defaults to errors="replace", so unquote("%FF") returns U+FFFD instead of rejecting it. A vector with key U+FFFD and alternate "%FF" would pass, though "%FF" is not valid UTF-8 and violates spec rule 1. New decodes_to() helper percent-decodes with errors="strict" and treats a UnicodeDecodeError as a non-match, so an invalid escape can never masquerade as a conformant alternate. - Duplicate alternate accepted (Kody, correctness). The rewrite dropped the alt != encoded distinctness guard, so an encoded_alternates entry that repeats the row's reference encoded form passed silently, weakening spec rule 4 (an alternate is a distinct conformant wire form). Guard re-added as the first check in the loop. self_test gains two mutations — "alternate repeats encoded" and "alternate non-utf8 escape" — so each new guard is proven to trip, per the file's own doctrine that every guard has a poisoned-copy mutation. Also sorted the import block (ruff I001). The FBT001/FBT003/TRY003 and EXE001/LOG015 that CodeRabbit's assertive-profile ruff reports are the same patterns the already-merged sibling tools/file-backend-reference.py carries; rebutted on the PR rather than diverged from the established verifier convention in one file. No CI ruff gate exists; default ruff check is clean bar EXE001/LOG015. Verifier passes: self-test + 15 vectors. Spec text (saas-api.md) unchanged — the cache-key format the expert panel blessed in rounds 1-2 is untouched; this is test-tooling hardening only. --- tools/path-encoding-verify.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/tools/path-encoding-verify.py b/tools/path-encoding-verify.py index 79700ba..9155b16 100644 --- a/tools/path-encoding-verify.py +++ b/tools/path-encoding-verify.py @@ -13,9 +13,9 @@ import copy import json import logging -from pathlib import Path import re import sys +from pathlib import Path from urllib.parse import quote, unquote ROOT = Path(__file__).resolve().parents[1] @@ -36,6 +36,19 @@ def check(condition: bool, name: str, detail: str) -> None: raise ValueError(f"{name}: {detail}") +def decodes_to(segment: str, key: str) -> bool: + """True iff percent-unescaping ``segment`` yields exactly ``key`` as valid UTF-8. + + ``unquote`` defaults to ``errors="replace"``, which maps a non-UTF-8 escape such as + ``%FF`` to U+FFFD instead of rejecting it (spec rule 1 forbids non-UTF-8 wire forms). + Decode strictly so an invalid escape can never masquerade as a conformant alternate. + """ + try: + return unquote(segment, errors="strict") == key + except UnicodeDecodeError: + return False + + def verify(document: dict) -> int: for vector in document["vectors"]: key = vector["key"] @@ -49,8 +62,9 @@ def verify(document: dict) -> int: check(vector["decoded"] == key, name, "decoded != key (interop is defined on the decoded key)") check(vector["encoded"] == reference, name, f"encoded {vector['encoded']!r} != reference {reference!r}") for alt in vector.get("encoded_alternates", []): + check(alt != vector["encoded"], name, f"alternate {alt!r} repeats the reference encoded form") check(ALT_SEGMENT.fullmatch(alt) is not None, name, f"alternate {alt!r} has a raw reserved character or bad %HH") - check(unquote(alt) == key, name, f"alternate {alt!r} does not decode to key") + check(decodes_to(alt, key), name, f"alternate {alt!r} does not decode to key") return len(document["vectors"]) @@ -71,6 +85,8 @@ def set_field(key: str, field: str, value: object): "reserved key not flagged": (lambda v: row(v, "..").update(reject=False, encoded="%2E%2E", decoded=".."), "must be a reject row"), "alternate raw slash": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f(x)!*'/"), "raw reserved character"), "alternate decodes elsewhere": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("f%28x%29"), "does not decode to key"), + "alternate repeats encoded": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append(row(v, "f(x)!*'")["encoded"]), "repeats the reference"), + "alternate non-utf8 escape": (lambda v: row(v, "f(x)!*'")["encoded_alternates"].append("%FF"), "does not decode to key"), } for label, (mutate, expected) in mutations.items(): poisoned = copy.deepcopy(document) From 54dde277bfa84f20f0c6468cc4a32adee7eff93a Mon Sep 17 00:00:00 2001 From: Ray Walker Date: Fri, 4 Sep 2026 18:05:35 +1000 Subject: [PATCH 5/5] =?UTF-8?q?docs(saas-api):=20fix=20changelog=20mutatio?= =?UTF-8?q?n=20count=20(7=E2=86=929)=20+=20annotate=20set=5Ffield=20return?= =?UTF-8?q?=20(LAB-2879)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit @6abf38d, 2 actionable MINOR: - CHANGELOG: self-test count 7 → 9 (matches mutations dict) - path-encoding-verify.py: ANN202 return annotation on nested set_field --- CHANGELOG.md | 2 +- tools/path-encoding-verify.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 937bc45..ef1a403 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,7 @@ All notable changes to the CacheKit Protocol Specification. reserved segments with no wire form, one `encoded_alternates` row for the `encodeURIComponent` form), CI-verified by [`tools/path-encoding-verify.py`](tools/path-encoding-verify.py) (stdlib; - 7-mutation self-test first, each mutation tripping a distinct guard). + 9-mutation self-test first, each mutation tripping a distinct guard). - [`sdk-feature-matrix.md`](sdk-feature-matrix.md) Compliance Status gains a path-encoding row with actual state: all three SDKs percent-encode one segment but none yet rejects the reserved segments — Python's v0.18.0 `%2E` diff --git a/tools/path-encoding-verify.py b/tools/path-encoding-verify.py index 9155b16..2357484 100644 --- a/tools/path-encoding-verify.py +++ b/tools/path-encoding-verify.py @@ -15,6 +15,7 @@ import logging import re import sys +from collections.abc import Callable from pathlib import Path from urllib.parse import quote, unquote @@ -73,7 +74,7 @@ def self_test(document: dict) -> None: def row(vectors: list, key: str) -> dict: return next(v for v in vectors if v["key"] == key) - def set_field(key: str, field: str, value: object): + def set_field(key: str, field: str, value: object) -> Callable[[list], None]: return lambda v: row(v, key).__setitem__(field, value) # label: (poison, substring the tripped guard's message must contain)