writeSchedule lowers only the writing node's claim floor, but rows are residency-routed to their owner — ~75% of writes can strand rows silently

[harper-joseph]

Rewritten 2026-08-19. The first version of this issue blamed replication. That was wrong —
RenderSchedule is not replicated; rows are residency-routed and each key lives on exactly one
node (verified: a 27-key probe returned 1/16/10/0 across four nodes, perfectly disjoint). The
actual mechanism is below, and it is worse, because it fires from ordinary in-plugin writes.

Summary

writeSchedule lowers the claim floor of the node executing the write. But the row it just
wrote is residency-routed to whichever node owns that cacheKey. On a 4-node cluster ~75% of
writes therefore lower a floor belonging to the wrong node, and the owner's floor is never lowered
at all.

Any row filed with a due minute older than queue.claimFloor.guard (default 5 min) lands below
its owner's floor and is never claimed again — silently, permanently, from a fully funnel-routed
write.

The mechanism

export const writeSchedule = async (cacheKey, { nextRenderTime, fromSitemap } = {}) => {
	...
	await scheduleTable().put(cacheKey, { nextRenderTime, fromSitemap });
	lowerFloorFor(nextRenderTime);   // ← local SharedBuffer, i.e. THIS node's floor
};

lowerFloorFor → leaseTable().lowerFloorTo(...) is an Atomics CAS-min against the local
coordination.SharedBuffer (replicate: false, node-local by construction). The put, however,
is residency-routed — and per the module comment on writeSchedule, deliberately does not block
on the owner:

a write to a residency-pinned key this node does not own does NOT block on the owner (measured:
500 writes in 10.7 ms, mean 0.021 ms, against residency pinned to a node that does not exist)

Ownership is getResidencyByUrl → fnv1a32(\${url}|${node}`)` rendezvous hashing, so for any
given writer 3 of 4 keys are owned elsewhere.

This is already documented in the tree

resources/Target.js, in revalidate's phase 2 — the hazard is known and currently mitigated
only by writers remembering to recompute the minute per URL:

THE CURRENT MINUTE, PER URL — never captured once for the whole sweep. Phase 2 writes up to
scan.collectCap × devices rows with a PrerenderedPage.get per key, which at scale takes tens
of minutes. Rows are residency-routed, so ~75% land on nodes whose claim floor this process
cannot lower and which hold it at nowMinute − queue.claimFloor.guard: every row filed with a
minute more than the guard band old lands BELOW the owner's floor and is never claimed again —
silently, from a fully funnel-routed in-plugin write, and permanently where resetInterval: 0.
Sitemap.js already computes it per entry for the same reason.

So the invariant "never let more than guard elapse between computing a due minute and writing
it" is load-bearing across every schedule writer, enforced nowhere, and unobservable when broken.
Two call sites currently carry hand-written comments to remember it.

Impact

Observed on a 4-node production cluster (plugin 0.49.0 / harper-pro 5.2.3). One node accumulated
92,873 rows below its floor across 2,101 distinct due-minutes, oldest due 2026-08-01 — 18 days
of a growing slice of its shard silently not rendering, while the node reported healthy and kept
serving. Peer nodes were at 507 / 3 / 1 overdue.

Downstream, those keys held damaged blobs that could only be replaced by a re-render, so
replication base copies from the affected node latched indefinitely (harper-pro#699 territory) —
a queue bug surfacing as a storage symptom.

Why the nominated recovery does not cover it

queue.claimFloor.resetInterval's docstring names the out-of-plugin write paths it exists to
cover — the operations API and the exported REST surface. Residency-routed in-plugin writes
are not mentioned, and they are the high-volume path. The reset is also worker-0-gated inside
syncQueueState (see the open question in #110).

Possible directions

1. Lower the owner's floor, not the writer's. The write already routes to the owner; the floor
   lowering needs to ride with it rather than being applied locally. That is the fix that makes
   the invariant unnecessary.
2. Bound the forward advance so a floor can never outrun rows the pass never scanned (Orphaned nextRenderTime index entries make the claim scan non-monotonic; one break at row 0 silently stalls a node's entire backlog (2 of 4 nodes affected) #110).
   That alone would stop the stranding even if the lowering stays local.
3. Make the below-floor condition observable. The backlog snapshot already computes
   below_floor; a node with essentially its whole due backlog below the floor should alarm.
4. Failing 1, at minimum assert the invariant at the funnel — reject or warn on a writeSchedule
   whose nextRenderTime minute is already older than guard, so a slow batch fails loudly
   instead of silently stranding rows on a peer.