Skip to content

fix(plugin): never read an empty peer list as "this node owns every URL" - #121

Merged
harper-joseph merged 1 commit into
mainfrom
fix/residency-peerless-node-list
Aug 21, 2026
Merged

fix(plugin): never read an empty peer list as "this node owns every URL"#121
harper-joseph merged 1 commit into
mainfrom
fix/residency-peerless-node-list

Conversation

@harper-joseph

@harper-joseph harper-joseph commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Residency is the shard map for the entire RenderSchedule keyspace, and it was read once at module load from server.nodes — a value Harper initialises to [] and only fills when subscribeToNodeUpdates runs its hdb_nodes scan. Anything evaluated before that scan sees no peers, and with no peers rendezvous hashing makes the local node the owner of every URL, so every schedule row that worker wrote was stored locally instead of routed to its owner. The list is now read per call, and a list with no usable peer in it means "not known yet" rather than "this node is alone."

Found in production, not in review: a multi-percent share of a corpus misplaced, in bursts that each end within seconds of a node restart. The failure is silent and self-sustaining — nothing deletes a schedule row from a node that does not own it, and the non-owner's own reschedule routes away from the row it just claimed, so it re-claims and re-renders the same URL forever while the real owner never gets the row. The ordering that causes it repeats on every worker start, not only the first, which is why one node can carry far more of it than its peers.

For the human reviewer

  1. A read with no usable peer falls back and warns; it does not throw. Throwing would be the loud option and would prevent the one case this PR does not fix — a process that has never seen a peer at all — but a genuine single-node deployment has no peers forever, and a throw from a residency function makes Harper drop the record entirely rather than misplace it, trading silent misplacement for silent loss. Reversible in one branch. Ruling the other way costs you failed schedule writes wherever the peer list is legitimately empty.
  2. Any list containing a peer is adopted immediately, rather than a monotonic never-shrink union. Monotonic would be safer against transient states, but a retired node then stays in the ring until every process restarts, and retiring a node is an anticipated operation. I checked the thing this depends on rather than assuming it: rebuildKnownNodes refills with an incremental push per row, which would expose short non-empty rings — but scanNodesForSubscription is a plain synchronous for...of with no await, and the per-update path filters and pushes before its first await, so both are atomic to any observer. If either ever becomes async, this decision needs revisiting and the module comment says so.
  3. The cache is keyed on server.nodes array identity plus length. This is a hot path — reconcile and orphanSweep call it once per row across a whole corpus — so hashing names per call was rejected. The known gap is a same-length in-place rename, which knownNodes.ts has no path for: it reassigns (filter-then-push) or pushes, and both change identity or length.
  4. Existing misplaced rows are deliberately left in place. This PR stops new ones; deleting rows a node does not own is a data operation on live tables and wants its own change and its own review. Nothing here depends on that cleanup. Target.schedulerNode persists what residency answered at write time, so affected rows are identifiable by comparing that field against the computed owner.

Two smaller faults on the same path are fixed in passing, both reachable independently of the main bug: a nameless node descriptor (which knownNodes.ts can push after a decode miss) entered the ring as undefined and, when it won the hash, produced a residency excluding every node — Harper then stored the record nowhere; and peer.js's isKnownNode security guard read the same stale snapshot and rejected legitimate peers, breaking the explainer's cross-node fetch.

Verification

Route: unit tests plus a live measurement. The trigger — a module evaluated before its worker's hdb_nodes scan — is a startup ordering race that is not reproducible in CI, so it is not observable end-to-end here. That is stated rather than papered over.

  • packages/plugin/test/residency.test.js, 13 new tests. 9 fail on base when the base implementation is restored behind a getNodes shim so only the defect is under test — including the peer-list-after-load case, the nameless descriptor, the all-nameless list, and the isKnownNode regression. The other 4 pass on base by design: they pin invariants this change must not break (HRW agreement across nodes, the mapping fixture, and the two properties the frozen array got for free).
  • Full plugin suite: 776 pass, 0 fail (node --test in packages/plugin), rebased onto main after feat(plugin): decide the render order from a scored ready set, not from the index #120 — none of its 48 new tests are affected, and it adds no residency read of its own.
  • The mapping is pinned by a four-node fixture, one key per node, so no future refactor can silently re-shard a running keyspace — the hazard util/hash.js already warns about for the mixing constant. A separate test proves deduplicating a repeated node name moves no key, since HRW takes the max over the names present.
  • Live evidence for the diagnosis: per-row write timestamps (record.getUpdatedTime()), sampled evenly across each node's full misplaced set, cluster in tight bursts — one node put ~96% of its share into a single minute during cluster bring-up — and every node's last misplaced write lands seconds before that node's process start.
  • npm run format:write and eslint clean on all touched files. Three pre-existing no-unused-vars errors in packages/console/test/ are present on main and left alone as out of scope.

Review coverage

Authored by Opus 5. No independent cross-model coverage — every outside leg failed in this environment. From the run's own log: gemini ✗ (agy not on PATH), cursor-grok ✗ (cursor-agent --version exited -127), cursor-composer ✗ (pruned), codex ✗ (auth, rc=1). The CLI then offered a same-family Claude pass, which it labels NOT independent and which is therefore not counted here. No per-SHA receipt was written, so the review-need field below fails closed to grade 4 — correctly.

What this PR has instead is a self-review, which is not a substitute. It did find real defects after the first commit: a peerless branch allocating a fresh array on every call (the single-node case hits that path every call); a populated server.nodes of only nameless descriptors resolving to self-only with no warning and overwriting a good list; and an overclaim in my own rationale — I had described the hdb_nodes rebuild and per-update paths as observable windows, and checking them showed both are synchronous and therefore atomic, which narrows the mechanism to the startup ordering race alone. All three are corrected here. A fourth defect, a logger?.warn?.() guard that does not protect an undeclared identifier, was caught by the existing suite rather than by me.

The outside lenses this change most wants are the GitHub review bots on push, and a human on the judgment call in item 2.

Human-Review-Need: 4 @ fb377e0

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request bumps the package version to 0.49.1 and refactors the node residency resolution logic in @harperfast/prerender. Instead of statically capturing the cluster's node list at module load, the implementation now lazily reads and caches server.nodes dynamically. This prevents transient empty peer lists (e.g., during subscription restarts) from being incorrectly treated as a single-node deployment, which previously caused schedule rows to be stored locally instead of being routed to their correct owners. Comprehensive unit tests have been added to verify this behavior. There are no review comments to address, and I have no additional feedback to provide.

@harper-joseph
harper-joseph force-pushed the fix/residency-peerless-node-list branch from 17e11cd to d21b387 Compare August 21, 2026 20:48
…RL"; v0.50.1

`util/residency.js` captured the cluster node list once, at module load, from
`server.nodes`. That value is derived state, not configuration: harper-pro
initialises it to `[]` at module scope and only fills it when
`subscribeToNodeUpdates` runs a full `hdb_nodes` scan, so anything evaluated
before that scan sees no peers on a healthy multi-node cluster — and that
ordering repeats for every worker start, not only the first.

With no peers, rendezvous hashing makes the local node the owner of every URL.
`setResidencyById` then reported self for every key, so Harper stored every
RenderSchedule row locally instead of routing it to its owner — silently and
permanently, because nothing deletes a schedule row from a node that does not
own it, and the non-owner's own reschedule routes away from the row it just
claimed. One unlucky evaluation therefore poisoned a worker for its whole
lifetime; this has been observed in production affecting a multi-percent share
of a corpus.

The list is now read per call. A list with no usable peer in it means "not known
yet", whether it is empty or populated only by nameless descriptors: the last
list that did contain a peer wins over it, while any list that contains one is
adopted at once so a genuine membership change still takes effect. A node that
has never seen a peer warns and owns everything — correct for a single-node
deployment, a misconfiguration on a cluster — rather than throwing, which would
make Harper drop the record instead of misplacing it.

Note that the later `hdb_nodes` rebuilds are NOT windows of their own: the scan
is a plain synchronous `for...of`, and the per-update path filters and pushes
before its first await, so both are atomic to any observer. They matter only as
evidence that this list is rebuilt from a table rather than fixed at boot, which
is why capturing it once is wrong in principle.

Also fixes two smaller faults on the same path:

- A nameless node descriptor (which `knownNodes.ts` can push after a decode miss)
  entered the ring as `undefined`, and returning it read as a residency excluding
  every node, so Harper stored the record nowhere at all.
- `peer.js`'s `isKnownNode` guard read the same stale snapshot and rejected
  legitimate peers, breaking the explainer's cross-node fetch. It reads through
  the new accessor and heals with it.

The hash and the mapping are untouched. Duplicate names are now collapsed, which
cannot move a key because HRW takes the max over the names present; a test pins
that, and a four-node mapping fixture, so no refactor can silently re-shard a
running keyspace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@harper-joseph
harper-joseph force-pushed the fix/residency-peerless-node-list branch from d21b387 to fb377e0 Compare August 21, 2026 20:52
@harper-joseph
harper-joseph marked this pull request as ready for review August 21, 2026 20:53
@harper-joseph
harper-joseph merged commit fd4f828 into main Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant