fix(plugin): never read an empty peer list as "this node owns every URL" - #121
Merged
Merged
Conversation
There was a problem hiding this comment.
Code Review
This pull request bumps the package version to 0.49.1 and refactors the node residency resolution logic in @harperfast/prerender. Instead of statically capturing the cluster's node list at module load, the implementation now lazily reads and caches server.nodes dynamically. This prevents transient empty peer lists (e.g., during subscription restarts) from being incorrectly treated as a single-node deployment, which previously caused schedule rows to be stored locally instead of being routed to their correct owners. Comprehensive unit tests have been added to verify this behavior. There are no review comments to address, and I have no additional feedback to provide.
harper-joseph
force-pushed
the
fix/residency-peerless-node-list
branch
from
August 21, 2026 20:48
17e11cd to
d21b387
Compare
…RL"; v0.50.1 `util/residency.js` captured the cluster node list once, at module load, from `server.nodes`. That value is derived state, not configuration: harper-pro initialises it to `[]` at module scope and only fills it when `subscribeToNodeUpdates` runs a full `hdb_nodes` scan, so anything evaluated before that scan sees no peers on a healthy multi-node cluster — and that ordering repeats for every worker start, not only the first. With no peers, rendezvous hashing makes the local node the owner of every URL. `setResidencyById` then reported self for every key, so Harper stored every RenderSchedule row locally instead of routing it to its owner — silently and permanently, because nothing deletes a schedule row from a node that does not own it, and the non-owner's own reschedule routes away from the row it just claimed. One unlucky evaluation therefore poisoned a worker for its whole lifetime; this has been observed in production affecting a multi-percent share of a corpus. The list is now read per call. A list with no usable peer in it means "not known yet", whether it is empty or populated only by nameless descriptors: the last list that did contain a peer wins over it, while any list that contains one is adopted at once so a genuine membership change still takes effect. A node that has never seen a peer warns and owns everything — correct for a single-node deployment, a misconfiguration on a cluster — rather than throwing, which would make Harper drop the record instead of misplacing it. Note that the later `hdb_nodes` rebuilds are NOT windows of their own: the scan is a plain synchronous `for...of`, and the per-update path filters and pushes before its first await, so both are atomic to any observer. They matter only as evidence that this list is rebuilt from a table rather than fixed at boot, which is why capturing it once is wrong in principle. Also fixes two smaller faults on the same path: - A nameless node descriptor (which `knownNodes.ts` can push after a decode miss) entered the ring as `undefined`, and returning it read as a residency excluding every node, so Harper stored the record nowhere at all. - `peer.js`'s `isKnownNode` guard read the same stale snapshot and rejected legitimate peers, breaking the explainer's cross-node fetch. It reads through the new accessor and heals with it. The hash and the mapping are untouched. Duplicate names are now collapsed, which cannot move a key because HRW takes the max over the names present; a test pins that, and a four-node mapping fixture, so no refactor can silently re-shard a running keyspace. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
harper-joseph
force-pushed
the
fix/residency-peerless-node-list
branch
from
August 21, 2026 20:52
d21b387 to
fb377e0
Compare
harper-joseph
marked this pull request as ready for review
August 21, 2026 20:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Residency is the shard map for the entire
RenderSchedulekeyspace, and it was read once at module load fromserver.nodes— a value Harper initialises to[]and only fills whensubscribeToNodeUpdatesruns itshdb_nodesscan. Anything evaluated before that scan sees no peers, and with no peers rendezvous hashing makes the local node the owner of every URL, so every schedule row that worker wrote was stored locally instead of routed to its owner. The list is now read per call, and a list with no usable peer in it means "not known yet" rather than "this node is alone."Found in production, not in review: a multi-percent share of a corpus misplaced, in bursts that each end within seconds of a node restart. The failure is silent and self-sustaining — nothing deletes a schedule row from a node that does not own it, and the non-owner's own reschedule routes away from the row it just claimed, so it re-claims and re-renders the same URL forever while the real owner never gets the row. The ordering that causes it repeats on every worker start, not only the first, which is why one node can carry far more of it than its peers.
For the human reviewer
rebuildKnownNodesrefills with an incrementalpushper row, which would expose short non-empty rings — butscanNodesForSubscriptionis a plain synchronousfor...ofwith no await, and the per-update path filters and pushes before its first await, so both are atomic to any observer. If either ever becomes async, this decision needs revisiting and the module comment says so.server.nodesarray identity plus length. This is a hot path —reconcileandorphanSweepcall it once per row across a whole corpus — so hashing names per call was rejected. The known gap is a same-length in-place rename, whichknownNodes.tshas no path for: it reassigns (filter-then-push) or pushes, and both change identity or length.Target.schedulerNodepersists what residency answered at write time, so affected rows are identifiable by comparing that field against the computed owner.Two smaller faults on the same path are fixed in passing, both reachable independently of the main bug: a nameless node descriptor (which
knownNodes.tscan push after a decode miss) entered the ring asundefinedand, when it won the hash, produced a residency excluding every node — Harper then stored the record nowhere; andpeer.js'sisKnownNodesecurity guard read the same stale snapshot and rejected legitimate peers, breaking the explainer's cross-node fetch.Verification
Route: unit tests plus a live measurement. The trigger — a module evaluated before its worker's
hdb_nodesscan — is a startup ordering race that is not reproducible in CI, so it is not observable end-to-end here. That is stated rather than papered over.packages/plugin/test/residency.test.js, 13 new tests. 9 fail on base when the base implementation is restored behind agetNodesshim so only the defect is under test — including the peer-list-after-load case, the nameless descriptor, the all-nameless list, and theisKnownNoderegression. The other 4 pass on base by design: they pin invariants this change must not break (HRW agreement across nodes, the mapping fixture, and the two properties the frozen array got for free).node --testinpackages/plugin), rebased ontomainafter feat(plugin): decide the render order from a scored ready set, not from the index #120 — none of its 48 new tests are affected, and it adds no residency read of its own.util/hash.jsalready warns about for the mixing constant. A separate test proves deduplicating a repeated node name moves no key, since HRW takes the max over the names present.record.getUpdatedTime()), sampled evenly across each node's full misplaced set, cluster in tight bursts — one node put ~96% of its share into a single minute during cluster bring-up — and every node's last misplaced write lands seconds before that node's process start.npm run format:writeandeslintclean on all touched files. Three pre-existingno-unused-varserrors inpackages/console/test/are present onmainand left alone as out of scope.Review coverage
Authored by Opus 5. No independent cross-model coverage — every outside leg failed in this environment. From the run's own log:
gemini ✗ (agy not on PATH),cursor-grok ✗ (cursor-agent --version exited -127),cursor-composer ✗ (pruned),codex ✗ (auth, rc=1). The CLI then offered a same-family Claude pass, which it labelsNOT independentand which is therefore not counted here. No per-SHA receipt was written, so the review-need field below fails closed to grade 4 — correctly.What this PR has instead is a self-review, which is not a substitute. It did find real defects after the first commit: a peerless branch allocating a fresh array on every call (the single-node case hits that path every call); a populated
server.nodesof only nameless descriptors resolving to self-only with no warning and overwriting a good list; and an overclaim in my own rationale — I had described thehdb_nodesrebuild and per-update paths as observable windows, and checking them showed both are synchronous and therefore atomic, which narrows the mechanism to the startup ordering race alone. All three are corrected here. A fourth defect, alogger?.warn?.()guard that does not protect an undeclared identifier, was caught by the existing suite rather than by me.The outside lenses this change most wants are the GitHub review bots on push, and a human on the judgment call in item 2.
Human-Review-Need: 4 @ fb377e0