Orphaned nextRenderTime index entries make the claim scan non-monotonic; one break at row 0 silently stalls a node's entire backlog (2 of 4 nodes affected)

[harper-joseph]

Summary

A claim pass that sees no due rows advances the floor to nowMinute - guard. That is correct
on a caught-up queue, but it is an unconditional forward jump with no way back: any row that later
appears below the floor is invisible to every subsequent pass, because the scan is a single
nextRenderTime >= floor condition. Nothing in the claim path can lower the floor again.

Observed in production as one node of four silently ceasing to render a growing slice of its
shard: 92,873 rows stranded below the floor across 2,101 distinct due-minutes, oldest due
2026-08-01 — 18 days.

#111 is the companion: writeSchedule lowers only the writing node's floor while the row is
residency-routed to its owner, so ~75% of writes leave the owner's floor untouched. That is what
keeps depositing rows underneath a floor that has already moved past them.

Root cause — ORPHANED SECONDARY INDEX ENTRIES (verified on two nodes)

The nextRenderTime index contains entries that point at rows whose stored value has since moved
forward. The record is correct; the index entry was never removed from its old position. The walk
visits the orphan, loads the record, and yields a future value at the head of an otherwise
ascending scan.

runClaimPass breaks on the first future row it sees:

if (at > nowMs) { earliestNotYetDueMinute = minuteOf(at); break; }

So it breaks at iteration 0, every pass, forever. firstDueMinute stays null, sawDue is
false, and:

const floorTo = Math.max(0, observed ?? nowMinute - guard);
floorAdvanced = leases.advanceFloor(floorFrom, floorTo);   // unconditional CAS, no next > expected check

the floor ratchets to nowMinute - guard, abandoning everything below it.

Measured, live

Scanning each node's index from 0, limit: 200, sorted ascending:

node	monotonic violations	head of walk	future rows in window	overdue in window
cd5	1	2026-08-20T11:24	2	198 / 200
yc0	1	2026-08-29T23:03	1	199 / 200
e9v	0	2026-08-19T16:01	158	42
v3t	0	2026-08-19T16:01	126	74

Two of four nodes are affected, independently. cd5 has been stalled ~18 days with ~92,873 rows
stranded; yc0 is in the early stage of the same slide (507 overdue and climbing).

The orphans, with the position bisected out of the index:

cd5  <product-A>|mobile      record 2026-08-20T11:24  indexed 2026-08-14T02:06  fromSitemap=true
cd5  <product-B>|desktop record 2026-08-20T11:24  indexed 2026-08-14T02:06  fromSitemap=true
yc0  <product-C>|desktop record 2026-08-29T23:03 indexed 2026-08-15T23:02 fromSitemap=false

Point reads confirm the records hold the later value. A walk starting at that later value does
not return these keys — so there is no index entry at the correct position, only the orphan. This
is a missing index update on rewrite, not a duplicate.

Three orphaned entries are withholding ~93k rows of work.

Why every recovery fails

• reset-claim-floor is futile. Confirmed live: reset, and the floor was back at
  now - guard immediately. From a zeroed floor the walk returns the same orphan first and the
  pass breaks at iteration 0 again.
• claimFloor.resetInterval is the same reset, so equally futile.
• claimFloor.enabled: false is worse than futile (see below).
• unpinAfter is irrelevant — the pass never gets far enough to name a pinned row.

Observed live: the floor sat at 15:38 for five minutes, then stepped to 15:43 — tracking
now - guard in guard-sized hops while ~92.9k rows sat below.

Current-minute work still renders (a pass whose window happens to open on a due row reports
sawDue: true), which is why the node looks healthy and QueueStatus reads queued.

Upstream: this is Harper core, and it is already filed

The index-maintenance defect belongs to harper (resources/Table.ts), not to this plugin,
and it is already open as HarperFast/harper#2211 — "Secondary index permanently diverges from
the primary store, and rewriting the record does not repair it (5.2.1 / rocksdb-js 2.7.1)".

That issue reports the missing-entry signature (an indexed read returns a strict subset) and
hypothesises, without source confirmation, that "a put of a record whose indexed attribute value
is unchanged appears not to (re)write the index entry". The updateIndices early-continue quoted
below is that hypothesis confirmed in code.

This cluster shows a second signature of what is most likely the same defect: the old entry
retained and the new entry absent. Same function, same consequence — an index that no
longer describes the record and does not self-heal.

(harper#1894, cited in the updateIndices comments, is a CLOSED, separate TTL/eviction
orphaning bug — related code, not this defect.)

Origin of the orphans — confirmed possible in Harper's index maintenance

core/resources/Table.ts → updateIndices(id, existingRecord, record, options) is the single place
secondary index entries are maintained. Two properties of it make an orphan reachable:

const value         = record == null ? undefined : (resolver ? resolver(record) : record[key]);
const existingValue = existingRecord && (resolver ? resolver(existingRecord) : existingRecord[key]);
if (value === existingValue && !isIndexing) continue;          // (A) no index work at all
...
let valuesToRemove = getIndexedValues(existingValue, indexNulls);
if (valuesToRemove?.length > 0) { ... index.remove(...) }      // (B) removal ONLY from existingRecord
else if (...) { /* "no old values, just new" — treated as a fresh insert */ }

• (A) If existingValue already equals the incoming value, the function returns without
  touching the index. Any index entry that does not actually correspond to the record's current
  value is then left in place, and no entry is written at the new position.
• (B) The old entry is removed only from what existingRecord says. If existingRecord is
  null/undefined for a record that did have an indexed value, valuesToRemove is empty, the
  old key is never removed, and the write is treated as a fresh insert.

existingRecord is priorStaged ? priorStaged.value : existingEntry?.value — so it is whatever
the write path read, not a re-read under the index lock.

The observed production state — an entry at the old value, and no entry at the new value
(a walk starting at the record's own nextRenderTime did not return the key) — matches shape
(A): the index was not updated at all, in either direction, while the record moved forward.

This function already carries a comment citing harper#1894 (F-149) for a previous orphaning
bug in the same code, so orphaning here has precedent.

The aborted transactions are NOT the cause

Two independent reasons, both from source:

1. Record and index writes share one transaction. The call site is
   updateIndices(id, existingRecord, recordToStore, transaction && { transaction }), and that
   same transaction is passed down into index.remove(...)/add. So a crash, an abort, or a
   thread death rolls back both — it cannot half-apply a record without its index entry. This
   also argues against a restart of any kind being the origin.
2. Different databases. The aborts name only PrerenderedPage/ and VisitFilter/
   (page_cache, render_service); the orphans are in render_schedule. processJobResult
   writes the page and the schedule as sequential awaits, and per the plugin's own note tables in
   separate databases are "independent commits rather than one atomic write". Harper also
   serializes writes per database.

The abort volume (~800–1,100/hour, sustained, on PrerenderedPage) is a real problem and deserves
its own issue — it is simply not this one.

What is NOT established

The precise trigger that produces the (A) state. Dating the divergent writes from
record.nextRenderTime - interval is unreliable — the demand ladder rewrites intervals, so the
interval at write time is not knowable after the fact. The affected rows have since been repaired
and re-rendered, so the original values are no longer recoverable. Frequency is roughly 3 rows in
~1.3M over several days.

Two defects

1. Index maintenance drops the old entry on some rewrite path. Whose bug this is — plugin
   write shape vs. Harper/RocksDB secondary-index maintenance — is the open question. Only 3 rows
   out of ~650k per node are affected, so it is a rare path, not systemic corruption. Both cd5
   rows share a due minute (08-14T02:06) but are unrelated products; the yc0 row is a different
   minute and differs in fromSitemap.
2. runClaimPass trusts the ordering absolutely. One break on the first future row turns a
   single bad index entry into a permanent, silent, whole-node queue stall. Draining the window
   and filtering — rather than breaking — would have contained this to "a few extra rows scanned".

Defect 2 is the one worth fixing first: it is cheap, local to this plugin, and it makes the claim
path robust to any future ordering violation regardless of who caused defect 1.

Production evidence

Plugin 0.49.0, harper-pro 5.2.3, 4-node cluster, ~1.6M keys, all queue.* at defaults.

rows below floor        92,873      <- invisible to claim
rows floor..now              2      <- all claim can see
distinct due-minutes     2,101
oldest stranded row      2026-08-01T00:57Z

Other three nodes: 507 / 3 / 1 overdue. This bites whichever node's floor gets ahead, not the
cluster uniformly.

Claim was granting nothing — granted 0 of 5 in 256 of 270 warned passes — and the reported scan
cap tracked in-flight lease count exactly across samples (90/90, 100/100, 173/174), consistent
with scanLimit = grantLimit + occupancy + grantLimit.

Ruled out

Recording these so nobody re-walks them:

• Tie-pileup at one minute. The stranded set spans 2,101 distinct minutes at ~44 rows/minute
  average, not one saturated minute.
• A single poison row wedging the head. The head of the index is ordinary product URLs, not
  the deep-facet catalog.jsp?CN=… URLs that appear in the floor-holder warnings.
• unpinAfter not firing. Pin age lives in the SharedBuffer and resets on restart; node
  uptime was under the 1h default, so zero unpin warnings is expected here. It is also explicitly
  "a fix for index degradation, not a way to keep throughput up" — one row per interval per node
  would not touch 92.9k rows.
• claimFloor.enabled: false as a mitigation. It is counterproductive, see below.

claimFloor.enabled: false makes this worse, and should probably say so

Tried as the documented kill switch. Over 75s it recovered zero stranded rows, and the overdue
count accelerated from ~7/min drift to ~200/min. Two reasons:

1. maybeUnpinFloor early-returns when the floor is disabled:

   if (!(unpinAfter > 0)) return null;
   if (!config.queue.claimFloor.enabled) return null;   // kill switch disables the escape hatch too

2. With the floor off, runClaimPass calls leases.resetFloor() every pass, so each pass
   restarts from the absolute index minimum while still capped at
   grantLimit + occupancy + grantLimit (~100 rows).

The option text presents this as a safe "changes nothing else" kill switch. For a node already
stranded it is a downgrade.

Observability: the detector works, but the gauge is capped into looking benign

backlogSnapshot runs and reports this correctly — it is not broken, and the worker-0 gate is
fine (an earlier revision of this issue speculated otherwise; disproved by the snapshot's own
persisted lastRun, which is worker-0-gated and has fresh, error-free entries on every node).

The problem is the value. On the affected node:

belowFloor         = 1998                  <- real figure is ~92,873
oldestBelowFloorMs = 2026-08-14T02:07Z
floorMs            = 2026-08-19T14:39Z
scanned = 2000, cap = 2000, truncated = true

belowFloor is counted during a management.scanCap-bounded walk, so it saturates at the cap and
is emitted that way to metrics.queueHealth(stats.belowFloor, 'below_floor'). A node with 92,873
stranded rows and a node with 1,998 publish the same number. The gauge that exists precisely to
make this condition visible is the one that hides its magnitude.

The option text for management.scanCap states the opposite intent — that the below-floor
detector "is found in the FIRST rows of the ascending scan and is unaffected by the cap". That
holds for detection but not for the count, and it is the count that gets alarmed on.

One unexplained detail worth a look: oldestBelowFloorMs reads 2026-08-14T02:07 while the table's
true minimum nextRenderTime is 2026-08-01T00:57. If the walk really is ascending from the
absolute minimum, those should agree.

Asks

1. Bound the forward advance. A pass should only advance the floor as far as it actually
   scanned — e.g. to earliestNotYetDueMinute when it reached a not-yet-due row, and not at all
   when the scan was cap-truncated. observed ?? nowMinute - guard is the defect.
2. Reconsider advanceFloor's contract. It is named "advance" but is an unconditional CAS
   swap; a next > expected assertion would have made this a no-op instead of a ratchet.
3. Document/repair the kill switch. Either stop claimFloor.enabled: false from disabling
   maybeUnpinFloor, or say plainly in the option text that it does.
4. Make below_floor unbounded, or publish it as a saturating flag. It is currently counted
   inside a management.scanCap-bounded walk, so it maxes out at the cap and under-reports by
   ~46x on the affected node. Either count it without the cap, or emit an explicit
   below_floor_truncated alongside it so an alarm can tell "1,998" from "at least 2,000 and we
   stopped counting".
5. Per-node ConfigOverride scoping. This condition was node-local, but every lever for it is
   cluster-wide: ConfigOverride.path is the sole primary key and the table is deliberately
   replicated and not residency-pinned, so rescuing one node changes claim behaviour on three
   healthy ones. The only node-scoped alternative is hand-editing the deployed config.yaml,
   which the next harper deploy silently reverts. QueueControl already implements the needed
   pattern — scope as primary key, 'all' plus per-hostname rows, delete to inherit. A scope
   column on ConfigOverride (compound key with path) would follow existing precedent in the
   same plugin.