feat(plugin): decide the render order from a scored ready set, not from the index

packages/plugin/README.md

@@ -518,6 +518,114 @@ The floor advances to **the first due row a pass observed**, which is the same t
 Set `queue.claimFloor.enabled: false` to roll the floor back to the old full seek. It changes nothing
 else; leases stay where they are either way.
 
+#### The ready set: which of the due rows goes first
+
+The floor decides _where the scan starts_. `queue.ready` decides _which rows get the leases_ — and it
+had to stop being a property of the scan at all.
+
+`claim` takes the first rows it finds from the floor, so the queue serves whatever is oldest-due. Two
+production measurements say that is the wrong order under scarcity
+([#80](https://github.com/HarperFast/prerender-plugin/issues/80)):
+
+- **Provenance.** During a multi-hour backlog, **239,090 of 521,929 overdue rows (~46%)** were
+  bot-discovered rather than sitemap-submitted.
+- **TTL-blindness.** Absolute due time treats a 1 h-TTL homepage 3 h overdue exactly like a 48 h-TTL
+  product page 3 h overdue — 300% stale against 6%. Simulated over the real corpus the 1 h route sits
+  at **4.78× its own TTL even at full capacity**, and 48.83× at half.
+
+**Why it cannot be fixed by re-sorting the claim window.** The window is anchored at the _oldest_ due
+time and is already an EDF prefix. Under a backlog every row in it is ancient, so a homepage two of
+its own cadences late is never read at all — and a wider window anchored in the same place is just
+more ancient rows. (Ranking by relative lateness also cannot be an _index_: `(t − dueAt) / interval`
+has slope `1/interval`, so two rows with different intervals cross exactly once and no stored key can
+express an order that changes with the clock.)
+
+So the ordering moved out of the index. A background sweep on worker 0 scores the **whole** due set
+and publishes the best few thousand into a shared buffer; `claim` pops from that in priority order and
+reads no index at all. That is affordable because of one measured fact
+([#119](https://github.com/HarperFast/prerender-plugin/pull/119)): a projected one-sided read costs
+**~2.4 µs/row, flat** from 200 to 20,000 rows, and yielding every 200 rows is free — so 200,000 rows
+cost ~480 ms and a 500k-row overdue set ~1.2 s, with **zero writes**. Writes are 76–89 µs/row, 32× a
+read, so reading liberally and writing not at all is the cheap direction.
+
+The score is `max(0, now − dueAt) / effectiveInterval`, multiplied by `queue.ready.sitemapBoost` for a
+sitemap-sourced row. Lateness rather than age, deliberately: `dueAt − interval` is not when the page
+last rendered for every row — suppression rechecks schedule 7 days, `backoffWait` up to `maxBackoff`,
+the unpin hatch a `defaultInterval` — so an age-based ratio would put a 7-day recheck on a 48 h route
+at the _head_ of the queue reading as 3.5 cadences stale.
+
+**The cadence is the row's own, not the route's** — and this is load-bearing rather than a detail.
+A route grants a _ceiling_ (`/catalog/` at 24 h) and `render.demand` promotes bot-visited targets
+beneath it to 12 h or 6 h, writing `now + rung` as the due time. Resolving the denominator from config
+would therefore divide a promoted page's lateness by 24 h when it is really on 6 h — a **4× under-
+statement, on precisely the pages the ladder singled out as worth rendering more often**, silently
+undoing the ladder's work. So every schedule writer files the cadence it used onto the row as
+`effectiveInterval` (`resolveEffectiveInterval`: rung > route > stored > default, the rung clamped to
+the ceiling so a stale one cannot read as slower), and the sweep divides by that. It rides along in a
+projection the sweep already pays for, so it costs no extra read — recovering it from `RenderTarget`
+instead would be a cross-database point read per row over the whole due set, ~75% of them replication
+fetches on a residency-pinned table.
+
+The field is **absent and self-healing** on rows written before it existed, and on the writers with no
+cadence in hand: the sweep falls back to resolving from config, which is exactly what it did before,
+and each row fills in when it next renders. `queue_health` `ready_cadence` (`carried`/`resolved`) is
+the gauge — all `resolved` on the first sweep after an upgrade, crossing over within one cadence.
+
+Three properties are worth knowing:
+
+- **It is a cache in front of the old path, not a replacement.** Cold (a fresh worker generation),
+  exhausted (claims outrunning the sweep), disabled, or a buffer that could not be sized — all of them
+  fall through to the floored index scan. Every failure mode is _the previous behaviour_, which is why
+  it ships on by default.
+- **Nothing here is a correctness invariant.** An entry naming a row that has since been rescheduled
+  or deleted costs at most one redundant render: the lease CAS refuses a duplicate and
+  `processJobResult` already drops a result whose target is gone. It cannot lose a page, because the
+  next sweep re-reads the table. Compare the claim floor, where a row filed below it is never read
+  again — silently and terminally.
+- **The sweep owns the floor now.** Once claims are served from memory they observe nothing, and a
+  floor nothing observes freezes — measured, an unfloored seek degrades **0.073 → 5.60 ms over 40,000
+  reschedules** while a floored one stays flat at 0.07 ms. So the sweep applies the same floor rule,
+  and is better informed doing it: it sees every due row, so "the first due row observed" is the true
+  minimum rather than the minimum of a window.
+
+`sitemapBoost` is a multiplier and never a tier, so it cannot starve discovered URLs: an unserved
+row's lateness grows without bound while the boost stays constant, so a discovered page is served
+within roughly `sitemapBoost ×` the worst sitemap ratio.
+
+Watch `queue_health` `claim_granted` (jobs per claim, split `ready`/`index`). The ready set reorders a
+fixed amount of work and moves no total, so this is the only series that shows whether prioritisation
+is engaging — a node quietly serving every claim from `index` looks identical to a healthy one
+everywhere else. `ready_sweep_ms` with method `capped` means the sweep never reached a not-yet-due row,
+so it is ordering over the oldest part of the backlog only and recently-due pages are going unranked;
+raise `queue.ready.sweepCap`.
+
+`queue.ready.enabled: false` claims straight from the index scan and stops the sweep — a true revert.
+
+##### What the live cluster says about the defaults
+
+Checked against the reference deployment (4 nodes, 16 workers, Harper Pro 5.2.3) before rollout:
+
+- **`claim_scan_ms` reports `method: capped` on essentially every pass** — 280 samples over six hours.
+  `capped` means the pass read its whole window and never reached a not-yet-due row. With
+  `lease_occupancy` at 75–155 the window is ~205 rows, so the queue is choosing ~4 jobs out of ~205
+  rows that are all ancient and never seeing the rest of the due set. That is the anchoring problem
+  this section describes, measured in production rather than argued from a simulation.
+- **The marginal per-row read cost is uncertain by an order of magnitude.** A synthetic benchmark says
+  ~2.4 µs/row; live, a ~205-row window takes a 5–6 ms median (p95 9–12 ms) and `empty` passes average
+  25 ms with 47 ms observed, which are seek-dominated. That is why `sweepInterval` defaults to five
+  minutes rather than one, and why `ready_sweep_ms` exists — it is the only thing that will tell you
+  the real number for your corpus.
+- **The due-set size cannot be read from the backlog snapshot.** `overdue` saturates at
+  `management.scanCap` (observed pinned at 2,000), so "how many rows will the sweep walk" is answered
+  by `ready_sweep_ms` and the sweep's own `scanned` count, not by the overview.
+- **The nodes swap** (4.6 GB in use, 5.2 GB free of 33.6 GB). This is why the sweep streams rows
+  through a bounded heap and retains only `capacity` of them: its memory is a function of the set
+  size, never of the due set. A design that sorted the due set would be actively unsafe here, and it
+  is why `capacity` should not be raised casually.
+- **The sweep cannot make a cross-node request.** It reuses the same query the claim scan uses, which
+  carries `replicateFrom: false`, and it performs no point reads at all — so it cannot take the
+  untimed replication fetch that an unowned point read on this residency-pinned table would.
+
 ## HTTP & resource API
 
 | Method & path                                | Purpose                                                             |

packages/plugin/extension.js

@@ -15,7 +15,7 @@ import {
 	seedOverrideFingerprint,
 	startOverrideWatch,
 } from './src/util/configOverride.js';
-import { startQueueStatusSync } from './src/resources/RenderQueue.js';
+import { startQueueStatusSync, startReadySweep } from './src/resources/RenderQueue.js';
 import { startSitemapRefreshScheduler } from './src/resources/Sitemap.js';
 import { startScheduleReconciler } from './src/util/reconcile.js';
 import { startUnroutedReporter } from './src/util/unrouted.js';
@@ -99,6 +99,7 @@ export async function handleApplication(scope) {
 	// self-gate by worker/node. The reconciler is deliberately NOT pinned to one node:
 	// every node repairs the schedule rows it owns (see util/reconcile.js).
 	startQueueStatusSync();
+	startReadySweep();
 	startSitemapRefreshScheduler();
 	startScheduleReconciler();
 	// Keeps the console's backlog histogram off the page-load path: the scan walks the same

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.49.0",
+	"version": "0.50.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/configSchema.js

@@ -1262,6 +1262,101 @@ export const configSchema = group('Prerender plugin configuration.', {
 				'about could not be detected.',
 			{ min: 1, scope: 'restart' }
 		),
+		ready: group(
+			'THE READY SET — which of the due rows the next leases go to, decided by a background sweep ' +
+				'instead of by the order the index happens to be in.\n\n' +
+				'WHY: `claim` takes the first rows it finds from the claim floor, so the queue serves ' +
+				'whatever is oldest-due. Two production measurements say that is the wrong order under ' +
+				'scarcity (prerender-plugin#80): ~46% of a 521,929-row overdue queue was bot-discovered ' +
+				'rather than sitemap-submitted, and absolute due time treats a 1h-TTL homepage 3h overdue ' +
+				'exactly like a 48h-TTL product page 3h overdue — 300% stale against 6%. Simulated over ' +
+				'the real corpus the 1h route sits at 4.78x its own TTL even at FULL capacity.\n\n' +
+				'It could not be fixed by re-sorting the claim window, because the window is ANCHORED AT ' +
+				'THE OLDEST DUE TIME: under a backlog every row in it is ancient, so the homepage is never ' +
+				'read at all and a wider window is just more ancient rows. So a sweep scores the WHOLE due ' +
+				'set and keeps the best few thousand in shared memory; claims pop from that and touch no ' +
+				'index. Affordable because a projected one-sided read measures ~2.4us/row, flat — 200,000 ' +
+				'rows in ~480ms, with zero writes.\n\n' +
+				'ORDERING ONLY. Total render volume cannot change: every row it reorders is already due. ' +
+				'And it is a CACHE in front of the old path — cold, exhausted or disabled, claims fall back ' +
+				'to the index scan, so every failure mode here is the previous behaviour rather than a ' +
+				'stalled queue.',
+			{
+				enabled: option(
+					true,
+					'Kill switch. `false` claims straight from the index scan, exactly as before v0.50.0. The ' +
+						'sweep also stops, so nothing is spent maintaining a set nothing reads.'
+				),
+				capacity: option(
+					5000,
+					'Entries the ready set holds. Sized to cover several sweep intervals of claims so the set ' +
+						'does not run dry between sweeps: at the recorded fleet throughput a node grants roughly ' +
+						'5 jobs a second (observed live: ~70-75 claims a minute), so 5,000 entries is about 16 ' +
+						'minutes of work — three sweep intervals at the default.\n\n' +
+						'DO NOT RAISE THIS CASUALLY. The reference cluster runs with 4.6GB of swap in use and 5.2GB ' +
+						'free of 33.6GB, so shared memory on these nodes is not free. A larger set does not improve ' +
+						'the ordering either — the sweep already scores every due row and keeps the best of them, so ' +
+						'this only buys time between sweeps. What makes the sweep safe on a swapping node is that ' +
+						'its own memory is a function of THIS number and not of the due set: it streams rows through ' +
+						'a bounded heap and retains only the best `capacity`.\n\n' +
+						'Costs `capacity x ~276 x 2` bytes of shared memory — two slots, so ~2.8MB at the default. ' +
+						'Raising it does ' +
+						'NOT make the ordering better — the sweep already scores every due row and keeps the best ' +
+						'of them — it only makes the set last longer between sweeps.\n\n' +
+						'Restart-scoped: a named shared buffer is sized by its first allocation, so a live change ' +
+						'would give workers in one generation differently-sized views of the same buffer. A ' +
+						'mismatch is logged and the smaller size honoured.',
+					{ min: 0, scope: 'restart' }
+				),
+				sweepInterval: option(
+					5 * MINUTE,
+					'How often worker 0 re-scores the due set and republishes.\n\n' +
+						'This is the ORDERING STALENESS: a row that becomes due just after a sweep waits up to one ' +
+						'interval before it can be ranked. Five minutes against cadences of an hour and up is a ' +
+						'rounding error, and `capacity` covers roughly three of these intervals of claims, so the ' +
+						'set does not run dry between sweeps.\n\n' +
+						'FIVE MINUTES RATHER THAN ONE, on production evidence. A synthetic benchmark puts a ' +
+						'projected one-sided read at ~2.4us/row, which would make a sweep sub-second — but the live ' +
+						'cluster reports `claim_scan_ms` at a 5-6ms median over a window of roughly 205 rows ' +
+						'(grantLimit + in-flight + grantLimit, at an observed lease occupancy of 75-155), and ' +
+						'`empty` passes at a 25ms mean with 47ms observed, which are seek-dominated. So the real ' +
+						'marginal per-row cost sits somewhere between 2.4us and ~25us — an order of magnitude of ' +
+						'uncertainty — and the sweep shares a worker with bot traffic. At the wide end a ' +
+						'one-minute interval would spend a noticeable fraction of a core continuously, for no ' +
+						'benefit: the ordering does not go stale that fast.\n\n' +
+						'WATCH `ready_sweep_ms` AND TIGHTEN FROM THERE. It reports the real number for your corpus, ' +
+						'which is the only way to know it — the backlog snapshot cannot tell you the due-set size ' +
+						'either, because `overdue` saturates at `management.scanCap` (observed pinned at 2,000).\n\n' +
+						'`0` disables the sweep, which leaves the set to go stale and then empty; claims fall back ' +
+						'to the index scan as they always do. The ceiling is node\u2019s own timer limit of 2^31-1 ms ' +
+						'(~24.8 days) \u2014 past it a timer fires every millisecond rather than never, which would ' +
+						'turn the sweep into a hot loop over the due set.',
+					{ unit: 'ms', min: 0, max: 2147483647 }
+				),
+				sweepCap: option(
+					500_000,
+					'Ceiling on rows one sweep reads. The due set cannot exceed the corpus, so this is a ' +
+						'guard against a runaway rather than a tuning knob — at ~2.4us/row the default is ~1.2s of ' +
+						'reading.\n\n' +
+						'If a sweep hits the cap WITHOUT reaching a not-yet-due row it is ordering over a prefix ' +
+						'of the backlog, which is reported and warned about: the rows past the cap are the ' +
+						'youngest, so the effect is that recently-due pages go unranked — exactly the pages this ' +
+						'exists to protect.',
+					{ min: 1 }
+				),
+				sitemapBoost: option(
+					2,
+					'How much a sitemap-sourced row outranks a discovered one at the same overdue ratio. `1` ' +
+						'disables the preference and orders on overdue ratio alone.\n\n' +
+						'A MULTIPLIER, not a tier, so it cannot starve discovered URLs: an unserved row\u2019s ' +
+						'lateness grows without bound while the boost stays constant, so a discovered row wins as ' +
+						'soon as its ratio passes `sitemapBoost x` the highest sitemap ratio in the set. With ' +
+						'sitemap pages held ~1.2 cadences late, a discovered page is served within ~2.4 cadences ' +
+						'of its own interval at the default.',
+					{ min: 1 }
+				),
+			}
+		),
 		claimScanCap: option(
 			1000,
 			'Ceiling on schedule rows read per claim pass. A leased row keeps its overdue position in the ' +

packages/plugin/src/http_handlers/bot_request.js

@@ -6,7 +6,13 @@ import { canonicalizeUrl } from '../util/url.js';
 import { config } from '../config.js';
 import { sanitizeDeviceType } from '../util/device_type.js';
 import { resolveForwardedRequest } from '../util/ingress.js';
-import { classifyPath, isForwardedMode, routeScopeForEntry, PRERENDER } from '../util/routeClass.js';
+import {
+	classifyPath,
+	isForwardedMode,
+	resolveEffectiveInterval,
+	routeScopeForEntry,
+	PRERENDER,
+} from '../util/routeClass.js';
 import { Target } from '../resources/Target.js';
 import { QueueState } from '../resources/QueueState.js';
 import { fetchOriginResource } from '../util/upstream.js';
@@ -371,7 +377,7 @@ async function renderNow({ url, cacheUrl, deviceType, cacheKey, request, routeSc
 	// The target is the field's source of truth (`RenderQueue` re-derives it from there on every
 	// reschedule for exactly that reason). NO target is the legitimate render-now one-off shape, where
 	// `false` is the true answer.
-	const renderTarget = await Target.get({ id: cacheUrl, select: ['sitemapUrl'] });
+	const renderTarget = await Target.get({ id: cacheUrl, select: ['sitemapUrl', 'renderInterval', 'demandInterval'] });
 
 	// Force an immediately-claimable, one-off schedule. No Target is created, so
 	// processJobResult won't reschedule it — and drops the schedule row once the result
@@ -383,7 +389,17 @@ async function renderNow({ url, cacheUrl, deviceType, cacheKey, request, routeSc
 	// would strand: on this node the funnel lowers the floor in-process, and on any other node —
 	// which is ~75% of keys, since schedule rows are residency-pinned — the guard band is what
 	// keeps the row above the owner's floor and therefore claimable.
-	await writeSchedule(cacheKey, { nextRenderTime: currentMinuteMs(), fromSitemap: !!renderTarget?.sitemapUrl });
+	await writeSchedule(cacheKey, {
+		nextRenderTime: currentMinuteMs(),
+		fromSitemap: !!renderTarget?.sitemapUrl,
+		// PRESERVED WHEN THERE IS A TARGET, `null` WHEN THERE IS NOT — and the difference matters because
+		// `put` replaces the record. This key may be a real target's recurring row (a warm-on-demand
+		// render-now), and filing `null` there would strip its cadence and demote the page in the next
+		// sweep. With no target this is the render-now one-off shape, which has no cadence to record: the
+		// row is dropped once the result lands. Either way it is due at the current minute, so its own
+		// ranking is unaffected — this is about not damaging the row on the way past.
+		effectiveInterval: renderTarget ? resolveEffectiveInterval(cacheUrl, renderTarget) : null,
+	});
 
 	// Wake idle consumers now instead of waiting out the periodic status sync. Non-force
 	// so a paused queue stays paused (the render then simply times out to the fallback).