SUPERSEDED by #116: rank the due set by relative lateness (refuted approach)

packages/plugin/README.md

@@ -129,6 +129,10 @@ rest: true # required for the @export-ed table REST endpoints
     statusSyncInterval: 60000 # 1m  — pause convergence, status broadcast, claim-floor reset
     maxLeases: 4096 # lease slots in the node-local shared buffer (restart-scoped)
     claimScanCap: 1000 # ceiling on schedule rows read per claim pass
+    priority: # WHICH of the due rows the next leases go to (ordering only, no cadence change)
+      enabled: true # false = grant in index order (absolute due time), as before v0.50.0
+      sitemapBoost: 2 # how much a sitemap row outranks a discovered one at equal overdue ratio
+      candidatePool: 8 # multiples of `limit` to choose from; 1 keeps the pre-0.50.0 window
     claimFloor: # the lower bound the claim scan seeks from (see "The claim floor")
       enabled: true # false = seek the absolute index minimum, as before v0.34.0
       guard: 300000 # 5m — the floor is always held at least this far behind now
@@ -518,6 +522,63 @@ The floor advances to **the first due row a pass observed**, which is the same t
 Set `queue.claimFloor.enabled: false` to roll the floor back to the old full seek. It changes nothing
 else; leases stay where they are either way.
 
+#### Which of the due rows goes first
+
+The floor decides _where the scan starts_. `queue.priority` decides which of the rows it drained get
+the leases — and only that. It changes no cadence, creates no work, and cannot move total render
+volume, because every row it reorders is already due.
+
+Absolute due time cannot express this, which is the whole reason the option exists:
+
+| page | cadence | due    | overdue, in its own cadence |
+| ---- | ------- | ------ | --------------------------- |
+| home | 1h      | 2h ago | **2.0 intervals**           |
+| PDP  | 48h     | 3h ago | 0.06 intervals              |
+
+Index order hands the lease to the PDP, because 3h > 2h. Nothing looks wrong while it does: the floor
+advances, the scan stays fast, no row is wedged. The only symptom is the one the served-age numbers
+already show — worst-case age is `interval + swrTtl`, which is several multiples of a fast route's
+cadence and a fraction of a slow one's — and it is easy to spend that incident tuning
+`renderInterval`.
+
+So a due row is ranked by `(now − dueAt) / renderInterval`, i.e. how late it is **relative to its own
+cadence**, with sitemap-sourced rows multiplied by `queue.priority.sitemapBoost`. Three details are
+load-bearing:
+
+- **Lateness, not age.** `dueAt − interval` is not when the page last rendered: `Target.suppress`
+  schedules `render.suppression.recheckInterval` (7 days), `backoffWait` schedules up to
+  `maxBackoff`, and the unpin hatch pushes by `render.defaultInterval`. An age-based ratio would put
+  a 7-day suppression recheck on a 48h route at the _head_ of the queue reading as 3.5 cadences
+  stale. Lateness is zero the moment any row comes due, whatever gap preceded it, so those rows enter
+  at the back and climb like anything else.
+- **The cadence is read off the row.** `renderInterval` is denormalized onto `RenderSchedule` for the
+  same reason `fromSitemap` is: `claim` takes no `RenderTarget` read. It matters because the
+  effective cadence includes the demand ladder's rung, and resolving the route at claim time would
+  rank a promoted catalog page at its 24h _ceiling_ — the opposite of what promoting it was for. The
+  field is optional; a row written by a path that does not have it (a reconcile repair, an
+  invalidation re-enqueue, a render-now one-off) falls back to the route-resolved interval until that
+  URL's next completed render re-stamps it.
+- **The boost is a multiplier, never a lane.** An unserved row's ratio grows without bound while the
+  boost stays constant, so a discovered URL wins as soon as its ratio passes `sitemapBoost ×` the
+  highest sitemap ratio in the window. With sitemap pages held ~1.2 cadences late that is ~2.4
+  cadences at the default — bounded, and it scales with the boost.
+
+`queue.priority.candidatePool` is the part to actually think about. The scan window exists to read
+_past_ the in-flight lease pile (`limit` + pile + `limit`), so beyond the pile it holds about as many
+grantable rows as the pass is about to hand out — "pick the best 25" out of 25. `candidatePool`
+widens only that last term, to `limit × candidatePool` rows past the pile, still hard-capped by
+`queue.claimScanCap`. The pile is counted first, so a large pile can consume the cap and leave the
+pool no room: if the truncation warning starts naming the cap, raise `claimScanCap` before raising
+`candidatePool`.
+
+Watch `queue_health` `claim_lateness_pct` — how overdue each _granted_ job was as a percentage of its
+own interval, split sitemap/discovered. It is the normalized companion to `route_page_age`: one p95
+covers every route, so a p95 well above 100 across the board reads as a capacity shortfall rather
+than something to infer by dividing two dashboards. No ordering fixes that one.
+
+`queue.priority.enabled: false` grants in index order and walks the old, narrower window — a revert
+of the behaviour, not a neutral weighting of it.
+
 ## HTTP & resource API
 
 | Method & path                                | Purpose                                                             |

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.49.0",
+	"version": "0.50.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/configSchema.js

@@ -1262,6 +1262,63 @@ export const configSchema = group('Prerender plugin configuration.', {
 				'about could not be detected.',
 			{ min: 1, scope: 'restart' }
 		),
+		priority: group(
+			'WHICH of the due rows the next leases go to. Ordering only — this changes no cadence, ' +
+				'creates no work and cannot change total render volume, because it reorders rows that are ' +
+				'ALREADY due inside a single claim pass.\n\n' +
+				'The queue seeks from the claim floor along one index (`nextRenderTime`) and that stays ' +
+				'true. What changes is the comparison used to pick from the rows it drained: instead of ' +
+				'absolute due time, a row is ranked by how overdue it is RELATIVE TO ITS OWN cadence — ' +
+				'`(now - dueAt) / renderInterval` — with sitemap-sourced rows multiplied by ' +
+				'`sitemapBoost`. Absolute due time cannot express this: a 48h PDP due 3h ago outranks a ' +
+				'1h homepage due 2h ago even though the homepage is 2 cadences late and the PDP is 6% ' +
+				'late, so under any backlog the short-cadence routes sit behind a wall of long-cadence ' +
+				'ones and the only symptom is a served age of `interval + swrTtl` that is many multiples ' +
+				'of a fast route’s interval and a fraction of a slow one’s.\n\n' +
+				'IT REORDERS A WINDOW, NOT THE BACKLOG. The pass drains ' +
+				'`min(grantLimit + in-flight + grantLimit, queue.claimScanCap)` rows, so it chooses ~25 ' +
+				'out of a few hundred, not out of the whole due set. It is a latency fix, not a capacity ' +
+				'fix: if a route is short of capacity outright, `renderInterval` and fleet size are the ' +
+				'levers. Raise `queue.claimScanCap` to widen the pool it chooses from.',
+			{
+				enabled: option(
+					true,
+					'Kill switch. `false` grants in index order (absolute due time), exactly as before ' +
+						'v0.50.0 — the claim pass then walks and stops at `grantLimit` as it used to, so this ' +
+						'is a true revert of the behaviour and not a neutral weighting of it.'
+				),
+				sitemapBoost: option(
+					2,
+					'How much a sitemap-sourced row outranks a discovered one at the same overdue ratio. ' +
+						'`1` disables the preference and leaves ordering on overdue ratio alone.\n\n' +
+						'A MULTIPLIER, not a tier, so it cannot starve discovered URLs: an unserved row’s ' +
+						'overdue ratio grows without bound while the boost stays constant, so a discovered row ' +
+						'wins as soon as its ratio passes `sitemapBoost x` the highest sitemap ratio in the ' +
+						'window. With sitemap pages held ~1.2 cadences late, a discovered page is therefore ' +
+						'served within ~`2 x 1.2` cadences of its own interval at the default. Raising this ' +
+						'raises that bound proportionally.',
+					{ min: 1 }
+				),
+				candidatePool: option(
+					8,
+					'How many times `limit` grantable rows a claim pass tries to choose from, as a multiple ' +
+						'of the batch it is granting.\n\n' +
+						'WITHOUT THIS THE ORDERING HAS ALMOST NOTHING TO ORDER. The scan window exists to read ' +
+						'PAST the in-flight lease pile — `limit` + pile + `limit` — so beyond the pile it holds ' +
+						'about as many grantable rows as the pass is about to hand out, and "pick the best 25" ' +
+						'degenerates to "take the 25 that were there". This widens only that last term, so the ' +
+						'pass reads `limit x candidatePool` rows past the pile and grants the most overdue ' +
+						'`limit` of them.\n\n' +
+						'COST: index-ordered reads inside a window the seek has already landed in, and still ' +
+						'hard-capped by `queue.claimScanCap` — which is the ceiling to watch, because the pile ' +
+						'is counted first, so a large pile can consume the cap and leave the pool no room. If ' +
+						'the truncation warning starts naming the cap, raise `claimScanCap` before raising ' +
+						'this. `1` keeps the historical window (and makes the ordering close to a no-op); `0` ' +
+						'is treated as `1`.',
+					{ min: 0 }
+				),
+			}
+		),
 		claimScanCap: option(
 			1000,
 			'Ceiling on schedule rows read per claim pass. A leased row keeps its overdue position in the ' +

packages/plugin/src/metrics.js

@@ -377,15 +377,23 @@ export const METRICS = Object.freeze({
 					'paused = 1 when this node’s queue is paused at snapshot time, else 0 — makes "paused for hours" ' +
 					'alertable without polling the REST surface. ' +
 					'claim_scan_ms = claim-pass duration; watch the p95 trend, not the level. ' +
+					'claim_lateness_pct = how overdue each GRANTED job was as a percentage of ITS OWN render ' +
+					'interval (100 = one cadence late), split sitemap/discovered in the method slot — the ' +
+					'normalized companion to route_page_age, and the measure of whether queue.priority is ' +
+					'keeping short-cadence routes on time. A p95 that sits well above 100 for every route at ' +
+					'once is a capacity shortfall, not an ordering problem, and no ordering fixes it. Emitted ' +
+					'only while queue.priority.enabled is on, because the ratio is a by-product of the ' +
+					'ordering pass. ' +
 					'reconcile_restored / reconcile_missing = schedule gaps repaired / found per sweep (they differ ' +
 					'when the per-sweep restore cap truncates the pass); expect zero — a steady rate means ' +
 					'something is CREATING gaps, and the reconcile log line names the URLs.',
 			},
 			method: {
-				name: 'result (claim_scan_ms only)',
-				values: ['granted', 'empty', 'capped'],
+				name: 'result (claim_scan_ms) | source (claim_lateness_pct)',
+				values: ['granted', 'empty', 'capped', 'sitemap', 'discovered'],
 				description:
-					'Only claim_scan_ms uses this slot: granted = jobs handed out, empty = nothing due, capped = the ' +
+					'claim_lateness_pct uses this slot for sitemap | discovered. On claim_scan_ms: ' +
+					'granted = jobs handed out, empty = nothing due, capped = the ' +
 					'scan hit queue.claimScanCap without reaching a not-yet-due row (in-flight work is filling the ' +
 					'window). Every other series emits null here.',
 			},
@@ -641,6 +649,20 @@ export const metrics = Object.freeze({
 	/** One claim pass's duration and how it ended — a queue_health series, so the queue reads in one scan. */
 	claimScan: (durationMs, result) => server.recordAnalytics(durationMs, 'queue_health', 'claim_scan_ms', result, null),
 
+	/**
+	 * How overdue one GRANTED job was, in hundredths of its own render interval, split by whether the
+	 * URL is sitemap-listed. 100 = exactly one cadence late.
+	 *
+	 * This is the outcome measure for `queue.priority`, and it is normalized on purpose: `page_age`
+	 * and `route_page_age` are absolute, so a 48h route and a 1h route are not comparable in them and
+	 * a regression on the fast route hides inside the slow route's numbers. Here every route shares
+	 * one scale — its own cadence — so one p95 covers the whole corpus, and "the fleet cannot keep up
+	 * with what has been asked of it" is a level (well above 100) rather than something to be
+	 * inferred by dividing two dashboards.
+	 */
+	claimPriority: (ratioPct, source) =>
+		server.recordAnalytics(ratioPct, 'queue_health', 'claim_lateness_pct', source, null),
+
 	/** One origin proxy on the serve path: time to response headers, status, and why. */
 	originFetch: (durationMs, statusCode, reason) =>
 		server.recordAnalytics(durationMs, 'origin_fetch', statusCode, reason, null),

packages/plugin/src/resources/RenderQueue.js

@@ -384,7 +384,16 @@ export class RenderQueue extends Resource {
 				// path costs one atomic load and moves the floor not at all. That is load-bearing: a
 				// lowering on every completed render would rewind the floor to the current minute
 				// continuously and the whole 14× seek win would evaporate.
-				await writeSchedule(cacheKey, { nextRenderTime, fromSitemap: !!renderTarget.sitemapUrl });
+				// `interval` — the DEMAND-RESOLVED cadence, not `base` — is what the claim pass scores this
+				// row's lateness against (util/renderPriority.js). This is the only writer that knows the
+				// ladder's answer, so it is the one that has to record it: `resolveRenderInterval` at claim
+				// time sees the route's ceiling and would rank a promoted catalog page as if it were still
+				// on 24h, which is the opposite of what promoting it was for.
+				await writeSchedule(cacheKey, {
+					nextRenderTime,
+					fromSitemap: !!renderTarget.sitemapUrl,
+					renderInterval: interval,
+				});
 
 				// Persist the rung ONLY on an actual move. 'held' must not write even when the
 				// stored field is absent — absence already resolves to the base ceiling, so writing
@@ -724,7 +733,12 @@ export class RenderQueue extends Resource {
 			`Retrying ${cacheKey} in ${Math.round(wait / 60000)}m (failure strike ${strikes}` +
 				`${fromSitemap ? '' : ', non-sitemap'})`
 		);
-		await writeSchedule(cacheKey, { nextRenderTime, fromSitemap });
+		// THE CADENCE, NOT THE BACKOFF. `wait` is how long this failing key is being pushed out for;
+		// `interval` is still what it is supposed to render at, and it is the cadence the claim pass
+		// measures lateness against. Recording `wait` instead would give a backed-off row an interval
+		// several times its real one, hence a near-zero overdue ratio, hence a permanent seat at the
+		// back of the queue — a deprioritization the backoff already applied once, compounding.
+		await writeSchedule(cacheKey, { nextRenderTime, fromSitemap, renderInterval: interval });
 		return 'slow';
 	}
 
@@ -754,6 +768,7 @@ export class RenderQueue extends Resource {
 		await writeSchedule(cacheKey, {
 			nextRenderTime: currentMinuteMs() + interval,
 			fromSitemap: !!renderTarget.sitemapUrl,
+			renderInterval: interval,
 		});
 	}
 
@@ -814,6 +829,13 @@ export class RenderQueue extends Resource {
 			// number proves it happens.
 			if (getResidencyByUrl(url) !== server.hostname) notOwnedHere++;
 
+			// One emit per granted job — ~25 per pass on a path that runs a fraction of a time per
+			// second, so it stays off any hot loop. `undefined` when the pass granted in index order and
+			// therefore never scored anything; reporting a 0 there would read as "everything is on time".
+			if (granted.priority !== undefined) {
+				metrics.claimPriority(Math.round(granted.priority * 100), granted.fromSitemap ? 'sitemap' : 'discovered');
+			}
+
 			jobs.push({
 				id: granted.cacheKey,
 				url,

packages/plugin/src/resources/Target.js

@@ -108,6 +108,12 @@ export class Target extends TargetTable {
 						? nextRenderTime
 						: getInitialRenderTime(cacheKey, interval),
 				fromSitemap,
+				// The cadence the claim pass ranks this row's lateness against. Route > stored > default
+				// here, which is one term better than the fallback `claim` can compute on its own (it has
+				// no Target read, so it never sees a stored sitemap `changefreq` interval). The demand
+				// ladder's rung is not known yet — a target has none until it has rendered — so the first
+				// cycle scores at the ceiling and the post-render write corrects it.
+				renderInterval: interval,
 			}))
 		);
 

packages/plugin/src/schemas/schema.graphql

@@ -114,6 +114,19 @@ type RenderSchedule @table(database: "render_schedule") @export {
 	# cross-database read just to flag sitemap-sourced jobs. Refreshed on every
 	# reschedule (job result), so it self-corrects when a URL leaves its sitemap.
 	fromSitemap: Boolean
+	# The EFFECTIVE cadence (ms) this due time was scheduled at — route interval, stored
+	# interval and demand-ladder rung already resolved. Denormalized for the same reason as
+	# `fromSitemap`: `claim` orders the due set by how overdue each row is RELATIVE TO ITS OWN
+	# cadence (util/renderPriority.js), and reading it per row would be a cross-database Target
+	# read on the hot claim path for every candidate in the window.
+	#
+	# OPTIONAL, unlike `fromSitemap`, and the asymmetry is deliberate. `put` replaces the record,
+	# so a writer that omits either field clears it — but omitting `fromSitemap` makes the
+	# renderer stop serializing a non-indexable sitemap page (a silent stop-caching bug), whereas
+	# omitting this one only makes the claim pass fall back to the ROUTE-resolved interval for
+	# that row's priority. That degrades ordering until the row's next render re-stamps it, and
+	# degrades nothing else — so it is an optional argument rather than a required one.
+	renderInterval: Long
 }
 
 type QueueStatus @table(database: "render_service") @sealed @export(name: "queue_status") {

packages/plugin/src/util/invalidationReenqueue.js

@@ -316,7 +316,15 @@ export const accelerateHeal = async ({ url, cacheKey, invalidatedBy }) => {
 		// schedule row we just read: `put` REPLACES the record, and the target is the field's source of
 		// truth, so this self-corrects a row whose flag went stale (same choice as the reschedule path).
 		await writeSchedules(
-			eligible.map((row) => ({ cacheKey: row.cacheKey, nextRenderTime: dueAt, fromSitemap: !!target.sitemapUrl }))
+			eligible.map((row) => ({
+				cacheKey: row.cacheKey,
+				nextRenderTime: dueAt,
+				fromSitemap: !!target.sitemapUrl,
+				// Already resolved above for the eligibility arithmetic, so recording it costs nothing
+				// and keeps an accelerated row's priority cadence rather than clearing it — `put`
+				// replaces the record, and a cleared field falls back to the route's interval.
+				renderInterval: interval,
+			}))
 		);
 	} catch (e) {
 		logger.error(e, `[prerender] could not accelerate ${cacheKey} after an invalidation`);