feat(plugin): render lanes — priority in the high bits of nextRenderTime

packages/plugin/README.md

@@ -518,6 +518,95 @@ The floor advances to **the first due row a pass observed**, which is the same t
 Set `queue.claimFloor.enabled: false` to roll the floor back to the old full seek. It changes nothing
 else; leases stay where they are either way.
 
+#### Render lanes: which due row goes first
+
+The floor decides _where the scan starts_. `queue.lanes` decides _which scan_ — and that distinction
+is the whole feature, because a single scan can only start in one place.
+
+`claim` is strictly `nextRenderTime`-ascending, so under any capacity deficit the queue serves
+whatever is oldest-due. Two production measurements say that is the wrong order
+([#80](https://github.com/HarperFast/prerender-plugin/issues/80)):
+
+- **Provenance.** During a multi-hour backlog, **239,090 of 521,929 overdue rows (~46%)** were
+  bot-discovered rather than sitemap-submitted — half the capacity going to pages nobody submitted
+  while submitted pages aged out of their SWR window.
+- **TTL-blindness**, which is sharper. Absolute due time treats a 1 h-TTL homepage 3 h overdue exactly
+  like a 48 h-TTL product page 3 h overdue. The first is 300% stale, the second 6%. Simulated over the
+  real corpus, the 1 h route sits at **4.78× its own TTL even at full capacity** and 48.83× at half,
+  against 1.08× / 2.00× for the 48 h route.
+
+**Why not just rank the due window by relative lateness.** Because it cannot be an _order_:
+`(t − dueAt) / interval` is linear in `t` with slope `1/interval`, so two rows with different
+intervals cross exactly once, and no stored key can express an order that changes with the clock.
+Re-ranking inside the claim window does not rescue it either — the window is anchored at the _oldest_
+due time and is already an EDF prefix, so a homepage 600 minutes down the queue is never read at all.
+Widening the window does not help for the same reason.
+
+So relative lateness is the _rationale_ for which lane a row is in, never the comparator. Lanes are
+cut so intervals inside one are similar; within a lane the order is unchanged (earliest due first),
+which is optimal for maximum lateness.
+
+**The encoding is `nextRenderTime = lane × 2^42 + dueAt`.** One column, the existing index, no schema
+change, no second index, no migration — every pre-existing row is already a valid lane-0 row, and a
+lane change is an in-place numeric update. Measured: three lanes interleaved in one index, each with
+its own watermark, read in **0.29–0.32 ms per lane**; the same lane with its watermark reset costs
+**3.46 ms**. Interleaving is free — the watermark is the entire win, which is why each lane gets one
+(in its own shared buffer, so `renderLease.js` keeps its layout and its invariants).
+
+Lanes, in priority order, all **derived at write time** from provenance, cadence and failure count —
+so a band or route change applies on each URL's next render with no sweep:
+
+| lane               | from                                                                           |
+| ------------------ | ------------------------------------------------------------------------------ |
+| `urgent`           | operator intent — the admin re-render action, and an authenticated `renderNow` |
+| `submitted/b0…bN`  | has a `sitemapUrl`, banded by `queue.lanes.ttlBands`                           |
+| `discovered/b0…bN` | crawler-found, banded the same way                                             |
+| `cold`             | past the fast-retry lane, or a suppressed target's recheck                     |
+
+**Fairness is scheduler policy, deliberately not part of the key** — so it is tunable live with no
+rewriting of stored rows. `urgentMaxShare` caps lane 0's _drain share_ rather than its admission: a
+hard structural bound that needs no token bucket, so a bulk force-render cannot take more than that
+fraction of any batch. `minShare` reserves **floors**, not fixed shares: strict priority starves the
+tail at _every_ capacity level including 100% (and its lag numbers look good precisely because it
+drops work — starvation is invisible in a lag metric), while fixed shares summing to 1.0 leave nothing
+for the priority order to spend. Floors measured far better in the tail — discovery reaching 71 h
+instead of 133 h at 50% capacity. An unclaimed floor is released back to lane order within the same
+pass, so a floor for a class with nothing due costs nothing.
+
+**Enabling it is two steps, and the order matters.** Lane 0 is `urgent` _and_ lane 0 is the unencoded
+value — that is what makes the encoding migration-free, and it is also the trap: on a fresh deploy the
+whole corpus reads as urgent, and `urgentMaxShare` would ration the queue to a fifth of capacity. So:
+
+```sh
+# 1. re-stamp: rewrites each due time into its derived lane, in place, changing NO due time.
+#    Bounded and repeatable — call until "done": true.
+curl -sk -X POST https://<node>:9926/prerender_admin/queue -u <super-user>   -H 'content-type: application/json' -d '{"action":"restamp-lanes","limit":5000}'
+```
+
+```yaml
+# 2. then, in config.yaml:
+queue:
+  lanes:
+    enabled: true
+```
+
+The re-stamp refuses to run once `enabled` is true, because at that point a lane-0 row can no longer
+be told apart from one an operator deliberately marked urgent. It cannot derive a stored `changefreq`
+interval, a demand-ladder rung or `cold` (those live on the Target, and reading it would be a point
+read per row on an 814k corpus), so a promoted catalog page is stamped one band slow and a suppressed
+URL one lane fast — both corrected on that row's next render. Being one band optimistic for one cycle
+is the right direction to be wrong in.
+
+**Disabling is not an instant rollback.** Rows already encoded stay encoded until each renders again,
+and while lanes are off they sort after every unencoded row, so they drain slowly rather than
+promptly. Nothing is stranded — reads decode unconditionally — but to get the old behaviour back at
+once, re-stamp with lanes disabled, which writes every row back to lane 0.
+
+Watch `queue_health` `lane_granted` (jobs granted per pass, lane in the method slot). Lanes
+redistribute a fixed amount of work, so no total moves when they engage and this is the only series
+that shows the split — a lane sitting at zero while its backlog is non-empty is the failure to look
+for, and it is indistinguishable from health anywhere else.
+
 ## HTTP & resource API
 
 | Method & path                                | Purpose                                                             |

packages/plugin/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/prerender",
-	"version": "0.49.0",
+	"version": "0.50.0",
 	"type": "module",
 	"description": "Configurable Harper plugin for prerendering pages for bots and crawlers",
 	"license": "Apache-2.0",

packages/plugin/src/configSchema.js

@@ -1262,6 +1262,94 @@ export const configSchema = group('Prerender plugin configuration.', {
 				'about could not be detected.',
 			{ min: 1, scope: 'restart' }
 		),
+		lanes: group(
+			'RENDER PRIORITY. `claim` is strictly `nextRenderTime`-ascending, so under any capacity ' +
+				'deficit the queue serves whatever is oldest-due — regardless of whether the site owner ' +
+				'submitted the page or a crawler found it, and regardless of how tight its freshness ' +
+				'budget is. Two production measurements say that is the wrong order (see ' +
+				'prerender-plugin#80):\n\n' +
+				'  PROVENANCE: during a multi-hour backlog, 239,090 of 521,929 overdue rows (~46%) were ' +
+				'bot-discovered rather than sitemap-submitted — half the render capacity spent on pages ' +
+				'nobody submitted, while submitted pages aged out of their SWR window.\n\n' +
+				'  TTL-BLINDNESS: absolute due time treats a 1h-TTL homepage 3h overdue exactly like a ' +
+				'48h-TTL product page 3h overdue. The first is 300% stale, the second 6%. Simulated over ' +
+				'the real corpus, the 1h route sits at 4.78x its own TTL even at FULL capacity and ' +
+				'48.83x at half, against 1.08x / 2.00x for the 48h route.\n\n' +
+				'Lanes fix the ORDER, never the volume: the same number of pages render either way. A ' +
+				'lane is encoded in the high bits of `nextRenderTime` (`lane x 2^42 + dueAt`), so there ' +
+				'is no schema change, no second index and no migration — every pre-existing row is ' +
+				'already a valid lane-0 row. Within a lane the order is unchanged (earliest due first), ' +
+				'which is optimal for maximum lateness; lanes are cut so the intervals inside one are ' +
+				'similar, and the lane a row belongs to is DERIVED at write time from its provenance, ' +
+				'its cadence and its failure count, so a config change here applies on each URL’s next ' +
+				'render with no sweep.\n\n' +
+				'READ `enabled` BEFORE TURNING THIS ON. The rollout is two steps and the order matters.',
+			{
+				enabled: option(
+					false,
+					'Off by default, and enabling it is a TWO-STEP OPERATION.\n\n' +
+						'Lane 0 is `urgent` and lane 0 is the UNENCODED value — which is what makes the encoding ' +
+						'migration-free, and is also the trap: every row written before lanes existed is ' +
+						'numerically in lane 0, so on a fresh deploy the WHOLE CORPUS reads as urgent and ' +
+						'`urgentMaxShare` would ration the entire queue down to a fifth of capacity.\n\n' +
+						'So: (1) `POST /prerender_admin/queue {"action":"restamp-lanes"}` and let it finish — it ' +
+						'walks the schedule rows and rewrites each due time into its derived lane, in place, ' +
+						'changing no due time; (2) set this true. Both steps are safe in either order in the ' +
+						'sense that nothing is lost, but doing (2) first means running with the whole corpus in ' +
+						'lane 0 until (1) completes.\n\n' +
+						'DISABLING IS NOT AN INSTANT ROLLBACK. Rows already encoded stay encoded until each one ' +
+						'renders again, and while this is false they sort after every unencoded row — so they ' +
+						'drain slowly rather than promptly. Nothing is stranded (reads decode unconditionally), ' +
+						'but if you need the old behaviour back at once, re-stamp with lanes disabled, which ' +
+						'writes every row back to lane 0.'
+				),
+				ttlBands: option(
+					[HOUR, 12 * HOUR],
+					'Cadence cuts (ms, ascending) that split `submitted` and `discovered` into lanes. A row ' +
+						'goes in the first band its render interval does not exceed; anything slower lands in the ' +
+						'overflow band, so N cuts make N+1 bands.\n\n' +
+						'This is the half that fixes TTL-blindness, and it is why the coarse classes alone are ' +
+						'not enough: a class whose intervals are NOT similar reproduces the problem inside itself. ' +
+						'At the default, a 1h homepage, a 6-12h catalog page and a 48h product page occupy three ' +
+						'different lanes, so the homepage stops queueing behind product pages that are older in ' +
+						'absolute terms but barely stale in their own terms.\n\n' +
+						'Set cuts at your ACTUAL route cadences, not at round numbers — a band that contains two ' +
+						'route cadences an order of magnitude apart is a band that does nothing. An empty list ' +
+						'leaves one band per class, i.e. provenance-only priority.',
+					{ unit: 'ms' }
+				),
+				urgentMaxShare: option(
+					0.2,
+					'Ceiling on the fraction of one claim batch the `urgent` lane may take.\n\n' +
+						'A DRAIN-SHARE cap, not an admission limit: urgent is served strictly first but never ' +
+						'gets more than this share of a batch, so the lanes below always get at least ' +
+						'`1 - urgentMaxShare`. That is a hard structural bound with no token bucket and no ' +
+						'admission bookkeeping to get wrong, and it is what stops a bulk "re-render everything ' +
+						'now" from becoming a queue-wide outage. Earliest-due-first within the lane, so a flood ' +
+						'is still served oldest-first.\n\n' +
+						'`0` disables the lane entirely. Any share small enough to floor to zero jobs still ' +
+						'admits one, because a cap of zero would make an operator’s force-render silently never ' +
+						'run.',
+					{ min: 0, max: 1 }
+				),
+				minShare: option(
+					{ discovered: 0.1, cold: 0.02 },
+					'Minimum share of each claim batch reserved for a coarse class, keyed by name ' +
+						'(`urgent`, `submitted`, `discovered`, `cold`).\n\n' +
+						'FLOORS, NOT FIXED SHARES, and the simulation is unambiguous about why. Strict lane ' +
+						'priority starves the tail at EVERY capacity level including 100%, and its lag numbers ' +
+						'look good precisely because it drops work — starvation is invisible in a lag metric. ' +
+						'But fixed shares summing to 1.0 are also wrong: they leave nothing for the priority ' +
+						'order to spend, and deviating from earliest-due-first only ever costs. Reserving a ' +
+						'MINIMUM for the classes that need protecting and letting the rest compete in lane ' +
+						'order measured far better in the tail — discovery reaching 71h instead of 133h at 50% ' +
+						'capacity, 29h instead of 40h at 75%.\n\n' +
+						'A reservation that goes unclaimed is released back to lane order within the same pass, ' +
+						'so a floor for a class with nothing due costs nothing. Shares are of the batch, so they ' +
+						'need not sum to 1; a sum above 1 is clamped by the batch size itself.'
+				),
+			}
+		),
 		claimScanCap: option(
 			1000,
 			'Ceiling on schedule rows read per claim pass. A leased row keeps its overdue position in the ' +

packages/plugin/src/http_handlers/bot_request.js

@@ -383,7 +383,15 @@ async function renderNow({ url, cacheUrl, deviceType, cacheKey, request, routeSc
 	// would strand: on this node the funnel lowers the floor in-process, and on any other node —
 	// which is ~75% of keys, since schedule rows are residency-pinned — the guard band is what
 	// keeps the row above the owner's floor and therefore claimable.
-	await writeSchedule(cacheKey, { nextRenderTime: currentMinuteMs(), fromSitemap: !!renderTarget?.sitemapUrl });
+	// URGENT, because that is precisely what this path is: an authenticated request that says render
+	// this page now and wait for it. It is rate-limited by `queue.lanes.urgentMaxShare` like every
+	// other lane-0 write, so a spammed debug header degrades to slow render-nows rather than to a
+	// stalled queue.
+	await writeSchedule(cacheKey, {
+		nextRenderTime: currentMinuteMs(),
+		fromSitemap: !!renderTarget?.sitemapUrl,
+		urgent: true,
+	});
 
 	// Wake idle consumers now instead of waiting out the periodic status sync. Non-force
 	// so a paused queue stays paused (the render then simply times out to the fallback).

packages/plugin/src/metrics.js

@@ -377,15 +377,22 @@ export const METRICS = Object.freeze({
 					'paused = 1 when this node’s queue is paused at snapshot time, else 0 — makes "paused for hours" ' +
 					'alertable without polling the REST surface. ' +
 					'claim_scan_ms = claim-pass duration; watch the p95 trend, not the level. ' +
+					'lane_granted = jobs granted per claim pass to each render lane (queue.lanes), with the lane in ' +
+					'the method slot — `urgent`, `cold`, or `submitted/bN` / `discovered/bN` for the TTL bands. ' +
+					'Lanes redistribute a fixed amount of work, so no total moves when they engage and this is the ' +
+					'only series that shows the split. A lane sitting at zero while its backlog is non-empty is the ' +
+					'failure to look for (a minShare floor too low, or a ttlBands list that put two very different ' +
+					'cadences in one band); it is indistinguishable from health anywhere else. ' +
 					'reconcile_restored / reconcile_missing = schedule gaps repaired / found per sweep (they differ ' +
 					'when the per-sweep restore cap truncates the pass); expect zero — a steady rate means ' +
 					'something is CREATING gaps, and the reconcile log line names the URLs.',
 			},
 			method: {
-				name: 'result (claim_scan_ms only)',
-				values: ['granted', 'empty', 'capped'],
+				name: 'result (claim_scan_ms) | lane (lane_granted)',
+				values: ['granted', 'empty', 'capped', 'urgent', 'submitted/bN', 'discovered/bN', 'cold'],
 				description:
-					'Only claim_scan_ms uses this slot: granted = jobs handed out, empty = nothing due, capped = the ' +
+					'claim_scan_ms and lane_granted use this slot. On claim_scan_ms: ' +
+					'granted = jobs handed out, empty = nothing due, capped = the ' +
 					'scan hit queue.claimScanCap without reaching a not-yet-due row (in-flight work is filling the ' +
 					'window). Every other series emits null here.',
 			},
@@ -641,6 +648,20 @@ export const metrics = Object.freeze({
 	/** One claim pass's duration and how it ended — a queue_health series, so the queue reads in one scan. */
 	claimScan: (durationMs, result) => server.recordAnalytics(durationMs, 'queue_health', 'claim_scan_ms', result, null),
 
+	/**
+	 * Jobs granted to one lane by one claim pass, labelled by lane.
+	 *
+	 * WITHOUT THIS THERE IS NO WAY TO ANSWER "IS PRIORITISATION WORKING". The whole mechanism is a
+	 * redistribution of a fixed amount of work, so no total moves when it engages — only the split
+	 * does, and the split is invisible in every existing series. A lane whose share collapses to zero
+	 * while it still has due rows is the failure mode (a floor set too low, a band list that put two
+	 * cadences in one lane), and it looks exactly like health from the outside.
+	 *
+	 * Emitted per lane per pass rather than per job: a pass grants at most `maxClaimLimit` jobs across
+	 * a handful of lanes, so this is a few emits per claim rather than one per render.
+	 */
+	claimLane: (granted, laneLabel) => server.recordAnalytics(granted, 'queue_health', 'lane_granted', laneLabel, null),
+
 	/** One origin proxy on the serve path: time to response headers, status, and why. */
 	originFetch: (durationMs, statusCode, reason) =>
 		server.recordAnalytics(durationMs, 'origin_fetch', statusCode, reason, null),