Skip to content

SUPERSEDED by #116: rank the due set by relative lateness (refuted approach) - #115

Closed
harper-joseph wants to merge 1 commit into
mainfrom
feat/render-priority
Closed

SUPERSEDED by #116: rank the due set by relative lateness (refuted approach)#115
harper-joseph wants to merge 1 commit into
mainfrom
feat/render-priority

Conversation

@harper-joseph

@harper-joseph harper-joseph commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Superseded by #116. Closed — do not merge.

This PR ranked the drained claim window by relative lateness, (now − dueAt) / renderInterval, with a multiplicative sitemap boost. The diagnosis was right and the comparator was wrong, for two reasons that #80 had already recorded and this PR did not account for:

  1. Relative lateness cannot be an order. It is linear in t with slope 1/interval, so two rows with different intervals cross exactly once — no stored key can express an order that changes with the clock, so no index can serve it.

  2. Re-ranking the claim window cannot reach the row that matters. The window is anchored at the oldest due time and is already an EDF prefix. Under the deep backlog this exists to fix, every row in it is ancient, and a homepage two of its own cadences late is numerically nowhere near the head of the index. It is never read, so it is never re-ranked. candidatePool (added here to widen the window) does not help: a wider window anchored at the same place is more ancient rows.

That second point is decisive — it is the actual production symptom this was meant to fix, and this approach does not fix it.

#116 implements the design decided in #80 instead: the lane goes in the high bits of nextRenderTime, each lane gets its own watermark and its own seek, and fairness is allocator policy rather than part of the key. A lane's head is its oldest row, not the corpus's.

Two pieces from here survive into #116, reworked: the renderInterval the row is banded by is threaded from the same write sites, and the "measure lateness relative to the page's own cadence" idea remains as observability rather than as ordering.

Branch feat/render-priority can be deleted.

…n cadence; v0.50.0

`claim` ordered by `nextRenderTime` and nothing else. That expresses priority
perfectly while the queue is caught up, and not at all once two rows are both
past due — a due time encodes when a page last rendered plus its cadence, not
how much it matters:

    home (1h cadence)  due 2h ago -> 2.0 intervals late
    PDP (48h cadence)  due 3h ago -> 0.06 intervals late

Index order hands the lease to the PDP, and nothing looks wrong while it does:
the floor advances, the scan stays fast, no row is wedged. The only symptom is
the served age `config.yaml` already describes — `interval + swrTtl`, which is
7x the homepage's cadence and 0.125x a PDP's — and it reads like a cadence
mis-set rather than an ordering problem.

So a due row is now ranked by `(now - dueAt) / renderInterval`, with
sitemap-sourced rows multiplied by `queue.priority.sitemapBoost`. Ordering
only: every row it reorders is already due, so this creates no work and cannot
move total render volume, which is why it ships enabled.

Three things are load-bearing:

- LATENESS, NOT AGE. `dueAt - interval` is not when the page last rendered:
  suppression rechecks schedule 7 days, `backoffWait` schedules up to
  `maxBackoff`, the unpin hatch pushes by `defaultInterval`. An age ratio would
  put a 7-day recheck on a 48h route at the head of the queue reading as 3.5
  cadences stale — promoting exactly the rows worth deprioritizing.
- THE FLOOR IS STILL DERIVED IN INDEX ORDER. The claim pass is now two phases:
  phase 1 walks the drain in index order and derives the floor from the first
  due row it sees; phase 2 grants from those rows in priority order. Deriving
  the floor from the priority walk would pick the most-overdue-by-ratio row
  instead of the minimum and strand everything below it, silently and forever.
- THE CADENCE IS READ OFF THE ROW. `renderInterval` is denormalized onto
  `RenderSchedule` for the same reason `fromSitemap` is — `claim` takes no
  Target read — because the effective cadence includes the demand ladder's
  rung, and resolving the route at claim time would rank a promoted catalog
  page at its 24h ceiling. Optional, unlike `fromSitemap`: omitting it falls
  back to the route interval and degrades ordering only, where omitting
  `fromSitemap` silently stops a page being cached.

`queue.priority.candidatePool` is the part with a real trade in it. The scan
window exists to read past the in-flight lease pile (`limit` + pile + `limit`),
so beyond the pile it holds about as many grantable rows as the pass is about to
grant — "pick the best 25" out of 25. It widens that last term to
`limit x candidatePool`, still capped by `claimScanCap`. Without it the
ordering is close to a no-op, which is what the test at grantLimit 2 found.

`sitemapBoost` is a multiplier and never a lane: an unserved row's ratio grows
without bound while the boost stays constant, so a discovered URL wins once its
ratio passes `boost x` the highest sitemap ratio in the window — ~2.4 cadences
at the default. Starvation is bounded and the bound is statable.

New metric `queue_health` `claim_lateness_pct`: how overdue each granted job
was as a percentage of its own interval, split sitemap/discovered. Normalized
on purpose — `route_page_age` is absolute, so a regression on a 1h route hides
inside a 48h route's numbers.

`queue.priority.enabled: false` grants in index order and walks the old,
narrower window: a revert of the behaviour, not a re-weighting of it. The
existing 715 tests pass untouched, because `runClaimPass` takes the orderer as
an argument and defaults to none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a priority-based queue ordering mechanism (queue.priority) to determine which due rows in the render queue get leased first. Instead of using absolute due time, rows are ranked by how overdue they are relative to their own cadence, with an optional multiplier boost for sitemap-sourced rows. It also adds a new metric claim_lateness_pct to track lateness percentages and denormalizes renderInterval onto the RenderSchedule table to avoid hot-path database reads. The review feedback highlights an issue in intervalOf where row.renderInterval (a Long that can round-trip as a BigInt) needs to be coerced to a Number before calling Number.isFinite to prevent the check from failing.

Comment on lines +164 to +165
const intervalOf = (row, intervalFor) =>
Number.isFinite(row.renderInterval) && row.renderInterval > 0 ? row.renderInterval : intervalFor(row.cacheKey);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The renderInterval field is defined as a Long in the GraphQL schema, which means it can round-trip as a BigInt from the database. Since Number.isFinite returns false for BigInt values, we should coerce row.renderInterval to a Number first before performing the finiteness check. This ensures robust handling if orderByPriority is ever called with raw database rows in other contexts or future changes, adhering to the general rule for validating database column values.

Suggested change
const intervalOf = (row, intervalFor) =>
Number.isFinite(row.renderInterval) && row.renderInterval > 0 ? row.renderInterval : intervalFor(row.cacheKey);
const intervalOf = (row, intervalFor) => {
const interval = row.renderInterval != null ? Number(row.renderInterval) : undefined;
return Number.isFinite(interval) && interval > 0 ? interval : intervalFor(row.cacheKey);
};
References
  1. When validating database column values (which may be surfaced as BigInt/Long) using Number.isFinite, always coerce the value to a Number first, as Number.isFinite returns false for BigInts.

@harper-joseph harper-joseph changed the title feat(plugin): rank the due set by lateness relative to each page's own cadence SUPERSEDED by #116: rank the due set by relative lateness (refuted approach) Aug 21, 2026
@harper-joseph

Copy link
Copy Markdown
Contributor Author

Closing: superseded by #116, which implements the decided design in #80. See the updated body for why the relative-lateness comparator here cannot fix the reported symptom — the claim window is anchored at the oldest due time, so the row that needs promoting is never read at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant