Replace pre-enqueued scheduled jobs with a ScheduleState ledger#76
Open
davegaeddert wants to merge 1 commit into
Open
Replace pre-enqueued scheduled jobs with a ScheduleState ledger#76davegaeddert wants to merge 1 commit into
davegaeddert wants to merge 1 commit into
Conversation
The worker's scheduler used to enqueue each JOBS_SCHEDULE entry's next run as a future JobRequest, days ahead of its start time. Those materialized rows outlived the code that created them: removing or renaming a scheduled job class left a pending row that errored at pickup after the deploy, and changing an entry's timing let the old slot fire one last time. Scheduling is now evaluated at tick time against a ScheduleState ledger — one row per entry identity (job class + concurrency key + schedule), recording the last slot handled. Nothing is enqueued before its slot comes due, so removing an entry simply stops it from being evaluated. A slot is claimed with an optimistic UPDATE guarded by the previous ledger value, with the enqueue in the same transaction, which makes each slot fire exactly once across any number of workers with no leader election. Around the core: - A catch-up window (JOBS_SCHEDULE_CATCHUP_WINDOW, default 24h) replays the most recent missed slot after downtime while keeping a stale ledger row (an entry removed and re-added later) from firing a long-gone slot. The window also bounds the catch-up walk. - load_schedule validates entries fully (types, registration, cron ranges, duplicate identities) and a jobs.schedule preflight check reports every broken entry at deploy time. The admin gains a Schedule page that renders broken entries instead of failing. - JobClassNotRegistered replaces the bare KeyError when a stored job_class no longer resolves. - ScheduledCommand keys digest only when the raw command would overflow the 255-char concurrency_key with the slot stamp appended. - A PruneScheduleLedger chore deletes ledger rows for removed entries (queue-scoped workers can't tell those from another queue's). - Transition machinery for pre-ledger workers still running during the upgrade (a completed-run dedupe and a provenance-verified sweep of pre-enqueued future rows) is marked 'pre-ledger transition' and removable one release later. The ScheduleState unique constraint is created in the migration rather than convergence: workers get_or_create the same deterministic key at boot, and that race safety needs the constraint to exist before the first write. Claude-Session: https://claude.ai/code/session_01AEyVNMqpAhB1KVo6jrSPMk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Removing an already-scheduled job class left a pending
JobRequestthat errored at pickup after the deploy — the worker's scheduler pre-enqueued each entry's next run days ahead, and those materialized rows outlived the code that created them. Changing an entry's timing had the same shape: the old slot fired one last time.What
Scheduling is now evaluated at tick time against a
ScheduleStateledger — one row per entry identity (job class + concurrency key + schedule), recording the last slot handled. Nothing is enqueued before its slot comes due, so removing an entry simply stops it from being evaluated. Slots are claimed with an optimisticUPDATE ... WHERE last_enqueued_slot = <old>sharing a transaction with the enqueue: exactly-once across any number of workers, no leader election.JOBS_SCHEDULE_CATCHUP_WINDOW, default 24h): the most recent missed slot replays after downtime, while a stale ledger row (entry removed and re-added later) never fires a long-gone slot.load_schedulefully validates entries (types, registration, cron ranges — including string fields that previously bypassed range checks — and duplicate identities). Ajobs.schedulepreflight check reports every broken entry at deploy time.JobClassNotRegisteredreplaces the bareKeyErrorwhen a storedjob_classno longer resolves.PruneScheduleLedgerchore GCs ledger rows for removed entries.pre-ledger transition, removable one release later. Overdue pre-enqueued rows are deliberately kept so the old catch-up behavior holds through the upgrade.ScheduleState's unique constraint is created in the migration rather than convergence — workersget_or_createthe same deterministic key at boot, and that race safety needs the constraint to exist before the first write.Review
Five internal multi-agent review rounds, two external (Codex) reviews, and an implementation comparison against Solid Queue, GoodJob, sidekiq-cron, Oban, and Quartz (which validated the claim mechanism as stronger than leader-election designs and aligned the catch-up default with Quartz's misfire default). 36 new tests; 126 passing in plain-jobs.
https://claude.ai/code/session_01AEyVNMqpAhB1KVo6jrSPMk