fix: harden Threadmill for 1.0 qualification

[hemju]

Summary

Durable jobs could lose retry decisions, regress acknowledged execution updates, release another workflow's concurrency hold, or remain unfinished across datastore outages. A committed claim whose response was lost could also stay PROCESSING indefinitely because owner-wide heartbeats kept renewing it. This change implements the production-hardening audit in #135 and bounded topology qualification in #98 while preserving Threadmill's at-least-once delivery contract.

• Persist failure decisions and execution revisions; retain terminal finalizers through transient outages; renew only confirmed ID/claim-version pairs; withdraw stopped-node registrations and leases after their last in-flight write.
• Enforce workflow-hold ownership, bulk budgets and lifecycle size reserves across all stores. Use cutoff-eligible retention pages with opaque cursors, bounded recovery, fair Redis admission, idle metadata cleanup, and PostgreSQL queue counters/indexes. Queue cleanup prefilters active queues without locking their counter rows.
• Harden dashboard authorization, host security, mutation races and redaction; observe after-commit submission failures; preserve application classloaders and execution-scoped metrics/tracing cleanup. Give custom failure policy precedence over the retry fallback.
• Add shared and real-backend regressions, 0.3.0 upgrade fixtures, Redis version/topology/failover coverage, monitoring measurements, retention churn, and compatibility/operations/soak guides.

Upgrade impact

Existing PostgreSQL databases apply additive V7–V11 migrations; validation, index creation and queue-counter backfill require a maintenance window with applications stopped. Existing Redis namespaces require the offline, resumable format-2 index migration and Redis 7.4+ with noeviction. Stop all old workers/producers, back up, upgrade modules together, and recompile applications/custom stores. Custom stores must implement the complete updated SPI, including bounded active-attempt heartbeats and opaque retention cursors. Mixed 0.3.0/candidate operation is unsupported; rollback requires restore. Review legacy FAILED jobs whose retry decisions were not persisted. See docs/compatibility.md.

Validation

• The frozen e0e5b47 candidate's ./gradlew productionCheck -PdependencyScanRequired=true --max-workers=2 --no-build-cache passed in 10m 19s: 1,669 JUnit cases, zero failures/errors/skips, including 173 PostgreSQL, 413 Redis and 32 fixed-soak cases. The gate also passed 41 UI tests, five browser scenarios, all three short correctness simulations, both process-separated nudge simulations, required OSV/npm scans, the example, Javadoc and artifact inspection. All 14 binary JARs were also inspected.
• The final operations-guide-only clarification passed ./gradlew check in 4s. Its runtime JAR checksums are identical to that full production gate; the source-manifest record identifies that documentation-only delta. The later harness-only restart fix below still needs its own full production gate.
• Permanent regressions cover lost claim acknowledgement, exact active-attempt heartbeat matching, active-queue counter availability during idle cleanup, and retention deletion after the one-minute metadata grace. All 33 review threads received an implementation or reasoned response and were resolved.

Endurance qualification

The frozen e0e5b47 PostgreSQL mixed-workload run failed at the scheduled eight-hour restart on 2026-09-10 at 07:49 UTC, before its required 12-hour duration. Its producer exited with a new-connection error while reconciling an uncertain insert: PostgreSQL's shutdown/startup refusal (57P03) was missing from the harness retry classification. There is no final harness summary; progress.json saying finished with aborted=false is not evidence of a pass. The independent final audit found 720,007 SUCCEEDED jobs, matching counters, no other job states, and zero active concurrency holds. No job loss was observed in that audit, but the incomplete phase does not qualify correctness, performance or stability for the planned duration. The original datastore, dump, logs, trace, JFR/GC and failed verdict remain preserved.

The harness-only follow-up accepts PostgreSQL connection failures and restart states 57P01/57P02/57P03, retains the same job ID, and leaves unrelated SQL errors fatal. 16 producer unit cases and one real PostgreSQL restart regression pass; the real test holds PostgreSQL in smart shutdown, verifies 57P03, restarts the server and requires one durable insert. The follow-up check result and commit are recorded in the validation update comment. Full productionCheck will run after the independent Redis campaign to avoid resource interference; the prior gate does not automatically qualify the changed harness.

The seven outstanding Redis phases automatically started at 07:49:52 UTC under .local-reference/qualification/20260909T2258Z-redis-remaining, on new isolated volumes and the original unchanged 43-JAR runtime. Redis remains independent of the PostgreSQL failure. The plan includes 30-minute baselines, 12-hour mixed and retention runs, and 30-minute retry/check-in/nudge phases at 25 jobs/second (long-running deliberately 1/second with one-quarter planned timeouts). Long phases include node churn every ten minutes and 20-second pauses at hours 2, 6 and 10. Local focused validation overlaps the Redis mixed baseline; it is labelled diagnostic for performance and must be replaced by a clean baseline before performance sign-off.

After Redis finishes, validate and freeze the harness fix, collect the replacement clean Redis baseline, and run a new full PostgreSQL campaign, including preflight, both 12-hour phases and the remaining scenarios. Do not resume the failed eight-hour phase or replace its results. Carry forward Redis evidence only after verifying product JAR equivalence. Endurance, performance and stability qualification remain pending.

Artifacts for the original campaign are under .local-reference/qualification/20260909T2225Z. The five-minute soak-only monitor remains active; PR review/CI/merge polling was stopped at the user's request. Explicitly requested branch/worktree cleanup is complete: only main and codex/1.0-production-hardening remain locally/remotely, with one worktree. Qualification data and frozen runtimes were preserved.

The campaign retains live invariant checks, per-job traces and lifecycle latencies, rolling datastore-operation timings, host/datastore/JVM samples, GC/JFR, fault timelines, final raw-record/index/counter audits and database snapshots. An evidence aggregator produces 30-minute throughput, retention-population, memory and datastore windows with explicit exclusions and matching-scenario baseline comparisons. Performance and stability require analysis after full runs; a running phase is not a pass.

The predecessor campaign on 0fc9881 was stopped gracefully and preserved after review changed production code. Its five completed preflight/baseline phases passed correctness/drain for that old candidate; its Redis retention baseline is interrupted evidence. Those results do not qualify the current candidate. Hours-scale Redis 8.6/Sentinel/Cluster and a very large retained FAILED backlog remain separate qualification shapes; the bounded failover tests and mixed/short-retention campaign do not establish those performance claims. The pending-retry indexing/FAILED-population benchmark follow-up is recorded on #135.

Keep this PR in draft until endurance findings have been reviewed. No release or merge is part of this change.

Checklist

• At-least-once delivery remains explicit.
• Store behavior changes have shared contracts or named real-backend regressions.
• API/SPI and upgrade changes are documented.
• Review feedback is fixed or answered with reasons and resolved.
• Private notes and generated artifacts are excluded from Git.
• Complete endurance/performance/stability qualification.

Refs #135, #98.

Redis qualification audit correction

The Redis preflight finished all 2,251 jobs and passed the harness invariants, but monitoring found a private-auditor bug: its HVALS-based active-group check treated 11 idle_since timestamps as live work. The original RDB was restored into an isolated read-only Redis instance; all 20 shared/exclusive counter pairs were zero, pending/workflow holds were zero, and job records, state indexes and counters agreed. The original run recorded zero evictions. The original failed audit remains intact alongside audit-review.json and snapshot-audit-corrected.json, which explicitly accept preflight correctness/drain. Three auditor regressions now pass, including the preserved snapshot. No Threadmill source or frozen JAR changed.

The original PostgreSQL supervisor has now exited after the separate hour-eight harness failure described above. The Redis continuation started automatically and runs only its seven outstanding phases; no completed phase is repeated and no original datastore is reset. The soak monitor tracks the preserved failure and independent Redis progress. Endurance, performance and stability qualification remain pending.

[hemju]

🤖 AI-generated review (Claude Code) — this is an automated review, not a human reviewer

Scope. Full read of all 179 changed files (code and docs), the #135 tracker and #98, and the Spring Boot 4.0.8 auto-configuration ordering checked against the actual 4.0.8 jars/sources (the OAuth2 resource-server config orders itself before = SecurityAutoConfiguration, so the new host fallback chain correctly backs off for it). I did not run the full gate locally; CI on this head is green (build & check, browser smoke, dependency security).

Overall. The correctness design is sound and consistent with the repo's §6 principles: FailureDecision persisted in the same write as FAILED, onProcessingFinished as an engine-guaranteed finally hook keyed by context identity, the attempt-local execution revision in all three stores, PostgresTransactions (Error path, abort on failed rollback), key-routed script-cache recovery, the frozen v0.3.0 fixtures and nonempty upgrade tests. Nothing below affects the at-least-once contract. The main concerns are about the cost model of the new bounded maintenance: several activities that used to run once per retentionInterval (or only over eligible rows) became continuous full-population scans with body reads. For a 1.0 candidate about to run 12-hour soaks that defines the leader's baseline load, so I would address items 1–2 before the endurance campaign.

Findings (details are in the inline comments)

1. High — retention passes are O(total terminal population), not O(eligible), on all three stores. deleteFinishedPage pages by id with no cutoff predicate; the cutoff is applied per row after the read, and MaintenanceCycle.sweepTerminalState only ends a pass at a short page. With defaults (succeededRetention 7d) and the planned 50 jobs/s that is ~30M SUCCEEDED rows read (with body) per pass, ≤5,000 rows/tick, i.e. hours of continuous full-table reading, then retentionInterval, then again. v0.3.0 used the (state, current_state_at) index and touched eligible rows only. Redis does an HGETALL per id over the whole :ids index; the in-memory store sorts the whole map per page. Suggested fix: keep the cutoff in the query and keyset-page on (current_state_at, id) (index already exists; Redis ZRANGEBYSCORE with a (score,id) cursor), which still advances past protected rows. That needs RetentionPage.nextAfter to be an opaque cursor rather than a JobId — worth deciding now, since this is new public SPI that 1.0 freezes. (PostgresJobStore#1676, RedisJobStore#1711, MaintenanceCycle#337, RetentionPage#11, JobStore#391)
2. Medium — retry recovery and workflow reconciliation are now continuous full scans every maintenancePollInterval (1 s). 500 full bodies deserialized per tick from FAILED (30-day default retention, nearly all final) and from AWAITING, plus one findById per AWAITING child (the removed handledParents dedup did batch that). Options: a scalar retry_at (V7 already touches the table) / Redis ZSET so the scan only visits recoverable rows; a pause between complete passes (recovery latency = pass interval, the crash window is rare); parent-id dedup per page; pipelined Redis loads. (MaintenanceCycle#211, RetryInterceptor#134, WorkflowInterceptor#90, RedisJobStore#2330)
3. Medium — idle concurrency-group reclamation churns hot keys. Any zero-count group is deleted on the next tick, so a fixed key set that idles between jobs is deleted and re-created continuously (dead tuples on threadmill_concurrency_groups, and the new lockConcurrencyGroup retry loop on the claim path). last_modified already exists; an idle-age threshold would avoid this. Redis could store the last-claim time as the registry score. (PostgresJobStore#1595, cleanup_concurrency.lua#8)
4. Medium — threadmill_queue_counts is never pruned. 16 rows per distinct queue name ever seen; queue-family apps with per-tenant queue names grow it without bound, and queueDepths() (metrics refresh, dashboard, queue-family discovery) does a full GROUP BY over it on every call. Redis prunes its queue registry when empty. (V9__queue_monitoring.sql#2)
5. Medium (product decision) — the Redis 7.4 hard gate no longer has a technical driver in this PR. The HSCAN NOVALUES dependency the audit cited is removed here (the regression asserts zero hscan calls) and I found no other 7.4-only command. The gate now refuses Redis 7.0/7.2 and every Valkey build (Valkey reports redis_version:7.2.4, which covers ElastiCache/Aiven Valkey), and the only bypass also disables the noeviction check. (RedisJobStore#2240)
6. Medium (API) — onProcessingFailureDecision is unreachable for user interceptors in a ProcessingNode. ProcessingNode registers RetryInterceptor first (ProcessingNode.java:82-86, unchanged file) and it always returns a non-null decision, so "first decision wins" means builder-registered interceptors can never influence the disposition, although the Javadoc invites that. Either consult user interceptors first or document that the built-in policy is authoritative. (JobInterceptor#33)
7. Medium (behavior vs. docs) — recurring materialization visits 64 definitions per tick. With N definitions a task is visited every ceil(N/64)+1 ticks; docs/wake-driven-pollers.md:113 and the AGENTS.md nudge section still promise a one-maintenancePollInterval bound. Idle definitions are one state read each, so a 500-row page (like the other scans) with the 200 ms budget as the real bound would keep the documented latency; the docs need updating either way. (RecurringMaterializer#110)
8. Low / nits (inline): SQL string-replace in queryJobs (PostgresJobStore#2216); hardcoded '{threadmill}:storage_format'/'2' and index-suffix literals duplicated across Lua and RedisKeys (insert_all.lua#66); lifecycle compaction at budget 0 drops threadmill.* engine metadata and the truncation marker (JsonJobSerializer#148); V9 index makes threadmill_jobs_state_time_idx redundant while every transition now maintains four state-leading full-table indexes (V9#53); V7's CHECK forces a full validation scan under ACCESS EXCLUSIVE (V7#3); in-memory keyset pages sort the whole map (InMemoryJobStore#596); double withdrawal on normal registry shutdown (NodeRegistry#109); completedRetentionStates cleared while dedup still has backlog (MaintenanceCycle#327); unbounded while (true) in lockConcurrencyGroup (PostgresJobStore#2330); duplicated condition in Job.checkIn (Job#262); deferred enqueues serialized twice and byte arrays allocated just to count bytes (TransactionAwareJobScheduler#217); the starter-only security test cannot tell Threadmill's fallback chain from Boot's default chain (ThreadmillDashboardSecurityStarterAutoConfigTest#63).

Consistency notes

• docs/wake-driven-pollers.md:113 and AGENTS.md ("latency is bounded by maintenancePollInterval (default 1 s)") are stale for >64 recurring definitions (item 7).
• docs/compatibility.md mentions the V9 table lock but not the V7 constraint-validation scan.
• threadmill-store-redis/README.md: the key-layout table still says by_state_time:{STATE} is "used for retention" and does not list the new keys (storage_format, storage_format_migration, concurrency_counters, by_state_time:{STATE}:ids, cron_tasks:ordered, queue_keys:{q}:ordered, concurrency:{k}:pending:exclusive, ...:pending:ready:...); only the prose section describes format 2.
• Process: one 8.5k-line squashed commit for 22 tracked steps, while the repo convention asks for small commit-sized units. Not blocking for a draft, but bisectability would benefit from per-audit-item commits before merge.

Verified and found consistent (no action needed)

M01 hold-ownership guard in save_atomic.lua plus the shared contract test; M02 head-probe admission (id:MODE members, :exclusive mirror, memberJobId); M05/M06 quarantine and claim-poison paths in all three stores including Redis lock release on locked re-snapshot failure; M07 revision CAS in all three stores and the serialized ExecutionContext flushes; M08 PostgresTransactions; M10 auto-config ordering (host config after SecurityAutoConfiguration, before the dashboard config, which is before ServletWebSecurityAutoConfiguration; ERROR-dispatch permit); capFailureMessage UTF-8 boundary logic; CronExpression.previousOrSame DST/leap-year handling; NodeRegistry stop/finally withdrawal ordering; RedisIndexMigration lease/renew/resume flow; JsonJobSerializer wrapping of malformed-wire runtime failures into SerializationException.

I will re-check this PR periodically for new pushes and update this review.

[hemju]

🤖 AI follow-up review (automated) — new commit 0fc9881

Re-checked the PR after the push. The new commit fix(soak): recover producer writes after datastore outages is a fast-forward on top of the reviewed head 0b8e6be and is harness-only (RecoveringProducerStore, SoakHarnessRunner wiring, ProducerRecoveryTest, RedisProducerOutageTest, docs). It does not touch any of the 27 findings from the first review; all of them still apply to this head.

Notes on the new code (none blocking):

• The reconcile logic is sound for the three producer operations: lost-acknowledgement inserts are confirmed by id/spec/createdAt before retrying, a partially visible atomic batch fails the run explicitly, and non-outage failures are rethrown immediately. The docs correctly frame this as harness behavior rather than a Scheduler guarantee.
• One nit inline: the wrapped enqueueIfAbsent discards the caller's now argument.
• Observation: isOutage's SQL-state branch (08xxx, 57P01) is only reachable if the JDBC connection actually fails; a paused PostgreSQL container blocks the producer socket instead (unless socketTimeout is set on the harness DataSource), so only the Redis path is exercised by a real pause in RedisProducerOutageTest. Worth stating in docs/soak-plan-1.0.md next to the 20-second pause schedule so a blocked producer during the PostgreSQL pauses is not read as a harness defect.

CI for this head was still pending at the time of this check; the periodic re-check continues.

[hemju]

🤖 AI follow-up review (automated) — commit c412913

Re-checked after the push. c412913 is a fast-forward on the previously reviewed head and addresses the first review's findings. I verified each claim against the code in this head (not only the reply text); CI is green on it (build & check, browser smoke, dependency security).

Verified fixed (24 threads): cutoff-bounded (current_state_at, id) keyset retention in all three stores with an opaque RetentionCursor and a fixed per-pass cutoff; V11 index plus drop of the redundant two-column index; bodies fetched only for FAILED; Redis binary-search seek through timestamp ties in retention_candidates.lua; 30-second pause after complete recovery passes, per-page parent cache, pipelined Redis loads; 1-minute idle grace for concurrency groups (last_modified / idle_since reset by claim_commit); deleteIdleQueueMetadata; user decision hooks consulted before the built-in retry fallback without changing notification order; JOB_PROJECTION instead of SQL string replace; Lua token substitution pinned by LuaProtocolTest; threadmill. metadata and truncation markers never dropped (reject instead); ConcurrentSkipListMap maintenance indexes in the in-memory store (I checked that every jobs mutation routes through the index helpers and that isTerminal includes FAILED); bounded group-lock retry with SQLSTATE 40001; Utf8.length; completedRetentionStates cleared only after the whole retention block; starter-test assertions; RecoveringProducerStore keeps the caller's now.

Declines accepted (3 threads): the Redis 7.4 gate as a qualification boundary (README now says so and scopes externallyValidatedMode()), the V7 CHECK (same-transaction validation would not release the lock sooner; documented), and the 64-definition recurring page (docs corrected; the one-poll bound claim removed). The NodeRegistry double withdrawal was already accepted earlier.

One new finding on the new code (medium, inline on PostgresJobStore#1640 / MaintenanceCycle#230): deleteIdleQueueMetadata selects every queue as a candidate and its per-queue DELETE takes FOR UPDATE SKIP LOCKED on all shard rows of each candidate before the zero-sum and no-ENQUEUED filters apply. On the leader this runs every maintenance poll, so active queues' counter rows are row-locked once per second inside one transaction, and the trigger's ON CONFLICT DO UPDATE from concurrent producers and claims on those queues waits for that commit. It also has no idle grace, so bursty queues that drain between batches churn their rows. Pre-filtering candidates to zero-sum queues without ENQUEUED work before locking, and running both metadata reclamations on the retention cadence rather than every poll, would remove the hot-path contention without weakening the reclamation guarantee.

Residual, not blocking (inline on the pass interval): the 30-second pause bounds the idle case; with a large FAILED population under the 30-day default retention a pass is roughly population/500 ticks long, so scanning still dominates. If the endurance run shows the leader continuously scanning FAILED, the indexed pending-retry representation remains the durable fix.

Reviewed head recorded as c412913; the periodic re-check continues.

[hemju]

🤖 AI follow-up review (automated) — commit 6fe1e6b

Fast-forward on c412913; verified against the code in this head.

Second-round findings: the queue-metadata lock scope is fixed as suggested — a read-only GROUP BY prefilter (zero-sum and no ENQUEUED job) selects idle candidates, only those enter the locked delete where both conditions are rechecked, and the cursor still advances over active queues; idleQueueCleanupDoesNotLockActiveQueueCountersWhileDeletingAnotherQueue proves an active queue's counter stays writable while an idle queue's delete is blocked. The cadence decline is accepted: a 100-name page per hour cannot keep up with sustained unique-name churn, and the prefilter removes the hot-path contention I was worried about. The pass-interval decline is accepted with the follow-up recorded on #135.

New engine change in this commit (not from the review): per-attempt execution heartbeats. touchExecutionHeartbeats(nodeId, activeClaims, now) replaces the owner-wide refresh on the engine path; the runner registers each execution/finalization context with its claim version and only those are renewed. This fixes a real liveness hole: a claim whose reply was lost (or a worker killed by a fatal error) used to stay PROCESSING forever on a live node because the owner-wide heartbeat shielded it. I checked the things that could regress:

• The dispatcher claims at most availablePermits() per poll and is the only acquirer of its semaphore, so acquire() never blocks; a claimed job is registered within the submit latency and the claim-time heartbeat covers that window. Queue-family lanes return the first non-empty claim per poll, so the same bound holds.
• Version matching is stable across processing: execution updates bump only the execution revision, finalizers keep the claim version until the terminal save commits, and a dashboard delete or replacement changes state, so a stale attempt cannot refresh a newer claim (pinned by the new shared contract test).
• Postgres uses a VALUES join with GREATEST; the Redis script matches state/owner/version per hash and is monotonic; the in-memory path routes through the index helpers.

Three low notes inline / below, none blocking:

• Dispatcher.dispatchClaimed (unchanged file, no inline anchor) still says a dispatch failure would leave claimed jobs "shielded by the node-wide owner heartbeat indefinitely"; that rationale is now inverted — undispatched claims expire — so the comment should describe the release as avoiding a needless heartbeatTimeout wait plus a burned attempt.
• ExecutionHeartbeats.snapshot rejects the whole batch when any entry has a non-persisted version, which would skip heartbeats for up to 500 live attempts because of one bad registration; dropping the offending entry (or rejecting it at JobRunner.run) fails smaller.
• An empty active set makes the heartbeat tick succeed without touching the store, which also clears claimSuspended; that changes the suspend-claiming semantics from "store reachable" to "nothing to renew". Probably acceptable since the dispatcher breaker covers claims, but worth a sentence in the operations guide.

CI on this head: browser smoke and dependency security green, build & check still pending at the time of this review; the periodic re-check continues. Reviewed head recorded as 6fe1e6b.

[hemju]

🤖 AI follow-up review (automated) — commits 905e117, e0e5b47

Fast-forward on 6fe1e6b. Two small commits: RetentionChurnSmokeTest now runs 85 seconds on the real stores so the sustained profile outlasts the one-minute idle-metadata grace while keeping its deletion assertion, and docs/operations.md documents that an idle execution-heartbeat tick makes no datastore call and clears heartbeat-related claim suspension without being a health probe. Both are consistent with the code reviewed on 6fe1e6b.

Thread outcomes: the ExecutionHeartbeats.snapshot fail-fast decline is accepted (whole-batch preflight is a coherent SPI contract, and the engine never registers non-persisted versions); the idle-tick semantics are now documented as suggested. The only remaining item from the last review is the stale rationale in the Dispatcher.dispatchClaimed Javadoc (unchanged file), which is a comment-only nit.

No open findings above nit level. CI: smoke and dependency scans green, build & check pending at the time of this check. Reviewed head recorded as e0e5b47.

[hemju]

The PostgreSQL mixed-workload endurance phase stopped at the scheduled hour-eight restart on 2026-09-10 at 07:49 UTC: exit 3, no final summary, and only eight of the required twelve hours. Its final independent audit found 720,007 SUCCEEDED jobs, matching counters and no remaining concurrency holds. This incomplete phase is not a qualification pass; all evidence and the datastore are preserved.

Fixed in d3e0b57: the harness now retries PostgreSQL connection failures and restart states 57P01/57P02/57P03, including failures while reconciling an uncertain insert. Unrelated SQL errors still fail immediately. The observed shutdown message maps to 57P03 in PostgreSQL 18's connection-startup code. The new real PostgreSQL test independently verifies that SQLSTATE and recovery of the original job ID.

Validation: 16 focused unit cases + one real PostgreSQL restart test passed, followed by ./gradlew check in 41 seconds, including all 61 soak unit cases, with no failures/errors/skips. The changed harness still needs a complete productionCheck; the earlier 1,669-case gate belongs to frozen e0e5b47.

The independent seven-phase Redis continuation started automatically at 07:49:52 UTC and remains running with the unchanged frozen 43-JAR runtime. Local validation overlapped its mixed baseline, so that baseline is diagnostic for performance and requires a clean replacement. After Redis completes, run the full production gate, freeze the corrected candidate, and execute a fresh complete PostgreSQL campaign on new volumes. Preserve this failure; do not resume it or count it as a pass. Endurance/performance/stability qualification and 1.0 sign-off remain pending. PR feedback polling stays stopped; soak monitoring continues.