Fix parallel catchup hanging on orphaned in-progress jobs

[jayz22]

what

• Worker script: make claiming a job atomic, so it can never land in the progress queue without an owner
• Job monitor: detect and requeue jobs that nothing is working on
• PCv2 mission: fail the mission when it stops making progress, rather than waiting forever
• Some hardening along the way: redis data structure tightening, client timeouts, no hard-fail on metrics extraction

why

A recent parallel catchup build hung due to a stale state (has job in progress but no worker working). Claiming a job and recording its owner were two separate steps, so a transient Redis failure in between left a job that no worker was running and that nothing in the system could recognize as stuck.

This removes the gap that loses the job, gives the monitor a way to recover if it happens anyway, and makes any future variant fail quickly.

[Copilot]

Pull request overview

Hardens parallel catchup recovery and stall handling to prevent indefinitely hung builds.

Changes:

• Adds atomic job claiming and orphan recovery.
• Introduces Redis timeouts and idempotent completion.
• Adds mission stall detection and configurable monitor images.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Review
src/MissionParallelCatchup/parallel_catchup_helm/values.yaml	Configures the monitor image.
src/MissionParallelCatchup/parallel_catchup_helm/templates/job_monitor.yaml	Uses the configured monitor image.
src/MissionParallelCatchup/parallel_catchup_helm/files/worker.sh	Completion is not ownership-safe.
src/MissionParallelCatchup/job_monitor.py	Orphan requeue operations are not atomic.
src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs	Stall detection excludes queued work; two minor wording issues remain.

Suppressed comments (4)

src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs:457

• The failure message is inaccurate when monitorNotAdvancing alone triggers the timeout: the last status can still report live workers. Name both failure conditions so operators know whether worker liveness or the monitor heartbeat caused the abort.

                        failwith "Catch up stalled, no progress and no live workers"

src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs:435

• Use “unable to update” to correct the grammar in this comment.

                // itself gets stuck unable to updating its internal metrics and

src/MissionParallelCatchup/job_monitor.py:170

• This detects only jobs with no hash entry, not jobs whose recorded owner is dead. With jobs owned by pods A and B, if A is down but B responds, the later workers_up == 0 gate skips all retries, so A's job remains in progress forever; the mission-side aggregate liveness check is also masked by B. Confirm liveness per owner and requeue only the jobs belonging to owners that remain down.

            # --- Phase 1b: Requeue orphaned jobs (in progress, but owned by nobody) ---
            orphans = [job for job in jobs_in_progress if job not in job_owners]
            for job in orphans:
                logger.error("Requeuing orphaned job %s: in %s with no owner in %s",
                             job, PROGRESS_QUEUE, JOB_OWNERS)
                requeue_job(job)

src/MissionParallelCatchup/parallel_catchup_helm/files/worker.sh:125

• A nonempty invalid value has already been moved into PROGRESS_QUEUE and registered in JOB_OWNERS by the claim script. This branch only logs and sleeps, so that entry remains owned but is never processed or completed; the orphan detector will not recover it because its owner exists. Atomically remove the claim and either place the malformed item in the failed set or terminate it according to the desired invalid-job policy.

    elif [ -n "$JOB_KEY" ]; then
        echo "Error: Unexpected claim reply, not a job key: $JOB_KEY"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.