feat(serverless): report fitness failure to host before force-exit#537
Open
deanq wants to merge 3 commits into
Open
feat(serverless): report fitness failure to host before force-exit#537deanq wants to merge 3 commits into
deanq wants to merge 3 commits into
Conversation
|
Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
There was a problem hiding this comment.
Pull request overview
Adds a best-effort “fitness failure” report from the worker to the host right before the existing force-exit path, allowing the host to emit a queryable worker.fitness_failed event without weakening the SLS-379 hard-exit semantics.
Changes:
- Introduces
_report_unhealthy(check, reason)to send a single GET over the existing ping channel withstatus=unhealthy,check,reason(truncated), andrunpod_version. - Wires the report into
run_fitness_checks()’s failure path immediately before_terminate_unhealthy(1), swallowing all errors. - Extends the existing force-kill regression test suite with new coverage for the report behavior and ordering.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
runpod/serverless/modules/rp_fitness.py |
Adds _report_unhealthy and calls it on fitness-check failure before force-exit. |
tests/test_serverless/test_modules/test_fitness/test_force_kill.py |
Adds unit tests covering the unhealthy report behavior and “report before exit” ordering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+113
to
+115
| # Bound the best-effort unhealthy report so it never delays the exit. | ||
| _REPORT_TIMEOUT_SECONDS = 2 | ||
|
|
Comment on lines
+121
to
+125
| Sends a single GET to the ping URL (same URL/credentials the heartbeat | ||
| uses) with status=unhealthy plus the failing check name and reason, so the | ||
| host can emit a queryable worker.fitness_failed event. Any failure — no | ||
| ping URL, no API key, HTTP error, timeout — is swallowed: this must never | ||
| delay or block the os._exit that follows. |
Comment on lines
+280
to
+285
| # Best-effort report to the host so the failure is queryable; never | ||
| # allowed to block the force-exit below. | ||
| try: | ||
| _report_unhealthy(check_name, f"{error_type}: {error_message}") | ||
| except Exception: # never let reporting block the force-exit | ||
| pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Worker side of SLS-380. On a startup fitness-check failure, the worker sends one best-effort report to the host (over the existing ping channel) just before force-exiting, so the host can emit a queryable
worker.fitness_failedevent. Follow-up to SLS-379 (#536).Details
_report_unhealthy(check, reason)inrp_fitness.py: single GET toRUNPOD_WEBHOOK_PING(same URL/creds as the heartbeat) withstatus=unhealthy,check=<check name>,reason=<ErrorType: message>[:256],runpod_version. Bounded 2s timeout, no retries.run_fitness_checks, before the force-exit. The report is fully swallowed and the call site is wrapped, so it can never delay or block_terminate_unhealthy(1)(the SLS-379 exit guarantee).RUNPOD_WEBHOOK_PINGorRUNPOD_AI_API_KEYis unset (local/test).Stacking
Based on the SLS-379 branch (
deanq/sls-379-fitness-force-kill, PR #536) because the report only makes sense with theos._exitforce-kill. Retarget tomainonce #536 merges.Test plan
make quality-check: 532 passed, coverage 94.18%.Consumed by
The ai-api host PR that reads
status=unhealthyand emits@sls.event:worker.fitness_failed. This PR degrades to a no-op if the host ignores the param.Follow-ups (from final review, non-blocking)
SLS-380