fix(celery-worker): add OLLAMA_BASE_URL + OMNIBIOAI_MYSQL_PASSWORD, matching workbench - #74
Merged
Merged
Conversation
…atching workbench Live audit finding (2026-08-31, plugins/bio_agent post-#518 scan): celery-worker runs the exact same omnibioai-workbench image as `workbench` (only 2 services in this compose file build from that image, confirmed by grepping every `context:` line), but was missing two env vars workbench already has. - Without OLLAMA_BASE_URL, LLMService fell back to the ollama client's own default (127.0.0.1:11434) -- nothing inside this container -- so every real BioQueryAI analysis Celery task failed immediately with "Failed to connect to Ollama" as soon as it reached the LLM step. - Without OMNIBIOAI_MYSQL_PASSWORD, Workflow Planner's MySQL workflow-candidate scan (_resolve_mysql_config in workflow_planner_service.py) failed with "MySQL password missing". Traced the real fallback chain before adding this (not just the two var names the error message named): host/port/user/db all already resolve correctly via this block's existing DB_HOST and hardcoded defaults -- only password had no working fallback. Verified post-fix: sha256 of all 7 relevant source files (views.py, 5 bio_agent handlers, workflow_planner_service.py) match exactly between celery-worker and the already-verified workbench/local source. Real end-to-end BioQueryAI + Workflow Planner submissions confirm the fix itself works -- Workflow Planner's MySQL scan genuinely succeeds and produces a real multi-block plan (301s, no MySQL error); BioQueryAI's Ollama call genuinely connects and returns a real 200 (no connection error). Two NEW, separate, real bugs surfaced by getting past this blocker (not fixed here, filed as follow-ups): 1. Neither `workbench` nor `celery-worker` mounts /app/work/planner or /app/work/bioqueryai as a shared volume (unlike /app/work/runs, /app/work/objects, etc., which are). Both status-polling endpoints (planner_ask_status, bioquery_ask_status) read from that path in workbench's own container, but the Celery task that writes the completion status file runs in celery-worker's container -- so a genuinely-completed task's result is written somewhere the poller can never see it. Confirmed live: Workflow Planner's task succeeded in celery-worker's log with a full real plan, but polling from workbench showed PENDING forever. 2. BioQueryAI's Celery worker process SIGSEGV'd immediately after a successful Ollama response, losing the task result via WorkerLostError. Circumstantial cause: two large models (deepseek-r1:32b ~55GB + llama3.1:70b) loaded concurrently, host swap at 15Gi/15Gi (100%) at crash time. Root cause not confirmed -- dmesg is permission-denied in this environment, so OOM-kill correlation could not be directly verified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VxRzmWKquRKGwebL21d6wH
Collaborator
Author
|
Follow-up issues filed for the two new bugs this fix's E2E testing surfaced (both in omnibioai-workbench, deliberately not fixed tonight):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to the bio_agent post-#518 live audit finding.
celery-workerruns the exact same image asworkbenchbut was missing 2 env vars workbench already has, breaking BioQueryAI's and Workflow Planner's real Celery-executed steps. Full detail in the commit message.Verified: sha256 match on all relevant source (7 files) between celery-worker and workbench/local. Real E2E confirms the fix itself works — Workflow Planner's MySQL scan genuinely succeeds now (real 301s plan generation, no MySQL error); BioQueryAI's Ollama call genuinely connects (real 200 OK, no connection error).
Not fixed here, filed as follow-ups (2 new, separate, real bugs this fix's own E2E testing surfaced):
/app/work/plannerand/app/work/bioqueryaiaren't shared volumes betweenworkbenchandcelery-worker— a genuinely-completed task's status file is invisible to the polling endpoint, which lives in a different container. Confirmed live via a real completed Workflow Planner run stuck at PENDING forever from the caller's side.WorkerLostError). Circumstantial cause: 2 huge models loaded concurrently + host swap fully exhausted at crash time — not confirmed,dmesginaccessible in this environment.🤖 Generated with Claude Code