RAG_EMBEDDING_PROVIDER config is incomplete — celery-worker/workbench never see it, and installed ragbio==2.0.3 predates the feature entirely

[man4ish]

Summary

docker-compose.yml's rag service block has an uncommitted diff (on
feat/regression-health-deployment) adding:

RAG_EMBEDDING_PROVIDER: sentence_transformers
RAG_EMBEDDING_MODEL: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract
RAG_EMBEDDING_DIMENSION: "768"

This is meant to be made safe by omnibioai-workbench PR #508
("embed_index_task derives --embed-model from RAG_EMBEDDING_PROVIDER,
not a hardcoded value"), which unified embed_index_task and
rag_query_task (plugins/literature_summarizer/tasks.py) on the same
env-var-driven embedding config so both agree on dimension.

Real end-to-end verification against the actual deployment (not PR
#508's own isolated verification) shows this is not safe to enable —
for two independent, compounding reasons.

What was actually verified (live stack, docker exec, real subprocess call)

1. The env vars are on the wrong service. They were added only to
   the rag service block (a separate standalone microservice — its
   own HTTP API on rag:8096, unrelated Node/Python codebase built
   fresh from omnibioai-rag source). The code PR #508 actually fixed —
   embed_index_task / rag_query_task — runs as Celery tasks inside
   the celery-worker (and workbench) containers. Confirmed via
   docker exec omnibioai-studio-celery-worker-1 env: neither
   RAG_EMBEDDING_PROVIDER, RAG_EMBEDDING_MODEL, nor
   RAG_EMBEDDING_DIMENSION is present there.

2. Even if the vars were set there, the installed ragbio package
   predates the feature entirely. celery-worker/workbench install
   ragbio from PyPI (Dockerfile.base: pip install ragbio --no-deps),
   currently version 2.0.3. Confirmed live:

   • ragbio.config in that container has no EMBEDDING_PROVIDER,
     EMBEDDING_MODEL, or EMBEDDING_DIMENSION attribute at all — only
     the old single-value MODEL_NAME (mxbai-embed-large).
   • python3 -m ragbio.embeddings.embedding_engine --help inside
     celery-worker shows only --study; no --embed-model flag
     exists.
   • Ran the exact subprocess command embed_index_task builds:

     python3 -u -m ragbio.embeddings.embedding_engine --study test_study --embed-model pubmedbert
     

     Result: hard crash —
     embedding_engine.py: error: unrecognized arguments: --embed-model pubmedbert

   By contrast, the standalone rag service — built directly from the
   local omnibioai-rag checkout (which already has PubMedBERT support
   merged, commit 89bc4d4) rather than pip-installed — correctly
   resolves EMBEDDING_DIMENSION=768 when given these env vars. That
   confirms the feature code is fine; it just isn't reaching the
   containers that need it.

   Root cause: omnibioai-rag's pyproject.toml is still at version
   2.0.3 even after the PubMedBERT commit — the feature was never
   version-bumped or published to PyPI, so pip install ragbio anywhere
   still resolves to the old release.

Why this matters

Enabling RAG_EMBEDDING_PROVIDER=sentence_transformers as currently
staged would not produce a silent 1024-vs-768 dimension mismatch (the
failure mode PR #508 was written to prevent) — it would hard-crash
every real embed_index_task run outright, immediately, on an
unrecognized CLI argument, since the deployed ragbio doesn't have the
flag the fixed task code passes it.

Required before this config is safe to enable

1. Add the same RAG_EMBEDDING_PROVIDER / RAG_EMBEDDING_MODEL /
   RAG_EMBEDDING_DIMENSION vars to celery-worker's (and
   workbench's) environment: blocks in docker-compose.yml, not
   just rag's.
2. Publish omnibioai-rag's PubMedBERT feature (commit 89bc4d4,
   already merged to omnibioai-rag main) as an actual new PyPI
   release of ragbio (version bump required — pyproject.toml is
   still 2.0.3).
3. Rebuild the workbench/celery-worker images against that new
   ragbio release so the installed package actually has
   EMBEDDING_PROVIDER/EMBEDDING_MODEL/EMBEDDING_DIMENSION and
   embedding_engine.py --embed-model.

Only once all three are done does PR #508's fix actually take effect in
this deployment. Until then, RAG_EMBEDDING_PROVIDER=sentence_transformers
stays uncommitted on feat/regression-health-deployment — see that
branch's working tree.

References

• omnibioai-workbench PR #508 (merged 2026-08-30T05:19:57Z):
  fix: embed_index_task derives --embed-model from RAG_EMBEDDING_PROVIDER, not a hardcoded value
• omnibioai-rag commit 89bc4d4: feat: support PubMedBERT 768-dimensional retrieval
• Verification performed live against the running
  omnibioai-studio compose stack on 2026-08-30, this repo,
  branch feat/regression-health-deployment.