Skip to content

fix: support generic OpenAI-compatible local model servers (Ollama) in the guided install - #92

Merged
bradtaylorsf merged 1 commit into
masterfrom
fix/openai-compatible-local-model-server
Aug 21, 2026
Merged

fix: support generic OpenAI-compatible local model servers (Ollama) in the guided install#92
bradtaylorsf merged 1 commit into
masterfrom
fix/openai-compatible-local-model-server

Conversation

@maxtechera

@maxtechera maxtechera commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What

create-answer-engine aborts the local (lmstudio-provider) install at stage 2/6 when the model server is a generic OpenAI-compatible server rather than LM Studio itself. prepareLmStudioModels probes LM Studio's native management API (GET <origin>/api/v1/models, POST <origin>/api/v1/models/load); servers like Ollama serve /v1/* (chat, embeddings, models) but 404 on the native endpoints, so the install dies with Could not inspect LM Studio model state: HTTP 404.

This PR treats a 404 from the native inspect endpoint as "no management API — models load on demand" and skips the pre-load step. Model availability is still validated against /v1/models earlier in resolveModelSetup, and a reachable LM Studio keeps the exact current behavior (inspect + explicit load).

Why

Found while doing the v1.1.1 guided install (INSTALL_AGENT.md) on Windows 11 WSL2 with Ollama as the local model runtime. Everything else in the local route already works with any OpenAI-compatible server:

  • --lm-studio-url http://127.0.0.1:11434/v1 + --models chat=...,embedding=... validates fine against Ollama's /v1/models
  • containerBaseUrl correctly rewrites localhost → host.docker.internal (verified reachable from containers)
  • The runtime's OpenAiCompatibleProvider chat + embeddings calls work against Ollama (verified from inside a container: chat completion with qwen3:30b-a3b-instruct-2507-q4_K_M, embeddings with nomic-embed-text at 768 dims)

This one function was the only hard blocker.

Testing

  • New unit test: 404 on the native endpoint → resolves without a load call (existing LM Studio test unchanged)
  • packages/create: 149/149 tests pass, type-check and build clean
  • Ran the built prepareLmStudioModels + listLmStudioModels against a real Ollama 0.20.2 server: lists models, gracefully skips pre-load

Extra setup notes from the same install attempt (not part of this diff)

  1. The v1.1.1 guided install is currently impossible for everyone: the v1.1.1 (and v1.1.0) GitHub Releases are still drafts with zero assets — the runbook's pinned bootstrap URLs 404. The Installer release assets workflow has never been dispatched. Inputs that line up for it: tag=v1.1.1, source_commit=a346a8c6a7833596c088ccdebadd69e3608c386b, runtime_image=ghcr.io/the-answerai/answer-engine@sha256:418c93dac3753e0850f5ba6627581f888acdd65f3b91b655bd898cee56f10701. Worth confirming the built bootstrap matches the checksum pinned in the tagged INSTALL_AGENT.md (ea7e4632…) before publishing.
  2. Ollama users need an embedding model pulled first (ollama pull nomic-embed-text, 768 dims) — Ollama typically ships with chat models only.
  3. Cosmetic: preflight's MODEL_RUNTIME check probes the lms CLI only, so it warns "LM Studio was not detected" even when an OpenAI-compatible server is up (non-blocking; probing the configured /v1/models URL could fix it in a follow-up).
  4. Model IDs must match /v1/models exactly — for Ollama that includes the tag (e.g. nomic-embed-text:latest).

🤖 Generated with Claude Code

Closes #93

prepareLmStudioModels aborts the local install when the model server
answers /v1 but 404s on LM Studio's native management API
(/api/v1/models). Generic OpenAI-compatible servers such as Ollama load
models on demand, so a 404 there now skips the pre-load step instead of
failing stage 2/6. Model availability is still validated against
/v1/models earlier in resolveModelSetup.

Verified against a real Ollama 0.20.2 server on WSL2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bradtaylorsf

Copy link
Copy Markdown
Contributor

Temporarily closing and reopening to retrigger CI after correcting the SHA-pinned Actions allowlist; no code state is being changed.

@bradtaylorsf bradtaylorsf reopened this Aug 20, 2026
@bradtaylorsf
bradtaylorsf merged commit 7d41d03 into master Aug 21, 2026
2 checks passed
@bradtaylorsf
bradtaylorsf deleted the fix/openai-compatible-local-model-server branch August 21, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: support OpenAI-compatible local model servers without LM Studio management APIs

2 participants