Skip to content

feat!: align media capabilities and add grounded-query tooling - #135

Merged
tulayha merged 6 commits into
mainfrom
benchmark/codex-mcp-ablation
Aug 31, 2026
Merged

feat!: align media capabilities and add grounded-query tooling#135
tulayha merged 6 commits into
mainfrom
benchmark/codex-mcp-ablation

Conversation

@tulayha

@tulayha tulayha commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add FineLAP-backed sound search and use semantic speech, scene, action, and sound capability names across the shared CLI, HTTP, MCP, Desktop, package, and generated contracts
  • add grounded query planning and synthesis with official Ollama qwen3.5:4b-q4_K_M, while preserving deterministic evidence retrieval when the model is unavailable
  • add a Desktop Local grounded answers option that reuses a healthy loopback Ollama service or, with explicit approval, installs Ollama through Winget/Homebrew, starts only a VidXP-owned service when necessary, streams the approved model download, and never stops an externally owned service
  • propagate the private model endpoint and selected model into browser, worker, API, Premiere, copied stdio MCP, and Codex registration without exposing a user-entered URL or forwarding unrelated environment variables
  • wire the Premiere panel to the shared durable grounded-query job, render locally generated claims with numbered evidence, and retain ranked evidence-only results when generation is unavailable
  • add a pinned Promptfoo/Codex SDK harness for matched MCP-on and MCP-off runs, deterministic temporal/tool-boundary scoring, and a ten-task LongVALE part-nine pilot spanning scenes, actions, environmental sound, speech, and joint evidence
  • document the dataset subset and media acquisition boundary, setup/indexing procedure, cost boundary, OVSD scope, model-selection evidence, managed local-answer lifecycle, and current evidence limitations
  • bump the index schema so repositories using removed capability names rebuild against the new collection contract

Breaking change

dialogue and videoprism are removed as public capability names and package extras. Use speech and action, then rebuild existing indexes. No compatibility aliases are retained.

Cost and ownership boundary

Promptfoo Community and the local VidXP/Ollama processes have no per-run fee. The approximately 3.4 GB Qwen model uses local disk, memory, compute time, and electricity. Codex benchmark runs consume the user's applicable Codex allowance or API usage; hosted judge calls would be additional paid model usage. Desktop asks before installing Ollama or downloading the model and does not bundle model weights.

Validation

  • uv run --no-sync ruff check .
  • uv run --no-sync python -m pytest -q: 732 passed, 5 skipped, 106 subtests passed
  • uv run --no-sync pytest -q tests/test_mcp.py tests/test_codex_plugin.py tests/test_cli.py: 99 passed, 7 subtests passed
  • npm --prefix desktop run check: typecheck, lint, production build, and 47 tests passed
  • cargo test --manifest-path desktop/src-tauri/Cargo.toml: 89 passed
  • cargo test --release --locked --manifest-path desktop/src-tauri/Cargo.toml: 89 passed
  • npm run model-catalog:check from desktop/
  • npm run notices:check from desktop/
  • Promptfoo 0.122.2 configuration validation under Node 24.19.0
  • npx --yes markdownlint-cli2@0.23.2 INSTALLATION_GUIDE.md docs/local-api.md docs/architecture/platform.md docs/integrations/openai-plugin.md
  • git diff --check
  • all ten pilot intervals checked against the pinned local LongVALE evaluation annotations
  • npm --prefix premiere run check: typecheck, lint, 14 tests across 6 files, and both UXP and CEP production builds passed
  • uv run --no-sync python -m pytest -q tests/test_api.py tests/test_query_service.py: 50 passed

No Codex inference runs, Ollama/Qwen model download, paid API call, dataset media download, or real Premiere host run was performed. The Ollama executable is not installed on this workstation, so the real Qwen schema/resource gate remains explicit follow-up validation rather than a claimed end-to-end result. Lychee was unavailable and the Docker daemon was stopped, so the pinned link check could not run. The direct uv run --no-sync pytest -q Windows shim omitted the checkout root when collecting five utils.* modules; the complete suite passed through uv run --no-sync python -m pytest -q.

Replace the dialogue and videoprism capability identifiers with speech and action across shared contracts, packages, extras, adapters, generated catalogs, benchmarks, and documentation. Bump the index schema so repositories rebuild under the new collection names.

BREAKING CHANGE: dialogue and videoprism are removed as capability names and package extras. Use speech and action, then rebuild existing indexes.
@tulayha
tulayha force-pushed the benchmark/codex-mcp-ablation branch from 34544b5 to c34373d Compare August 29, 2026 19:56
@tulayha tulayha changed the title feat(benchmarks): add Codex MCP ablation harness feat!: rename media capabilities and add MCP ablation harness Aug 29, 2026
@tulayha tulayha changed the title feat!: rename media capabilities and add MCP ablation harness feat!: align media capabilities and add grounded-query tooling Aug 30, 2026
@tulayha
tulayha merged commit 10237cc into main Aug 31, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant