feat(webeval): add OrcaRouter as a first-class chat completion provider

[putraperdana1207-pixel]

Summary

Fara runs its computer-use agents against OpenAI-compatible chat endpoints, and the webeval evaluation framework already models those endpoints through a small provider registry (CHAT_COMPLETION_PROVIDER). This PR adds OrcaRouter as a first-class named provider there, mirroring the existing openai branch.

OrcaRouter is an OpenAI-compatible AI gateway built for both models and agents. Like OpenRouter, it exposes a provider/model namespace across many models — but it also combines adaptive routing, automatic failover, zero-markup inference, observability, guardrails, and agent-tool governance behind the same endpoint. Adding orcarouter as a first-class provider means this project's users can use that stack directly, without treating OrcaRouter as an anonymous custom base URL. It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Changes

• webeval/src/webeval/oai_clients/wrapper.py — new OrcaRouterClientWrapper subclassing OpenAIClientWrapper. When no base_url/api_key is supplied it defaults to https://api.orcarouter.ai/v1 and the ORCAROUTER_API_KEY env var, so the config can be as small as:

  {
    "CHAT_COMPLETION_PROVIDER": "orcarouter",
    "CHAT_COMPLETION_KWARGS_JSON": { "model": "openai/gpt-4o" }
  }

  Models are addressed with vendor/model names, or orcarouter/auto for adaptive routing.
• webeval/src/webeval/oai_clients/create_utils.py — added the orcarouter branch to create_completion_client_from_env, right beside the openai branch, and documented it in the factory docstring.
• webeval/src/webeval/oai_clients/__init__.py — exported the new wrapper.
• webeval/tests/test_oai_clients.py — added test_orcarouter_provider_round_trip covering factory resolution, provider metadata, default base URL, and a stubbed create() round trip.

Validation

• webeval/tests/test_oai_clients.py — 13 passed; full webeval/tests/ suite — 32 passed, 2 skipped.
• ruff check src (repo root lint gate) — clean.
• Live smoke test against https://api.orcarouter.ai/v1 through the new provider path (both orcarouter/auto and openai/gpt-4o-mini) returned finish_reason=stop with real content and token usage.

---

I'm an engineer on the OrcaRouter team. Discord: discord.gg/YEubt8enRA · X: https://x.com/OrcaRouter

[putraperdana1207-pixel]

@microsoft-github-policy-service agree