[Fix] Avoid charging Jev API prices for local rail models - #17
alexwong10 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
really good find! Thank you.
Please make the solution more long-lasting instead of string matching
rails.py:124 decides pricing with decision_model.name == "jev". Every new backend with a price would need another string check there. Ask the model instead. DecisionModel already declares capability flags (supports_images, deterministic). Add a billing flag next to them, with no default so that every backend has to state it. A local/remote flag would misprice a free self-hosted server.
# s1a/decision_models/base.py
bills_input_tokens: bool # True when input tokens are priced at JEV_USD_PER_INPUT_TOKEN
# s1a/decision_models/jev.py
bills_input_tokens = True
# laya.py (LayaModel), cua.py (CuaS1Model), baselines.py (RandomModel, RuleModel), fakes.py (ScriptedModel)
bills_input_tokens = FalseThe tool and browser fronts have the same string check. Switch all three:
s1a/rails.py:124:... if decision_model.bills_input_tokens else 0s1a/browser/decision_model.py:778:... if self._decision_model.bills_input_tokens else 0s1a/tool/loop.py:277:ToolDecisionModelwraps the decision model. Copy the flag in its__init__next toself.name(s1a/tool/models.py:82), then gate the sum onisinstance(model, ToolDecisionModel) and model.bills_input_tokens. Keep excluding the"llm"fallback ticks (loop.py:208); their tokens are already priced as chat tokens.
Python doesn't enforce a bare annotation. A backend that forgets the flag raises AttributeError at its first summary. A smoke test catches that at test time:
# tests/test_decision_models_base.py
def test_every_decision_model_declares_whether_it_bills_input_tokens(self) -> None:
backends = {backend.__name__: backend for backend in DecisionModel.__subclasses__()}
self.assertLessEqual({"JevModel", "LayaModel", "CuaS1Model", "RandomModel", "RuleModel", "ScriptedModel"}, set(backends))
for name, backend in backends.items():
with self.subTest(backend=name):
self.assertIsInstance(vars(backend).get("bills_input_tokens"), bool)__subclasses__() only sees imported modules. The subset check fails if an import is missing, so import the backend modules at the top of the test file.
With the flag in place, the comment at rails.py:123 can go. Your two new tests should pass unchanged and cover the refactor.
Why
Running a rail evaluation with
--model layacurrently counts local input tokens as Jev tokens and prices them using the Jev API rate. For example, 300 local input tokens are reported as $0.000013 in API charges.How
Only the
jevbackend contributes tojev_input_tokensins1a/rails.py, matching the existing tool and browser fronts. Per-decision token usage remains available to the rail; local compute costs are not estimated as API charges.What
Local rail summaries, both returned and saved, now report zero Jev tokens and API cost. Jev accounting and evaluation metrics retain their existing behavior. Regression coverage uses the real Jev adapter over
ScriptedTransportand the real Laya adapter over a fake SDK, without API keys or model downloads.Verification
a0ae3e1with(jev_input_tokens, cost_usd) == (300, 0.000013)and passes with this fix.ruff format --check .,ruff check .,ty check,uv lock --check --offline, anduv build --offline.pytest -q: 481 passed, 41 skipped, 2 failed. Both failures are the existingTestBrowserHandsLifecycle.test_serving_shuts_the_static_server_down_when_the_session_ends/..._fails_to_openchecks: the closed-port probe raisesTimeoutErroron this Windows host. Both reproduce on unchangeda0ae3e1with the same environment.scripts/smoke.shstops at thelistcomparison because Windows CLI output contains CRLF. This also reproduces on unchangeda0ae3e1; the script itself was read with LF line endings for the check.