Judge ends the run at the first turn when a criterion has not been reached yet

[rogeriochaves]

Summary

The judge's decision phase ends a scenario run at the very first turn when a criterion has not been reached yet, even though the conversation was still moving towards it. The verdict is a failure the user cannot act on, because the agent never got the chance to do the thing being judged.

Reproduction

Demo agent: "ACME checkout".

• Scenario: "Guest completes checkout"
• Situation: "A guest wants to review their cart, pay with a credit card, and place the order in one conversation."
• Criteria:
  1. "Uses the review_cart tool before quoting totals"
  2. "Places the order after successful payment and returns an order number"
• Judge and simulator model: openai/gpt-5.2
• Library: @langwatch/scenario 1.3.0
• Run: scenariorun_0000w1dkXfFxS00PFJGjPUU7SBvjm (batch scenariobatch_0000w1dkXfFxS00PFJGjPUU7SBvjk), duration 17s

The whole transcript was two messages:

1. user (simulated): "show my cart then let me pay with credit card and place the order"
2. assistant: listed the cart with line items, subtotal, shipping and total, then asked for the card number, expiry and name on the card.

At that point the judge called make_verdict, then finish_test with verdict failure and this reasoning:

Criterion 1: The trace shows a tool call/result consistent with review_cart (the JSON cart payload with lines/subtotal/shipping/total) before the assistant quoted totals, so this is met.
Criterion 2: The assistant only requested card details; there is no evidence in the transcript or traces of a successful pay tool call, a subsequent place_order tool call, or an order number being returned. Therefore not met.

Criterion 2 was not violated. It was not reached yet: the agent had just asked for the card details it needed to reach it. The next turn was the one that would have produced the payment and the order number.

Verified against stored data: simulation_runs for that run has Status = FAILED, Verdict = failure, two messages in the transcript, DurationMs = 17308. The judge's ai.generateText.doGenerate spans show make_verdict with empty input, immediately followed by finish_test carrying the failure verdict.

Where the behavior comes from

Source: javascript/src/agents/judge/judge-agent.ts, DECISION_PHASE_RULE at line 241 and buildDecisionSystemPrompt around lines 270 to 300. The Python SDK carries the same text in python/scenario/judge_agent.py around line 828. In the published package the same text is in dist/index.js around lines 2476 and 2494, with the continue_test / make_verdict tool wiring around lines 2908 to 2935.

The decision-phase prompt asks a single question:

only decide whether the conversation has collected enough information to evaluate the criteria: call make_verdict when it has, or continue_test to let the conversation play out

"Enough information to evaluate the criteria" gets read by the model as "I can already tell criterion 2 is not satisfied", which is true of almost any unfinished conversation. The two existing rules do not stop it:

• "Call make_verdict as soon as the agent has clearly broken one of the 'do not' or 'should not' criteria" only covers violations.
• "while the conversation is still short, lean towards continuing" is a soft preference, and it lost here on turn one.

Nothing in the decision prompt distinguishes "the criterion is already violated" from "the criterion has not happened yet".

Why this matters

This is the first scenario a new user runs from the guided onboarding in the platform. The first result they see is a red failure on a criterion the agent was one turn away from meeting, with reasoning that reads as correct but describes a conversation that was cut short. There is nothing for them to fix in their agent.

The platform does not override the judge prompt: platform/app/src/server/scenarios/execution/scenario-child-process.ts calls ScenarioRunner.judgeAgent({ criteria, model }) and passes nothing else, so the prompt in this repo is the only place this is decided.

Suggested direction

Not prescribing an implementation, but the decision phase needs the missing distinction:

• Continue while a criterion is unmet and nothing in the conversation rules it out. Things that rule it out: a refusal, an error the agent does not recover from, a contradiction of the criterion, or the user leaving the conversation. Only verdict early when a criterion is already violated, which is what the first rule was for.
• Or a minimum-turn floor before an early failure verdict is allowed, so a run cannot end in failure on the agent's first reply.