Symptom
On phone-call (Twilio) scenario runs, the run's conversation view shows the simulated caller's lines as audio + text, but every callee (agent under test) turn is audio only. Seen in production and in the dev proof for langwatch/langwatch#8117 (screenshot 02-run-messages.png there).
Why
- Twilio inbound audio carries no text; the callee
AudioSegment/message added to state.messages (src/execution/scenario-execution.ts ~1301 / ~2476) is audio-only, and that is what MESSAGE_ADDED → emitMessageSnapshot() sends to LangWatch.
- The user-simulator and judge each run the shared STT pre-pass (
src/voice/judge-stt.ts transcribeAudioMessages) right before their LLM call, producing a text part for the model input only (with a transcript cache). The text is never written back onto the state message.
backfillSegmentTranscripts() (~918-1032) runs after the run ends, mutates only the recording manifest, and exports only voice.stt.transcript_chars on the span.
So the callee transcript exists in-process (the simulator/judge saw it; it is visible in the _UserSimulatorAgent.call / _JudgeAgent.call span inputs) but never reaches the run's messages.
Proposed fix
Transcribe each callee turn once, right after the agent step resolves and before addMessage, attach the transcript as a sibling { type: "text" } part on the audio message (same shape the caller messages use), and let the simulator/judge pre-pass skip parts that already have a transcript. Net STT calls unchanged (the simulator already transcribes that audio on the next turn). Mirror in Python. Verify the LangWatch message renderer shows the text part next to an assistant audio part.
Related: #989 (turn-taking / call trace), #993.
🤖 Generated with Claude Code
Symptom
On phone-call (Twilio) scenario runs, the run's conversation view shows the simulated caller's lines as audio + text, but every callee (agent under test) turn is audio only. Seen in production and in the dev proof for langwatch/langwatch#8117 (screenshot
02-run-messages.pngthere).Why
AudioSegment/message added tostate.messages(src/execution/scenario-execution.ts~1301 / ~2476) is audio-only, and that is whatMESSAGE_ADDED→emitMessageSnapshot()sends to LangWatch.src/voice/judge-stt.tstranscribeAudioMessages) right before their LLM call, producing a text part for the model input only (with a transcript cache). The text is never written back onto the state message.backfillSegmentTranscripts()(~918-1032) runs after the run ends, mutates only the recording manifest, and exports onlyvoice.stt.transcript_charson the span.So the callee transcript exists in-process (the simulator/judge saw it; it is visible in the
_UserSimulatorAgent.call/_JudgeAgent.callspan inputs) but never reaches the run's messages.Proposed fix
Transcribe each callee turn once, right after the agent step resolves and before
addMessage, attach the transcript as a sibling{ type: "text" }part on the audio message (same shape the caller messages use), and let the simulator/judge pre-pass skip parts that already have a transcript. Net STT calls unchanged (the simulator already transcribes that audio on the next turn). Mirror in Python. Verify the LangWatch message renderer shows the text part next to an assistant audio part.Related: #989 (turn-taking / call trace), #993.
🤖 Generated with Claude Code