Integration Request: Support for LiveKit (Real-time Voice & Avatar AI Agents)

[satya394]

Hi LangWatch team! 👋

I'd like to request an integration with LiveKit, specifically focusing on their Agents Framework which is becoming heavily used for building real-time Voice and Avatar AI applications.

🌟 The Use Case

As developers build real-time voice and video AI agents (using WebRTC via LiveKit to stream audio/video to/from LLMs and rendering engines), observability becomes critical and uniquely challenging. We need to track:

• End-to-End Latency: Tracking the time from user speech to AI audio/video response (VAD -> STT -> LLM -> TTS -> Avatar generation/rendering).
• Transcript Logging: Correlating the spoken interactions with the actual LLM prompts, responses, and triggered avatar actions or emotions.
• Cost & Token Tracking: Monitoring the usage of the underlying conversational models and avatar generation services during a live session.
• Evaluation: Assessing the quality, safety, and accuracy of the agent's vocal and visual responses.

🛠️ Proposed Integration

It would be incredibly powerful if LangWatch provided a drop-in tracer or plugin for the LiveKit Agents Python/Node SDK. This would allow developers to seamlessly log the STT (Speech-to-Text) inputs, the LLM reasoning/generation, the TTS (Text-to-Speech) outputs, and Avatar rendering outputs directly into the LangWatch dashboard as a single, unified trace.

🔗 Helpful Links

• LiveKit Agents Framework: https://docs.livekit.io/agents/overview/
• LiveKit GitHub: https://github.com/livekit/agents

Given the massive surge in multimodal AI applications, having first-class observability for LiveKit in LangWatch would be a huge win for the community. Let me know if there's any specific information I can provide to help look into this!

[musaabhasan]

LiveKit-style voice and avatar agents need a trace model that is more granular than ordinary chat observability. The critical path is not just prompt -> completion; it is VAD -> STT -> dialogue state -> retrieval/tools -> LLM -> TTS -> avatar/rendering -> client playback.

For a LangWatch integration, I would model each live session as a parent trace with synchronized spans for:

• audio input segment, VAD timing, and STT confidence;
• transcript text plus language/dialect metadata;
• LLM prompt, response, tool calls, and safety decisions;
• TTS latency, voice profile, and interruption/barge-in events;
• avatar action/render event if applicable;
• end-user playback latency and dropped/late frames.

Evaluation should also be multi-dimensional. Real-time agents can be factually correct but still poor if turn-taking is slow, interruptions are mishandled, the tone is inappropriate, or transcript confidence was low. It would help to expose metrics such as time-to-first-audio, end-to-end turn latency, transcript correction rate, interruption recovery, and safety escalation rate.

Privacy defaults matter here. Raw audio/video should not be logged by default; transcript, timestamps, and redacted metadata are usually enough for most debugging and compliance workflows.