Skip to content

Add a WebRTC realtime voice provider on Android (full-duplex speech models) - #69

Open
s546126 wants to merge 1 commit into
Intent-Lab:mainfrom
s546126:feat/webrtc-realtime-voice-provider
Open

Add a WebRTC realtime voice provider on Android (full-duplex speech models)#69
s546126 wants to merge 1 commit into
Intent-Lab:mainfrom
s546126:feat/webrtc-realtime-voice-provider

Conversation

@s546126

@s546126 s546126 commented Aug 10, 2026

Copy link
Copy Markdown

Summary

Adds a WebRTC-based realtime voice path on Android, opening VisionClaw up to full-duplex speech-to-speech models beyond Gemini Live.

  • voice/RealtimeVoiceService — a provider-neutral interface extracted mechanically from GeminiLiveService's existing public surface (connect/disconnect, audio, video frames, tool responses, the eight callbacks). GeminiLiveService implements it unchanged in behavior.
  • voice/openai/OpenAIRealtimeService — an OpenAI Realtime API backend over WebRTC, using the documented ephemeral-key flow (POST /v1/realtime/client_secrets, then SDP exchange with POST /v1/realtime/calls). Model/tool events flow over the oai-events data channel. Reuses the stream-webrtc-android dependency already in the project.
  • Settings — a Voice Provider toggle (Gemini Live remains the default) plus an OpenAI API key field. Stored in SharedPreferences only; no Secrets.kt changes, so existing local setups keep compiling.

Why WebRTC for voice

Full-duplex conversation lives or dies on barge-in, and that's where a raw PCM-over-WebSocket pump struggles: speaker audio leaks back into the mic, and the model interrupts itself. WebRTC brings echo cancellation, jitter buffering, packet-loss concealment, and Opus for free — the peer connection owns the mic and speaker end to end (managesOwnAudio = true), so AudioManager's PCM pipeline stays idle on this path. A side benefit: this sidesteps the known Gemini-Live-vs-WebRTC audio device conflict for providers on this path.

The interface also gives other full-duplex voice backends a contained place to land as additional RealtimeVoiceService implementations, without touching the session/tool-routing layers again.

Design notes

  • Tool calls: ToolCallRouter now hands back (callId, name, result) and each provider shapes its own wire format (Gemini toolResponse JSON vs OpenAI function_call_output + response.create). The Gemini JSON building moved verbatim from the router into GeminiLiveService.
  • Vision: Gemini takes a continuous ~1fps frame stream; pushing that through Realtime conversation items would flood context. The OpenAI path instead caches the latest camera frame and attaches it as an input_image item once per user turn, on input_audio_buffer.speech_started.
  • Provider switch takes effect on the next session — the service is re-created in startSession().
  • Teardown from WebRTC observer callbacks hops to a separate thread (PeerConnection.close() blocks on observer callbacks returning).

Testing

  • Compile-verified: ./gradlew assembleDebug passes (DAT SDK 0.4.0 via GitHub Packages).
  • Not yet exercised against live glasses hardware; happy to iterate on review feedback.
  • iOS is untouched; the same interface extraction would be the follow-up there.

🤖 Generated with Claude Code

Extract GeminiLiveService's public surface into a provider-neutral
RealtimeVoiceService interface and add an OpenAI Realtime backend that
runs over WebRTC: Opus audio with built-in echo cancellation and jitter
buffering, model/tool events over the oai-events data channel, and the
documented ephemeral-key flow (client_secrets + SDP exchange via
/v1/realtime/calls).

Providers that manage their own audio (WebRTC) bypass the PCM
AudioManager pump entirely, which also sidesteps the known Gemini-Live
vs WebRTC audio device conflict on that path. Vision is per-turn for
OpenAI: the latest camera frame is attached as an image item when the
server detects speech start, instead of a continuous 1fps stream.

Tool responses are now built inside each provider (Gemini toolResponse
JSON vs OpenAI function_call_output), so ToolCallRouter hands back
(callId, name, result) instead of pre-shaped Gemini JSON.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant