Add a WebRTC realtime voice provider on Android (full-duplex speech models) - #69
Open
s546126 wants to merge 1 commit into
Open
Add a WebRTC realtime voice provider on Android (full-duplex speech models)#69s546126 wants to merge 1 commit into
s546126 wants to merge 1 commit into
Conversation
Extract GeminiLiveService's public surface into a provider-neutral RealtimeVoiceService interface and add an OpenAI Realtime backend that runs over WebRTC: Opus audio with built-in echo cancellation and jitter buffering, model/tool events over the oai-events data channel, and the documented ephemeral-key flow (client_secrets + SDP exchange via /v1/realtime/calls). Providers that manage their own audio (WebRTC) bypass the PCM AudioManager pump entirely, which also sidesteps the known Gemini-Live vs WebRTC audio device conflict on that path. Vision is per-turn for OpenAI: the latest camera frame is attached as an image item when the server detects speech start, instead of a continuous 1fps stream. Tool responses are now built inside each provider (Gemini toolResponse JSON vs OpenAI function_call_output), so ToolCallRouter hands back (callId, name, result) instead of pre-shaped Gemini JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a WebRTC-based realtime voice path on Android, opening VisionClaw up to full-duplex speech-to-speech models beyond Gemini Live.
voice/RealtimeVoiceService— a provider-neutral interface extracted mechanically fromGeminiLiveService's existing public surface (connect/disconnect, audio, video frames, tool responses, the eight callbacks).GeminiLiveServiceimplements it unchanged in behavior.voice/openai/OpenAIRealtimeService— an OpenAI Realtime API backend over WebRTC, using the documented ephemeral-key flow (POST /v1/realtime/client_secrets, then SDP exchange withPOST /v1/realtime/calls). Model/tool events flow over theoai-eventsdata channel. Reuses thestream-webrtc-androiddependency already in the project.Secrets.ktchanges, so existing local setups keep compiling.Why WebRTC for voice
Full-duplex conversation lives or dies on barge-in, and that's where a raw PCM-over-WebSocket pump struggles: speaker audio leaks back into the mic, and the model interrupts itself. WebRTC brings echo cancellation, jitter buffering, packet-loss concealment, and Opus for free — the peer connection owns the mic and speaker end to end (
managesOwnAudio = true), soAudioManager's PCM pipeline stays idle on this path. A side benefit: this sidesteps the known Gemini-Live-vs-WebRTC audio device conflict for providers on this path.The interface also gives other full-duplex voice backends a contained place to land as additional
RealtimeVoiceServiceimplementations, without touching the session/tool-routing layers again.Design notes
ToolCallRouternow hands back(callId, name, result)and each provider shapes its own wire format (GeminitoolResponseJSON vs OpenAIfunction_call_output+response.create). The Gemini JSON building moved verbatim from the router intoGeminiLiveService.input_imageitem once per user turn, oninput_audio_buffer.speech_started.startSession().PeerConnection.close()blocks on observer callbacks returning).Testing
./gradlew assembleDebugpasses (DAT SDK 0.4.0 via GitHub Packages).🤖 Generated with Claude Code