Skip to content

feat: camera device with WebRTC live view - #6

Merged
kakopappa merged 3 commits into
mainfrom
feat/camera-webrtc-live-view
Sep 16, 2026
Merged

kakopappa merged 3 commits into
mainfrom
feat/camera-webrtc-live-view

Conversation

@kakopappa

@kakopappa kakopappa commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a camera device with WebRTC live view in the SinricPro portal and app, plus an example that streams an ESP32 or ESP32-S3 camera over a WebRTC DataChannel. The signaling contract matches the Arduino SDK, so the existing portal and app viewers work unchanged.

It also fixes a core websocket bug that affects every device type.

The websocket fix

esp_websocket_client delivers a frame larger than its 2 KB buffer as several WEBSOCKET_EVENT_DATA events, each carrying the frame's total length and its own offset. The handler parsed every piece as a complete JSON message, so any server message above 2 KB was silently lost. A camera offer (base64 SDP plus TURN credentials) is several KB, so live view could never have worked without this.

sinricpro_frame_assembler now reassembles the pieces, up to CONFIG_SINRICPRO_MAX_MESSAGE_SIZE (16 KB). Oversized or out-of-sequence messages are dropped whole and logged once. Ping, pong and close frames are no longer passed to the JSON parser.

Changes

Area Change
Core sinricpro_frame_assembler; sinricpro_set_response_message(); Kconfig SINRICPRO_MAX_MESSAGE_SIZE
Camera device sinricpro_camera with PowerState and a CameraController for getCameraCapabilities and getWebRTCAnswer
Example examples/camera with a webrtc_camera component: session task, JPEG fragments over a DataChannel, viewer controls, automatic quality, XIAO Sense microphone
Tests Host tests for reassembly and the camera signaling contract; base64 decode and esp_event.h shims
CI build-camera job for esp32 and esp32s3 on ESP-IDF v5.5 and v6.1
Docs README, API reference, CHANGELOG, example README

The SinricPro component itself gains no dependencies. esp_peer and esp32-camera are required only by the example's webrtc_camera component, which can be copied into other projects.

Testing

  • test/host/run.sh: signing (23 checks), frame reassembly (14), camera signaling (18), all passing
  • examples/switch rebuilt with the core changes
  • examples/camera built for esp32 (1.38 MB) and esp32s3 (1.33 MB) on ESP-IDF v6.1, with no warnings in the new sources
  • On hardware, an AI-Thinker ESP32-CAM viewed from the portal on the test server: offer to answer in 1.9 s, DataChannel open 4.6 s after the offer, and streaming at RSSI -78 to -85 dBm with a stable internal heap

Notes for reviewers

  • The WebRTC offer callback blocks for up to about 5 s while ICE gathers, unlike other callbacks, because signaling has no trickle ICE and the answer must be complete when it is returned.
  • esp32-camera 2.1.5 and later require esp_hal_clock on IDF 6.x. ESP-IDF master snapshots from before that component existed fail at configure; released IDF versions are fine.
  • The camera example needs ESP-IDF 5.5 or later. esp_peer ships a prebuilt libpeer_default.a that calls esp_log(), which ESP-IDF added in 5.5, so it cannot link on older releases. CI builds the example on v5.5 (the floor) and v6.1. The SinricPro component itself keeps its existing IDF support.
  • Hardware coverage so far is one board (AI-Thinker). Remote viewing through the TURN relay, a long soak, the viewer controls, and the XIAO S3 microphone are not yet exercised on hardware.
  • Alexa and Google Home streaming are out of scope; they need a native H.264 track.

Adds a camera device that answers the SinricPro portal and app WebRTC
signaling, and an example that streams JPEG frames over a DataChannel.

- sinricpro_camera: getCameraCapabilities reports webrtc and webrtcAudio;
  getWebRTCAnswer decodes the base64 offer, flattens iceServers[].urls and
  returns a base64 answer. The component gains no new dependencies.
- examples/camera: a webrtc_camera component built on esp_peer and
  esp32-camera, with resolution, frame rate, flash, flip and mirror controls,
  automatic quality, and the XIAO ESP32S3 Sense microphone. Ten board
  profiles are selectable in menuconfig.
- sinricpro_set_response_message() lets a callback tell the client why a
  request failed.
- Kconfig SINRICPRO_MAX_MESSAGE_SIZE (default 16 KB).

Fixes a core bug: a server message larger than the websocket client's 2 KB
buffer is delivered as several data events, and each piece was parsed as a
complete message, so the message was lost. Pieces are now reassembled, and
ping, pong and close frames are no longer handed to the JSON parser.

Tested: host tests for signing, reassembly and camera signaling; the switch
example rebuilt; the camera example built for esp32 and esp32s3 on IDF v6.1;
live view on an AI-Thinker ESP32-CAM through the SinricPro portal.
…-IDF 5.x

esp_peer compiles its TLS server transport unconditionally, but ESP-IDF 5.x
declares esp_tls_cfg_server_t and esp_tls_server_session_create only when
CONFIG_ESP_TLS_SERVER is set, so the example failed to compile on 5.x. ESP-IDF
6.x declares the API unconditionally and ignores the option.
…gainst

esp_peer ships a prebuilt libpeer_default.a that calls esp_log(), which
ESP-IDF added in 5.5, so the camera example cannot link on older releases.
The example previously claimed 5.1 or later and CI built it on the 5.1 image.

- Build the camera example in CI on ESP-IDF v5.5, the floor, and v6.1.
- State the 5.5 minimum in both component manifests, the example README,
  the README and the CHANGELOG.
- Drop CONFIG_ESP_TLS_SERVER: from 5.3 the esp-tls server API is always
  declared, so the option only helped releases esp_peer cannot link on.
- Drop the webrtc_camera fallback to the pre-5.3 driver component.
@kakopappa
kakopappa merged commit 1d752a8 into main Sep 16, 2026
76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant