feat: camera device with WebRTC live view - #6
Merged
Merged
Conversation
Adds a camera device that answers the SinricPro portal and app WebRTC signaling, and an example that streams JPEG frames over a DataChannel. - sinricpro_camera: getCameraCapabilities reports webrtc and webrtcAudio; getWebRTCAnswer decodes the base64 offer, flattens iceServers[].urls and returns a base64 answer. The component gains no new dependencies. - examples/camera: a webrtc_camera component built on esp_peer and esp32-camera, with resolution, frame rate, flash, flip and mirror controls, automatic quality, and the XIAO ESP32S3 Sense microphone. Ten board profiles are selectable in menuconfig. - sinricpro_set_response_message() lets a callback tell the client why a request failed. - Kconfig SINRICPRO_MAX_MESSAGE_SIZE (default 16 KB). Fixes a core bug: a server message larger than the websocket client's 2 KB buffer is delivered as several data events, and each piece was parsed as a complete message, so the message was lost. Pieces are now reassembled, and ping, pong and close frames are no longer handed to the JSON parser. Tested: host tests for signing, reassembly and camera signaling; the switch example rebuilt; the camera example built for esp32 and esp32s3 on IDF v6.1; live view on an AI-Thinker ESP32-CAM through the SinricPro portal.
…-IDF 5.x esp_peer compiles its TLS server transport unconditionally, but ESP-IDF 5.x declares esp_tls_cfg_server_t and esp_tls_server_session_create only when CONFIG_ESP_TLS_SERVER is set, so the example failed to compile on 5.x. ESP-IDF 6.x declares the API unconditionally and ignores the option.
…gainst esp_peer ships a prebuilt libpeer_default.a that calls esp_log(), which ESP-IDF added in 5.5, so the camera example cannot link on older releases. The example previously claimed 5.1 or later and CI built it on the 5.1 image. - Build the camera example in CI on ESP-IDF v5.5, the floor, and v6.1. - State the 5.5 minimum in both component manifests, the example README, the README and the CHANGELOG. - Drop CONFIG_ESP_TLS_SERVER: from 5.3 the esp-tls server API is always declared, so the option only helped releases esp_peer cannot link on. - Drop the webrtc_camera fallback to the pre-5.3 driver component.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a camera device with WebRTC live view in the SinricPro portal and app, plus an example that streams an ESP32 or ESP32-S3 camera over a WebRTC DataChannel. The signaling contract matches the Arduino SDK, so the existing portal and app viewers work unchanged.
It also fixes a core websocket bug that affects every device type.
The websocket fix
esp_websocket_clientdelivers a frame larger than its 2 KB buffer as severalWEBSOCKET_EVENT_DATAevents, each carrying the frame's total length and its own offset. The handler parsed every piece as a complete JSON message, so any server message above 2 KB was silently lost. A camera offer (base64 SDP plus TURN credentials) is several KB, so live view could never have worked without this.sinricpro_frame_assemblernow reassembles the pieces, up toCONFIG_SINRICPRO_MAX_MESSAGE_SIZE(16 KB). Oversized or out-of-sequence messages are dropped whole and logged once. Ping, pong and close frames are no longer passed to the JSON parser.Changes
sinricpro_frame_assembler;sinricpro_set_response_message(); KconfigSINRICPRO_MAX_MESSAGE_SIZEsinricpro_camerawith PowerState and aCameraControllerforgetCameraCapabilitiesandgetWebRTCAnswerexamples/camerawith awebrtc_cameracomponent: session task, JPEG fragments over a DataChannel, viewer controls, automatic quality, XIAO Sense microphoneesp_event.hshimsbuild-camerajob for esp32 and esp32s3 on ESP-IDF v5.5 and v6.1The SinricPro component itself gains no dependencies.
esp_peerandesp32-cameraare required only by the example'swebrtc_cameracomponent, which can be copied into other projects.Testing
test/host/run.sh: signing (23 checks), frame reassembly (14), camera signaling (18), all passingexamples/switchrebuilt with the core changesexamples/camerabuilt for esp32 (1.38 MB) and esp32s3 (1.33 MB) on ESP-IDF v6.1, with no warnings in the new sourcesNotes for reviewers
esp32-camera2.1.5 and later requireesp_hal_clockon IDF 6.x. ESP-IDF master snapshots from before that component existed fail at configure; released IDF versions are fine.esp_peerships a prebuiltlibpeer_default.athat callsesp_log(), which ESP-IDF added in 5.5, so it cannot link on older releases. CI builds the example on v5.5 (the floor) and v6.1. The SinricPro component itself keeps its existing IDF support.