feat: camera device with WebRTC live view

.github/workflows/build-test.yml

@@ -57,6 +57,37 @@ jobs:
           . $IDF_PATH/export.sh
           idf.py size
 
+  # Camera live view. The example's board profiles cover ESP32 and ESP32-S3 only,
+  # so it cannot join the main matrix's S2 and C3 targets. esp_peer ships a
+  # prebuilt library that links only against ESP-IDF 5.5 or later, so 5.5 is the
+  # floor this job proves, alongside the current release.
+  build-camera:
+    runs-on: ubuntu-latest
+    container:
+      image: espressif/idf:${{ matrix.idf-version }}
+    strategy:
+      fail-fast: false
+      matrix:
+        idf-version: [v5.5, v6.1]
+        idf-target: [esp32, esp32s3]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
+
+      - name: Clean managed components
+        run: |
+          find . -type d -name "managed_components" -exec rm -rf {} + || true
+
+      - name: Build camera for ${{ matrix.idf-target }} on ${{ matrix.idf-version }}
+        working-directory: examples/camera
+        run: |
+          . $IDF_PATH/export.sh
+          idf.py set-target ${{ matrix.idf-target }}
+          idf.py build
+
   # Local control without the mDNS announcement. Keeps the UDP listener but
   # drops the mdns component, which is the configuration for a board whose app
   # partition cannot take it. A gate that is never built is a gate that breaks.

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## [Unreleased]
+
+### Features
+
+- feat: camera device with WebRTC live view in the SinricPro portal and app.
+  `sinricpro_camera_on_webrtc_offer()` answers `getWebRTCAnswer` using the ICE
+  servers the server sends, and `getCameraCapabilities` reports `webrtc` and
+  `webrtcAudio` so viewers know what the firmware supports.
+- feat: `examples/camera` streams JPEG frames over a WebRTC DataChannel through a
+  `webrtc_camera` component built on `esp_peer`, with resolution, frame rate,
+  flash, flip and mirror controls, automatic quality, and the XIAO ESP32S3 Sense
+  microphone. The SinricPro component itself gains no dependencies. The example
+  needs ESP-IDF 5.5 or later, because `esp_peer`'s prebuilt library links only
+  against 5.5 and newer.
+- feat: `sinricpro_set_response_message()`, so a callback can tell the client why
+  a request failed.
+- feat: Kconfig `SINRICPRO_MAX_MESSAGE_SIZE` (default 16 KB).
+
+### Fixes
+
+- fix: a server message larger than the websocket client's 2 KB buffer is posted
+  as several data events, and each piece was parsed as a complete message, so the
+  message was lost. The pieces are now reassembled.
+- fix: ping, pong and close frames are no longer handed to the JSON parser.
+
 ## [1.3.0]
 
 ### Features

CMakeLists.txt

@@ -23,6 +23,7 @@ idf_component_register(
         "src/core/sinricpro_event_limiter.c"
         "src/core/sinricpro_udp.c"
         "src/core/sinricpro_mdns.c"
+        "src/core/sinricpro_frame_assembler.c"
         "src/devices/sinricpro_switch.c"
         "src/devices/sinricpro_motion_sensor.c"
         "src/devices/sinricpro_contact_sensor.c"
@@ -39,6 +40,8 @@ idf_component_register(
         "src/devices/sinricpro_windowac.c"
         "src/devices/sinricpro_tv.c"
         "src/devices/sinricpro_speaker.c"
+        "src/devices/sinricpro_camera.c"
+        "src/capabilities/camera_controller.c"
         "src/capabilities/power_state_controller.c"
         "src/capabilities/setting_controller.c"
         "src/capabilities/push_notification.c"

Kconfig

@@ -69,6 +69,16 @@ menu "SinricPro Configuration"
             Maximum number of messages that can be queued
             for sending.
 
+    config SINRICPRO_MAX_MESSAGE_SIZE
+        int "Maximum incoming message size (bytes)"
+        default 16384
+        range 2048 65536
+        help
+            Largest server message the SDK reassembles. The websocket client
+            delivers frames in 2 KB pieces; a message above this limit is
+            dropped whole rather than parsed in fragments. A camera WebRTC
+            offer carries SDP plus TURN credentials and needs several KB.
+
     config SINRICPRO_AUTO_RECONNECT
         bool "Enable auto-reconnection"
         default y

README.md

@@ -13,6 +13,7 @@ Official ESP-IDF component for [SinricPro](https://sinric.pro) - Control your ES
 - ✅ **Voice Control** - Works with Alexa and Google Home
 - ✅ **Real-time** - WebSocket-based bidirectional communication
 - ✅ **Local Control** - Answers the app over the LAN when the cloud is down
+- ✅ **Camera Live View** - WebRTC streaming to the SinricPro portal and app
 - ✅ **Secure** - HMAC-SHA256 message signatures
 - ✅ **Reliable** - Auto-reconnection and heartbeat monitoring
 - ✅ **Event-driven** - ESP event loop integration
@@ -49,6 +50,9 @@ All devices below have complete API support and working examples:
 - ✅ **TV** - Volume, mute, media control, input selection, channels
 - ✅ **Speaker** - Volume, mute, media control, equalizer, modes
 
+### Cameras
+- ✅ **Camera** - WebRTC live view in the SinricPro portal and app, with remote resolution, frame rate and flash control (example needs ESP-IDF 5.5+)
+
 ### Additional Devices (API Only)
 - ✅ Air Quality Sensor - PM1, PM2.5, PM10 measurements
 - ✅ Power Sensor - Voltage, current, power monitoring
@@ -125,6 +129,7 @@ The component includes **13 complete working examples** demonstrating all device
 | [Blinds](examples/blinds/) | Motorized blinds/curtains | ⭐⭐⭐ Complex | DC motor, L298N |
 | [TV](examples/tv/) | Media control & channels | ⭐⭐⭐ Complex | Simulated/IR |
 | [Speaker](examples/speaker/) | Audio control & equalizer | ⭐⭐⭐ Complex | Simulated |
+| [Camera](examples/camera/) | WebRTC live view in the portal and app | ⭐⭐⭐ Complex | ESP32/ESP32-S3 camera with PSRAM |
 
 Each example includes:
 - Complete working code

docs/api-reference.md

@@ -13,6 +13,10 @@ esp_err_t sinricpro_deinit(void);
 bool sinricpro_is_connected(void);
 uint32_t sinricpro_get_timestamp(void);
 const char* sinricpro_get_version(void);
+
+/* From inside a device callback: replace the response's "OK" /
+ * "Device did not handle request" text with a reason the client can show. */
+esp_err_t sinricpro_set_response_message(const char *message);
 ```
 
 #### Configuration Structure
@@ -87,6 +91,46 @@ esp_err_t sinricpro_switch_send_notification(
 );
 ```
 
+### Camera Device API
+
+Live view in the SinricPro portal and app over WebRTC. The component handles
+signaling; the peer connection and streaming live in the `webrtc_camera`
+component of [examples/camera](../examples/camera/).
+
+```c
+sinricpro_device_handle_t sinricpro_camera_create(const char *device_id);
+esp_err_t sinricpro_camera_delete(sinricpro_device_handle_t device);
+
+esp_err_t sinricpro_camera_on_power_state(sinricpro_device_handle_t device,
+                                          sinricpro_camera_power_state_callback_t callback,
+                                          void *user_data);
+
+/* One entry per URL; strings are valid only during the callback. */
+typedef struct {
+    const char *url;         /* "stun:…", "turn:…?transport=udp", "turns:…:443?transport=tcp" */
+    const char *username;    /* "" when none */
+    const char *credential;  /* "" when none */
+} sinricpro_ice_server_t;
+
+/* Set *answer_sdp to a malloc()ed SDP answer containing every local candidate;
+ * the SDK frees it. May block while ICE gathers (up to ~5 s). */
+typedef bool (*sinricpro_camera_webrtc_offer_callback_t)(
+    const char *device_id, const char *offer_sdp,
+    const sinricpro_ice_server_t *ice_servers, size_t ice_server_count,
+    char **answer_sdp, void *user_data);
+
+/* Registering the callback makes getCameraCapabilities report webrtc: true. */
+esp_err_t sinricpro_camera_on_webrtc_offer(sinricpro_device_handle_t device,
+                                           sinricpro_camera_webrtc_offer_callback_t callback,
+                                           void *user_data);
+
+/* Reported as webrtcAudio, so viewers request an audio track. */
+esp_err_t sinricpro_camera_enable_webrtc_audio(sinricpro_device_handle_t device, bool enabled);
+
+esp_err_t sinricpro_camera_send_power_state_event(sinricpro_device_handle_t device,
+                                                  bool state, const char *cause);
+```
+
 ### Event System
 
 ```c
@@ -122,6 +166,7 @@ Access via `idf.py menuconfig` → `Component config` → `SinricPro Configurati
 - **Enable Debug Logging** - Verbose logging for troubleshooting
 - **Event Queue Size** - Maximum queued events
 - **Message Queue Size** - Maximum queued messages
+- **Maximum Incoming Message Size** - Largest reassembled server message (default 16 KB; camera offers need several KB)
 - **Auto-reconnection** - Enable/disable auto-reconnection
 - **Reconnection Interval** - Time between reconnection attempts
 - **Max Devices** - Maximum number of registered devices

examples/camera/CMakeLists.txt

@@ -0,0 +1,9 @@
+# The following lines of boilerplate have to be in your project's CMakeLists
+# in this exact order for cmake to work correctly
+cmake_minimum_required(VERSION 3.16)
+
+# Add the parent components directory so we can find sinricpro component
+set(EXTRA_COMPONENT_DIRS "${CMAKE_CURRENT_LIST_DIR}/../..") # Use local component
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+project(camera_example)

examples/camera/README.md

@@ -0,0 +1,82 @@
+# Camera Example (WebRTC live view)
+
+Streams an ESP32 or ESP32-S3 camera to the SinricPro portal and app, from anywhere. Video is JPEG over an encrypted WebRTC DataChannel; signaling runs through the SinricPro connection, and STUN/TURN servers arrive with each viewer's offer, so viewing works outside your LAN. Viewers can change resolution and frame rate and toggle flash, flip and mirror, and quality drops automatically on slow links. XIAO ESP32S3 Sense also streams its onboard microphone.
+
+## Requirements
+
+- ESP-IDF 5.5 or later. `esp_peer` includes a prebuilt library that links only against 5.5 and newer.
+- An ESP32 or ESP32-S3 camera board **with PSRAM**
+- 4 MB of flash or more
+- A Wi-Fi signal of **−75 dBm or better** at the board. Below about −80 dBm the Wi-Fi driver's transmit buffers stop recycling fast enough and the DTLS handshake cannot complete, even though free heap looks healthy.
+
+## Portal setup
+
+1. Create a device of type **Camera**.
+2. In **Camera Stream Configuration**, set Board to **ESP32** and Streaming Protocol to **WebRTC**.
+3. Copy the device ID, app key and app secret.
+
+## Configure and build
+
+Edit the credentials at the top of [main/camera_example.c](main/camera_example.c):
+
+```c
+#define WIFI_SSID      "WIFI_SSID"
+#define WIFI_PASS      "WIFI_PASS"
+#define DEVICE_ID      "DEVICE_ID"
+#define APP_KEY        "APP_KEY"
+#define APP_SECRET     "APP_SECRET"
+```
+
+Select the target, then your board under **SinricPro Camera Example → Camera board**:
+
+```bash
+cd examples/camera
+idf.py set-target esp32        # or esp32s3
+idf.py menuconfig
+idf.py build flash monitor
+```
+
+| Board | Target |
+| --- | --- |
+| AI-Thinker ESP32-CAM (default on esp32) | esp32 |
+| ESP-EYE, M5Camera A/B, ESP-WROVER-KIT, LILYGO T-Camera | esp32 |
+| XIAO ESP32S3 Sense (default on esp32s3) | esp32s3 |
+| Freenove ESP32-S3, ESP32-S3 WROOM (PWDN 38), GOOUUU ESP32-S3 | esp32s3 |
+
+`sdkconfig.defaults.esp32s3` assumes octal PSRAM, as on the XIAO Sense and Freenove N8R8. For a board with quad PSRAM, set `CONFIG_SPIRAM_MODE_QUAD` instead.
+
+Open **Preview** on the camera in the portal, or tap the camera in the app.
+
+## How it works
+
+| Piece | Where |
+| --- | --- |
+| `getCameraCapabilities`, `getWebRTCAnswer` | SinricPro component: `sinricpro_camera_on_webrtc_offer()` |
+| Peer connection, JPEG streaming, viewer controls | [components/webrtc_camera](components/webrtc_camera/), built on Espressif's `esp_peer` |
+| Pin mappings | [main/camera_boards.c](main/camera_boards.c) |
+
+The SinricPro component itself does not depend on `esp_peer` or `esp32-camera`; the `webrtc_camera` component can be copied into your own project.
+
+Signaling is a single offer/answer exchange without trickle ICE, so the session gathers every local candidate before answering, and the answer callback blocks for up to about 5 seconds. One viewer is served at a time; a new offer replaces the current viewer.
+
+## Memory settings
+
+[sdkconfig.defaults](sdkconfig.defaults) moves Wi-Fi, lwIP and mbedTLS allocations into PSRAM. On classic ESP32 that is what leaves enough contiguous internal RAM for the Wi-Fi driver's transmit buffers once the TLS connection to SinricPro is open. The example also keeps the DataChannel caches small on classic ESP32 for the same reason.
+
+`CONFIG_FREERTOS_HZ=1000` matters too: the session sends one DataChannel fragment per tick, so at the default 100 Hz throughput is capped at about 100 kB/s.
+
+## Troubleshooting
+
+| Symptom | Check |
+| --- | --- |
+| Viewer shows one frozen frame, device logs `streaming: yes` | Wi-Fi signal; the log prints RSSI at connect and every 30 s |
+| `dtlsState=connecting` in `chrome://webrtc-internals` | Wi-Fi signal and free internal heap in the 30-second log line |
+| Viewer reports "firmware does not support live view" | The WebRTC offer callback is not registered |
+| `Camera init failed` | Board selection in menuconfig, ribbon cable, and PSRAM mode |
+| `PSRAM is not available` | `CONFIG_SPIRAM`, and octal versus quad PSRAM on ESP32-S3 |
+
+## Limits
+
+- Portal and app only. Alexa and Google Home need a native H.264 video track.
+- WebRTC signaling needs the cloud connection: local control's UDP transport cannot carry an offer.
+- Snapshot and motion upload are not implemented yet.

examples/camera/components/webrtc_camera/CMakeLists.txt

@@ -0,0 +1,16 @@
+idf_component_register(
+    SRCS
+        "webrtc_camera.c"
+        "jpeg_streamer.c"
+        "camera_controls.c"
+    INCLUDE_DIRS
+        "include"
+    REQUIRES
+        esp_peer
+        esp32-camera
+    PRIV_REQUIRES
+        cjson
+        esp_driver_gpio
+        esp_timer
+        esp_wifi
+)