feat: adapt live JPEG/video streaming to device rotation - #322
Conversation
Both live streaming sessions now react to a device rotation mid-stream instead of continuing to capture at the original dimensions: - JpegStreamSession rebuilds its ImageReader at the new size and points the VirtualDisplay's surface at it; each JPEG is self-describing, so no protocol change is needed. - VideoStreamSession rebuilds the MediaCodec encoder (whose input Surface size is fixed at configure()) and swaps the drain thread over to it; the existing INFO_OUTPUT_FORMAT_CHANGED handling emits a fresh CONFIG frame with the new SPS/PPS automatically, and the sequence counter is never reset across the swap. - StreamingSession gains a shared SizeChangeMonitor: MediaProjection's onCapturedContentResize on API 34+, a DisplayManager.DisplayListener fallback below that. - Only an exact swap of the session's original raw dimensions is treated as a rotation, so a foldable's screen switch or entering split-screen doesn't stretch the picture. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`cmd window fixed-to-user-rotation` (used to override the foreground app's own orientation lock) was added in API 30, so it hard-failed the rotation tests on the API 29 CI job with "Unknown command". Make the rotation trigger best-effort for that override and skip the test outright if the underlying `user-rotation lock` call itself doesn't work, rather than failing on a platform that can't force a rotation via adb shell at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dumpsys activity services can briefly flicker right after a JPEG stream service stops, so a single-shot re-check immediately after waitForCondition already confirmed !isRunning() could occasionally observe a stale "true" again - as seen on the API 32 CI job. The preceding waitForCondition call is already the real assertion (it throws its own timeout error), so the redundant check is just removed rather than papered over with a delay. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
KazuCocoa
left a comment
There was a problem hiding this comment.
Tested 54b050f on an Android 16 / API 36 Google APIs arm64 emulator using an APK built from this PR. All 9 existing JPEG/video E2E tests passed, and both streams survived 6 consecutive orientation changes with increasing sequence numbers (video emitted 7 CONFIG units including the initial one).
An additional rotation/disconnect stress test reproduced an app-process crash in 1 of 20 video trials; all 20 JPEG trials passed. The inline comment includes the concrete failure. The emulator successfully ran the overlapping encoder instances, so the resource-limit concern was not reproduced in this environment.
| @Override | ||
| public void onCapturedContentResize(int width, int height) { | ||
| super.onCapturedContentResize(width, height); | ||
| sizeChangeMonitor.onCapturedContentResize(width, height); |
There was a problem hiding this comment.
[P1] Coordinate resize callbacks with teardown to prevent a process crash
I reproduced an uncaught NullPointerException here on an Android 16 / API 36 arm64 emulator. releaseCaptureResources() sets sizeChangeMonitor = null on the session thread, while an already-queued onCapturedContentResize() can still run on the callback HandlerThread. Unregistering the callback and calling quitSafely() later does not protect this field access from a queued callback.
Reproduction: start an H.264 stream (1280x720, 15 fps, 1 Mbps), wait for output, run cmd window user-rotation lock 1, then immediately close the client transport. In 20 trials varying the post-command delay from 0 to 750 ms, the zero-delay first trial killed the app process:
FATAL EXCEPTION: video-stream-callback
java.lang.NullPointerException: Attempt to invoke virtual method
'void io.appium.settings.streaming.StreamingSession$SizeChangeMonitor.onCapturedContentResize(int, int)'
on a null object reference
at io.appium.settings.streaming.VideoStreamSession$1.onCapturedContentResize(VideoStreamSession.java:154)
Please serialize resize handling and resource teardown, or otherwise make the callback lifetime safe, and add a regression test for rotation concurrent with disconnect/stop. The JPEG callback has the same unguarded field access, although its 20 trials did not reproduce a crash. A null check alone also would not address the separate race between encoder/display reconfiguration and teardown.
There was a problem hiding this comment.
Good catch, and thanks for the precise repro/stack trace. Confirmed root cause: unregisterCallback()/quitSafely() don't retract a resize callback message already sitting in the callback handler thread's queue, so it can still run after releaseCaptureResources() has nulled sizeChangeMonitor (and, in the same window, could touch virtualDisplay/videoEncoder/imageReader concurrently with their teardown too).
Fixed in 5e7e7c5 by serializing rather than just null-checking: the whole teardown of the fields resize handling touches (sizeChangeMonitor, virtualDisplay, mediaProjectionCallback, videoEncoder/imageReader, JPEG's reused bitmaps) is now marshaled onto that same handler thread via a runOnHandlerAndWait() helper, so it can never interleave with a queued resize callback - both run on the one thread in FIFO order instead of racing across two. Kept a null check on the field access itself as cheap defense in depth. This applies to both VideoStreamSession and the equivalent unguarded access in JpegStreamSession.
Also added a regression test (should not crash the app when a rotation races a client disconnect in video-stream.spec.ts) reproducing your repro steps (rotate, then immediately disconnect the client) and asserting the app process isn't crash-restarted - passed 12/12 local trials on an API 36 emulator with the fix applied.
An already-queued MediaProjection.Callback#onCapturedContentResize could still run on the callback handler thread after releaseCaptureResources() had nulled sizeChangeMonitor from the session thread, throwing an uncaught NullPointerException that killed the app process. unregisterCallback()/quitSafely() don't protect against a message already in that thread's queue. Marshal the whole teardown of the fields resize handling touches (sizeChangeMonitor, virtualDisplay, mediaProjectionCallback, videoEncoder/imageReader, and JPEG's reused bitmaps) onto that same handler thread instead, so it can never interleave with a queued resize callback - both run on one thread in FIFO order rather than racing across two. A null check on the field access itself is kept as cheap defense in depth. Adds a regression test reproducing the reported repro (rotate then immediately disconnect the client) that asserts the app process doesn't get crash-restarted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
## [8.2.0](v8.1.0...v8.2.0) (2026-09-28) ### Features * adapt live JPEG/video streaming to device rotation ([#322](#322)) ([c537b49](c537b49))
|
🎉 This PR is included in version 8.2.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Both live streaming sessions now react to a device rotation mid-stream instead of continuing to capture at the original dimensions: