Skip to content
This repository was archived by the owner on Oct 2, 2026. It is now read-only.

feat: adapt live JPEG/video streaming to device rotation - #322

Merged
mykola-mokhnach merged 4 commits into
masterfrom
landscape
Sep 28, 2026
Merged

mykola-mokhnach merged 4 commits into
masterfrom
landscape

Conversation

@mykola-mokhnach

Copy link
Copy Markdown
Contributor

Both live streaming sessions now react to a device rotation mid-stream instead of continuing to capture at the original dimensions:

  • JpegStreamSession rebuilds its ImageReader at the new size and points the VirtualDisplay's surface at it; each JPEG is self-describing, so no protocol change is needed.
  • VideoStreamSession rebuilds the MediaCodec encoder (whose input Surface size is fixed at configure()) and swaps the drain thread over to it; the existing INFO_OUTPUT_FORMAT_CHANGED handling emits a fresh CONFIG frame with the new SPS/PPS automatically, and the sequence counter is never reset across the swap.
  • StreamingSession gains a shared SizeChangeMonitor: MediaProjection's onCapturedContentResize on API 34+, a DisplayManager.DisplayListener fallback below that.
  • Only an exact swap of the session's original raw dimensions is treated as a rotation, so a foldable's screen switch or entering split-screen doesn't stretch the picture.

Both live streaming sessions now react to a device rotation mid-stream
instead of continuing to capture at the original dimensions:

- JpegStreamSession rebuilds its ImageReader at the new size and points
  the VirtualDisplay's surface at it; each JPEG is self-describing, so
  no protocol change is needed.
- VideoStreamSession rebuilds the MediaCodec encoder (whose input
  Surface size is fixed at configure()) and swaps the drain thread over
  to it; the existing INFO_OUTPUT_FORMAT_CHANGED handling emits a fresh
  CONFIG frame with the new SPS/PPS automatically, and the sequence
  counter is never reset across the swap.
- StreamingSession gains a shared SizeChangeMonitor: MediaProjection's
  onCapturedContentResize on API 34+, a DisplayManager.DisplayListener
  fallback below that.
- Only an exact swap of the session's original raw dimensions is
  treated as a rotation, so a foldable's screen switch or entering
  split-screen doesn't stretch the picture.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mykola-mokhnach and others added 2 commits September 27, 2026 22:29
`cmd window fixed-to-user-rotation` (used to override the foreground
app's own orientation lock) was added in API 30, so it hard-failed the
rotation tests on the API 29 CI job with "Unknown command". Make the
rotation trigger best-effort for that override and skip the test
outright if the underlying `user-rotation lock` call itself doesn't
work, rather than failing on a platform that can't force a rotation via
adb shell at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dumpsys activity services can briefly flicker right after a JPEG stream
service stops, so a single-shot re-check immediately after
waitForCondition already confirmed !isRunning() could occasionally
observe a stale "true" again - as seen on the API 32 CI job. The
preceding waitForCondition call is already the real assertion (it
throws its own timeout error), so the redundant check is just removed
rather than papered over with a delay.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@KazuCocoa KazuCocoa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested 54b050f on an Android 16 / API 36 Google APIs arm64 emulator using an APK built from this PR. All 9 existing JPEG/video E2E tests passed, and both streams survived 6 consecutive orientation changes with increasing sequence numbers (video emitted 7 CONFIG units including the initial one).

An additional rotation/disconnect stress test reproduced an app-process crash in 1 of 20 video trials; all 20 JPEG trials passed. The inline comment includes the concrete failure. The emulator successfully ran the overlapping encoder instances, so the resource-limit concern was not reproduced in this environment.

@Override
public void onCapturedContentResize(int width, int height) {
super.onCapturedContentResize(width, height);
sizeChangeMonitor.onCapturedContentResize(width, height);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Coordinate resize callbacks with teardown to prevent a process crash

I reproduced an uncaught NullPointerException here on an Android 16 / API 36 arm64 emulator. releaseCaptureResources() sets sizeChangeMonitor = null on the session thread, while an already-queued onCapturedContentResize() can still run on the callback HandlerThread. Unregistering the callback and calling quitSafely() later does not protect this field access from a queued callback.

Reproduction: start an H.264 stream (1280x720, 15 fps, 1 Mbps), wait for output, run cmd window user-rotation lock 1, then immediately close the client transport. In 20 trials varying the post-command delay from 0 to 750 ms, the zero-delay first trial killed the app process:

FATAL EXCEPTION: video-stream-callback
java.lang.NullPointerException: Attempt to invoke virtual method
'void io.appium.settings.streaming.StreamingSession$SizeChangeMonitor.onCapturedContentResize(int, int)'
on a null object reference
    at io.appium.settings.streaming.VideoStreamSession$1.onCapturedContentResize(VideoStreamSession.java:154)

Please serialize resize handling and resource teardown, or otherwise make the callback lifetime safe, and add a regression test for rotation concurrent with disconnect/stop. The JPEG callback has the same unguarded field access, although its 20 trials did not reproduce a crash. A null check alone also would not address the separate race between encoder/display reconfiguration and teardown.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and thanks for the precise repro/stack trace. Confirmed root cause: unregisterCallback()/quitSafely() don't retract a resize callback message already sitting in the callback handler thread's queue, so it can still run after releaseCaptureResources() has nulled sizeChangeMonitor (and, in the same window, could touch virtualDisplay/videoEncoder/imageReader concurrently with their teardown too).

Fixed in 5e7e7c5 by serializing rather than just null-checking: the whole teardown of the fields resize handling touches (sizeChangeMonitor, virtualDisplay, mediaProjectionCallback, videoEncoder/imageReader, JPEG's reused bitmaps) is now marshaled onto that same handler thread via a runOnHandlerAndWait() helper, so it can never interleave with a queued resize callback - both run on the one thread in FIFO order instead of racing across two. Kept a null check on the field access itself as cheap defense in depth. This applies to both VideoStreamSession and the equivalent unguarded access in JpegStreamSession.

Also added a regression test (should not crash the app when a rotation races a client disconnect in video-stream.spec.ts) reproducing your repro steps (rotate, then immediately disconnect the client) and asserting the app process isn't crash-restarted - passed 12/12 local trials on an API 36 emulator with the fix applied.

An already-queued MediaProjection.Callback#onCapturedContentResize
could still run on the callback handler thread after
releaseCaptureResources() had nulled sizeChangeMonitor from the session
thread, throwing an uncaught NullPointerException that killed the app
process. unregisterCallback()/quitSafely() don't protect against a
message already in that thread's queue.

Marshal the whole teardown of the fields resize handling touches
(sizeChangeMonitor, virtualDisplay, mediaProjectionCallback,
videoEncoder/imageReader, and JPEG's reused bitmaps) onto that same
handler thread instead, so it can never interleave with a queued resize
callback - both run on one thread in FIFO order rather than racing
across two. A null check on the field access itself is kept as cheap
defense in depth.

Adds a regression test reproducing the reported repro (rotate then
immediately disconnect the client) that asserts the app process
doesn't get crash-restarted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mykola-mokhnach
mykola-mokhnach merged commit c537b49 into master Sep 28, 2026
13 of 14 checks passed
@mykola-mokhnach
mykola-mokhnach deleted the landscape branch September 28, 2026 06:42
github-actions Bot pushed a commit that referenced this pull request Sep 28, 2026
## [8.2.0](v8.1.0...v8.2.0) (2026-09-28)

### Features

* adapt live JPEG/video streaming to device rotation ([#322](#322)) ([c537b49](c537b49))
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 8.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants