fix(capture-linux): measure the cursor against the window, not its mo… - #514
Draft
bosskopp wants to merge 1 commit into
Draft
fix(capture-linux): measure the cursor against the window, not its mo…#514bosskopp wants to merge 1 commit into
bosskopp wants to merge 1 commit into
Conversation
…nitor mutter never renegotiates the format for a window stream: it pins the stream to the window's monitor and carves the window out through SPA_META_VideoCrop, which can move on any buffer as the window does. The encoder already reads through that rect, so the file holds the window -- but the portal reports the pointer in STREAM pixels, measured from the monitor's corner, and emit_sample normalised it against the negotiated format. A monitor-relative position divided by monitor dimensions, then painted onto window-sized footage: wrong origin and wrong scale, in every window recording, from the first frame. A 640x480 window at (100, 50) on 1920x1080 put a pointer at the window's centre at 0.219, 0.269. Capture now remembers the rect it actually read -- read_origin at the committed size, so it follows a window that moves and keeps the clamp that stops a shrunken window reading past the buffer -- and content_rect() is the one place that answers "what does the file show". emit_sample takes the origin off the position and reports that rect's dimensions, so the accumulator's x/width lands in the space the compositor assumes. `visible` moves with it. It was tested against the monitor, so a pointer that had left the recorded window still reported visible: true. Full-screen capture is byte-identical: no crop means the content rect is the stream. A cursor-only session opens no encoder and keeps normalising against the stream, which is right -- its video comes from Electron. Not getopenscreen#511. That one is temporal (the video time-compresses under frame drops while the cursor keeps wall-clock time, so the error grows across the take); this one is spatial and constant. They stack.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recording a window on Linux drew the cursor overlay in the wrong place for the whole take — a constant offset, present from the first frame, visible in a three-second clip. Full-screen capture was unaffected.
mutter never renegotiates the format for a window stream. It pins the stream to the window's monitor and carves the window out through
SPA_META_VideoCrop, which can move on any buffer as the window does. The encoder already reads through that rect (Capture::read_origin), so the MP4 holds the window — but the portal reports the pointer in stream pixels, measured from the monitor's corner, andemit_samplenormalised it against the negotiated format. A monitor-relative position divided by monitor dimensions, painted onto window-sized footage: wrong origin and wrong scale. For a 640×480 window at (100, 50) on a 1920×1080 monitor, a pointer at the window's centre serialised as0.219, 0.269instead of0.5, 0.5.Capturenow remembers the rect it actually read —read_originat the committed size, so it follows a window that moves and keeps the clamp that stops a shrunken window reading past the buffer — andcontent_rect()is the single place that answers "what does the file show".emit_samplesubtracts that origin and reports that rect's dimensions, so the accumulator'sx / widthlands in the spaceframe_geometry.rsalready assumes.visiblemoves with it: it was tested against the monitor, so a pointer that had left the recorded window still reportedvisible: true.Full-screen capture is byte-identical — no crop means the content rect is the stream. A cursor-only session opens no encoder and keeps normalising against the stream, which is correct there: its video comes from Electron, not from the helper.
Files touched
electron/native/pipewire-capture/src/capture.rsCapture::contenttracked instage();content_rect()accessorelectron/native/pipewire-capture/src/main.rscontent_rect()helper;emit_samplemeasures against the content rectelectron/native-bridge/cursor/recording/pipeWireCursorAccumulator.tscx = x / widthwas already righttechnical-documentation/architecture/recording.mdRelated issue
Fixes #513
Type of change
Release impact
Desktop impact
Screenshots / video
Not included — the defect is a cursor-position offset in a screen recording, and a still frame cannot show where the pointer actually was. The numeric case in the summary (
0.219, 0.269vs0.5, 0.5) is what the tests pin instead.Testing
cargo testinelectron/native/pipewire-capture— 69 pass (63 before, 6 new). Note that CI does not build or test this crate (the onlycargojob inci.ymlis the macOS compositor), so this is the only automated signal on it.New tests:
capture::tests::the_content_rect_is_the_window_the_file_shows— initial rect, a window that moves mid-take, and the clamped out-of-buffer origin.cursor_sample_tests::a_window_capture_reports_coordinates_inside_the_window— the bug itself.cursor_sample_tests::a_full_screen_capture_reports_stream_coordinates— the no-regression case.cursor_sample_tests::a_pointer_outside_the_window_is_reported_invisible— both directions (past the far edge, and negative).cursor_sample_tests::nothing_is_emitted_before_the_format_is_knowncursor_sample_tests::the_content_rect_prefers_the_encoder_once_it_has_startedAlso run:
cargo clippy --all-targets— no new warnings (the 6 existing ones are pre-existing dead-code notes).npx tsc --noEmitandnpx tsc -p tsconfig.test.json --noEmit— clean.npm run lint— 14 warnings, identical to a cleanmain.npm run test— 2045 pass / 40 fail, identical counts to a cleanmain(pre-existing jsdomlocalStoragesetup failures insrc/components/**, unrelated to this change; verified by stashing and re-running).node scripts/check-docs.mjs— OK (31 files).npm run build:native:linux— the release helper builds and its startup probe reports{"cursorMetadataSupported":true,"pipewireVersion":"1.0.5","event":"ready"}.Manual verification on Ubuntu / GNOME / Wayland (mutter, AMD): the v1.10.0 release AppImage was extracted and run against the rebuilt helper via
OPENSCREEN_LINUX_CURSOR_HELPER_EXE, recording a window. The cursor lands where the pointer actually was. Full-screen recording re-checked for regressions.