-
Notifications
You must be signed in to change notification settings - Fork 134
fix(audio): guard the decode loop against pathological stalls #430
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e2e89c9
cb3aed5
6d03057
19d4aee
fc92426
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,6 +24,9 @@ const MIN_FRAME_SEC: f64 = 0.005; | |
| const DEFAULT_SEARCH_SEC: f64 = 0.01; | ||
| const TARGET_GRAINS: usize = 8; | ||
| const PASSTHROUGH_EPSILON: f64 = 1e-3; | ||
| const DECODE_BUDGET_SLACK: f64 = 8.0; | ||
| const MIN_DECODE_BUDGET_SEC: u64 = 60; | ||
| const MAX_DECODE_BUDGET_SEC: u64 = 3600 * 8; | ||
|
|
||
| pub type PlanarPcm = Vec<Vec<f32>>; | ||
|
|
||
|
|
@@ -306,15 +309,59 @@ unsafe fn decode_clip_audio_inner( | |
| // décalage inter-pistes est absorbé là, pas ici. | ||
| let seek_tb_sec = tracks[0].tb_sec; | ||
| let seek_stream_index = tracks[0].stream_index; | ||
| // Where decoding actually starts. The budget below scales with the amount | ||
| // of input the loop will consume, which is the requested window on the | ||
| // happy path — but when a failed seek forces a reset to t=0 the loop must | ||
| // decode the whole file from the start (mix_aligned_tracks then trims the | ||
| // samples before source_start_sec). Sizing the budget on the window alone | ||
| // would starve an unseekable-but-healthy long file: a 1 s window inside a | ||
| // 3 h recording would get the 60 s floor while having to decode 3 h. | ||
| let mut decode_start_sec = source_start_sec; | ||
| if seek_tb_sec > 0.0 { | ||
| let target = (source_start_sec / seek_tb_sec).floor() as i64; | ||
| if av_seek_frame(fmt, seek_stream_index, target, AVSEEK_FLAG_BACKWARD) >= 0 { | ||
|
Comment on lines
+319
to
322
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
rg -n -C 6 'decode_start_sec|av_seek_frame|AVSEEK_FLAG_BACKWARD|origin_sec|mix_aligned_tracks' crates/compositor/src/audio.rsRepository: getopenscreen/openscreen Length of output: 10367 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- audio decoder and loop ---'
sed -n '150,430p' crates/compositor/src/audio.rs
printf '%s\n' '--- FFmpeg interrupt/seek declarations and wrappers ---'
rg -n -C 5 'AVIOInterruptCB|interrupt|av_seek_frame|av_read_frame|avformat_open_input|avformat_find_stream_info|avcodec_send_packet|avcodec_receive_frame' crates/compositor/src
printf '%s\n' '--- decoder cleanup ---'
rg -n -C 8 'impl Drop for AudioTrackDecoder|avcodec_free_context|swr_free|avformat_close_input|av_packet_free|av_frame_free' crates/compositor/srcRepository: getopenscreen/openscreen Length of output: 50381 🌐 Web query:
💡 Result: In FFmpeg, the av_seek_frame function uses the AVSEEK_FLAG_BACKWARD flag to indicate that when an exact match for the requested timestamp is not available, the seek operation should land at a position at or before the target timestamp [1]. Regarding your specific areas of inquiry: Keyframe Behavior and Guarantees The flag AVSEEK_FLAG_BACKWARD does not inherently guarantee that the resulting position will be a keyframe [2][3]. The success of landing on a keyframe depends heavily on the demuxer implementation, the file container, and the underlying stream type [2][4]. While the default behavior (without AVSEEK_FLAG_ANY) is intended to prioritize keyframes, many demuxers—particularly those handling complex containers like MPEG-TS or specific video formats—may return a position that is not a keyframe if an exact keyframe match is not found or if the index information is insufficient [2][3][4]. Users often encounter situations where the first packet returned after a seek has flags = 0, indicating a non-keyframe [3]. Audio Streams FFmpeg's av_seek_frame is a legacy API that operates primarily based on a single stream index [1]. When you provide a stream index, it attempts to seek relative to that stream's timebase [5]. If you pass -1 as the stream_index, the function attempts to guess a default stream, which can lead to unpredictable results if the container has multiple streams (e.g., audio and video) with different interleaving or keyframe characteristics [1][3]. Because audio streams often have different (or effectively non-existent) keyframe structures compared to video, seeking in audio may result in landing at the requested timestamp without the "keyframe" restriction being meaningfully applied, or it may disrupt synchronization if the seek was intended to align video keyframes [1]. Practical Recommendations 1. Expected Behavior: You cannot assume the result of av_seek_frame with AVSEEK_FLAG_BACKWARD will be a keyframe [3][4]. If your application requires a keyframe (e.g., for accurate decoding), you must often manually discard non-keyframe packets or decode forward from a preceding keyframe until you reach the target timestamp [4]. 2. Alternative API: The newer avformat_seek_file function is generally preferred over av_seek_frame as it offers more granular control, including explicitly defining minimum and maximum timestamp bounds, which can lead to more predictable seeking behavior [6][1]. 3. Verification: Always inspect the packet flags of the first packet received after a seek if your application's logic depends on starting from a keyframe [3]. Citations:
Include seek pre-roll in the decode budget.
🤖 Prompt for AI Agents |
||
| for track in tracks.iter_mut() { | ||
| avcodec_flush_buffers(track.dctx); | ||
| } | ||
| } else { | ||
| eprintln!( | ||
| "[openscreen-compositor] decode_clip_audio: av_seek_frame a échoué (target={target}), tentative de retour à t=0" | ||
| ); | ||
| // A failed seek can flush the demuxer's packet queue and leave it | ||
| // mid-way through its fallback scan, so the next `av_read_frame` is | ||
| // not guaranteed to resume at t=0 — leading audio could be silently | ||
| // omitted. Reset to the start and flush every decoder; if even that | ||
| // reset fails, abort rather than risk an export that starts | ||
| // mid-stream. | ||
| if av_seek_frame(fmt, seek_stream_index, 0, AVSEEK_FLAG_BACKWARD) < 0 { | ||
| avformat_close_input(&mut fmt); | ||
| bail!( | ||
| "decode_clip_audio: av_seek_frame a échoué (target={target}) puis le retour à t=0 a échoué — abandon" | ||
| ); | ||
| } | ||
| decode_start_sec = 0.0; | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The seek-failure fallback restarts at t=0, but A MediaRecorder WebM with no cues (you already have Before this fallback existed the path decoded from wherever the demuxer sat, so the blowup is new. Dropping samples below |
||
| for track in tracks.iter_mut() { | ||
| avcodec_flush_buffers(track.dctx); | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| } | ||
|
|
||
| // Anti-loop guard: a container whose audio track is truncated or corrupt at | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not convinced the stall this guards against is reachable through this loop, and it's worth pinning down before adding a timer. At :382-388, If the real stall is instead ffmpeg's internal fallback scan or blocking I/O on a removable volume, it happens inside one Worth noting the three sibling video loops have the same shape and no guard ( |
||
| // end-of-stream can make `av_read_frame` never return AVERROR_EOF, so | ||
| // `decoder_eof` never propagates and the loop spins at 100% CPU forever. | ||
| // A TIME budget, not an iteration count: `av_read_frame` can be slow on a | ||
| // corrupt stream, so a count would either never fire or cut healthy long | ||
| // clips short. The budget scales with the requested window | ||
| // (`DECODE_BUDGET_SLACK`), with a floor (`MIN_DECODE_BUDGET_SEC`) and a hard | ||
| // ceiling (`MAX_DECODE_BUDGET_SEC`, so a WebM reporting duration = Infinity | ||
| // cannot disable it). | ||
| let loop_start = std::time::Instant::now(); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The budget formula has real edge cases and none of them can be exercised: it's inline in an
This same file shows the pattern that would fix it — :476 says |
||
| let span_sec = (source_end_sec - decode_start_sec).max(0.0); | ||
| let loop_budget_secs = ((span_sec * DECODE_BUDGET_SLACK) as u64) | ||
| .max(MIN_DECODE_BUDGET_SEC) | ||
| .min(MAX_DECODE_BUDGET_SEC); | ||
| let loop_budget = std::time::Duration::from_secs(loop_budget_secs); | ||
|
|
||
| let mut packet = av_packet_alloc(); | ||
| let mut frame = av_frame_alloc(); | ||
| let mut input_eof = false; | ||
|
|
@@ -323,6 +370,20 @@ unsafe fn decode_clip_audio_inner( | |
| // piste dont il porte l'index. On continue tant qu'AU MOINS une piste a encore quelque | ||
| // chose à produire. | ||
| while tracks.iter().any(|t| !t.reached_end && !t.decoder_eof) { | ||
| if loop_start.elapsed() > loop_budget { | ||
| // A real stall, not a slow decode: abort rather than emit a | ||
| // truncated/silent clip. The downstream pipelines already degrade a | ||
| // `bail!` here into their documented silent-fallback path in one | ||
| // place, instead of inventing an invisible one. | ||
| av_frame_free(&mut frame); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This adds a third hand-rolled So a user retrying a failing export leaks an AVFormatContext plus its I/O buffers per attempt, and the next |
||
| av_packet_free(&mut packet); | ||
| avformat_close_input(&mut fmt); | ||
| bail!( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This All three callers — And it throws away work that's already done: |
||
| "decode_clip_audio: decode loop exceeded {loop_budget_secs}s budget \ | ||
| (source_end={source_end_sec}s) — aborting to avoid exporting a \ | ||
| truncated clip" | ||
| ); | ||
| } | ||
| if !input_eof { | ||
| let read = av_read_frame(fmt, packet); | ||
| if read == AVERROR_EOF { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor:
decode_start_secis amutbinding plus a 7-line justification comment, and its only consumer isspan_sec39 lines later.Since the budget already clamps to the 8h ceiling, worst-case sizing straight from
source_end_secgives the same clamped result in every branch and drops both the binding and the comment. Small thing, but it's one less piece of state to keep correct if the fallback branch changes.