fix(link): prevent unbounded requestdl API calls on persistent CDN 404 - #381
Open
TwistedRat wants to merge 2 commits into
Open
TwistedRat wants to merge 2 commits into
TwistedRat wants to merge 2 commits into
Conversation
A 404 during CDN link validation was classified as CategoryPermanent, causing the error to be cached indefinitely in the link service's in-memory validated map. If a file's CDN URL returned 404 at any point (e.g. Plex probed a season pack episode before decypharr finished processing it), every subsequent request for that file returned the cached failure without ever retrying — even after the file became available seconds later. The fix reclassifies HTTP 404 as CategoryRefetchable. On the next request the link service invalidates the cached failure, calls fetchDownloadLink to generate a fresh CDN URL, and validates that URL instead. This correctly handles the "file not yet ready" race without permanently poisoning the link cache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the 404→refetchable fix (4a7777c), a file whose CDN URL persistently returns 404 (still processing or genuinely gone) triggered one requestdl call per rclone/Plex poll, indefinitely. invalidateAndRefetch clears the account cache on every call, so each subsequent poll sees a cache miss and hits the provider API again — no upper bound. Add a per-file refetch cooldown (5 min) tracked in a new refetchCooldowns map keyed by infohash:filename. After the first 404 refetch, further requestdl calls for that file are blocked until the window expires. The cooldown is cleared immediately when the file becomes accessible, so recovery is instant once the CDN URL is valid. The CDN-expiry path (line 121) is intentionally excluded — that is a normal 3-hour rotation, not a failure loop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two commits together cause unbounded
requestdlAPI calls that can trigger TorBox'scooldown_untilmechanism:4a7777c(already in beta via this PR) — CDN 404 was changed fromCategoryPermanenttoCategoryRefetchableso that expired CDN URLs are re-fetched rather than cached as permanent failures. This is the correct behaviour for a stale URL, but it has a side-effect:The loop:
invalidateAndRefetchdeletes the link from the account-level cache and callsrequestdlto get a fresh CDN URL. The new URL is returned but not stored ins.validated. On the next rclone/Plex poll (every few seconds),s.validatedhas no entry → HEAD check → 404 → anotherinvalidateAndRefetch→ anotherrequestdlcall. This repeats on every poll for as long as the file returns 404 — which can be minutes while TorBox is still processing a newly-added torrent.The result is hundreds of
requestdlcalls per file per hour. TorBox's API terms treat this as automated abuse and apply an extendedcooldown_untilfreeze on the API key.Fix
pkg/manager/link/errors.go— no change from what landed in beta; 404 staysCategoryRefetchable.pkg/manager/link/service.go— adds a per-file refetch cooldown map:invalidateAndRefetch, the file key (infohash:filename) is recorded with a 5-minute expiry.requestdl.The cooldown is intentionally short (5 min) — long enough to stop the flood, short enough that a newly-processed torrent is retried automatically.
Also included
CategoryRetryable(previously fell through toCategoryPermanent).s.validated— a 502 that clears after 10 s would otherwise block a file for the lifetime of the process.Reproducer
requestdlis called every few seconds until TorBox finishes processing.requestdlcall on first 404, then silence for 5 min, then one retry.Testing
Confirmed against a live TorBox Pro account. The
cooldown_untilfield in/v1/api/user/mewas set by the pre-fix behaviour and expired after the fix was deployed.