feat(usenet) - repair: clear replaced broken entries, and add stale NZB cleanup to Browse - #353
Open
TheMightyBattleCat wants to merge 7 commits into
Conversation
When you set up a username and password and then later changed any setting and hit Save Configuration, authentication would be silently disabled on the next restart - forcing you to re-enter your credentials every time. The credentials themselves were never lost, but the flag that tells the app to actually enforce authentication was being reset to false on every save because the settings form doesn't include it in what it sends to the server. Fixed by preserving the auth enabled flag when saving config changes. Co-authored-by: Brad Dennison <bradleydennison@googlemail.com>
…ot01#333) Users who configure port 563 (NNTPS) or 443 without 'ssl: true' in their usenet provider config get a plain-TCP connection. The NNTP server expects a TLS ClientHello and never sends the greeting, so decypharr blocks in ReadString until HandshakeTimeout (10s) and returns 'read greeting: i/o timeout'. Fix: auto-set SSL=true in updateUsenetProvider when port is 563 or 443. Fixes sirrobot01#319
…3) (sirrobot01#330) GetTorrent() and UpdateTorrent() called /api/torrents/mylist/ with a trailing slash. Cloudflare returns a 307 redirect for that URL and Go's http.Client strips the Authorization header on redirect, causing the re-sent request to fail authentication. getTorrents() and loadDownloadPresent() already used the correct URL without a trailing slash — this aligns the two outliers. Fixes sirrobot01#309 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…rrobot01#329) TorBox changed rate limiting to per-API-key, synchronous across all servers in v8.4 (Feb 2026). Three gaps remained after the initial 429-retry work in PR sirrobot01#324/sirrobot01#325: 1. No proactive rate limiter when rate_limit is omitted from config. ParseRateLimit("") returns nil so TorBox got zero throttling. Fix: default to 300 req/min (10% slack burst) in torbox.New(). 2. Retry-After header ignored on 429 responses. retryablehttp used fixed jitter backoff regardless of what the server requested. Fix: retryAfterBackoff() in request.go parses Retry-After as seconds or HTTP date and sleeps exactly that long, falling back to DefaultBackoff when the header is absent. 3. Retry events not logged. retryClient.Logger was nil, suppressing all retry/429 visibility. Fix: zerologAdapter bridges zerolog to retryablehttp.Logger so retry attempts appear in decypharr's structured debug log. Changes are generic where possible: - retryAfterBackoff and zerologAdapter benefit all providers, not just TorBox. - The 300/min default is TorBox-specific (in torbox.go). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
When a broken file gets healed, the replacement is imported as a new entry, but the old broken entry and its health record used to stick around forever - rechecking it just re-probed the dead release's own missing articles and kept re-confirming it as broken for something the library stopped using long ago, and fixing it could even trigger a pointless blocklist and re-search for a file Sonarr or Radarr had already replaced. Every check is now aware of this: before probing a broken entry, it asks whether any Sonarr or Radarr instance still references its files. If nothing does, the entry is cleared from the broken list instead of being probed again. This is decided purely by what the Arrs currently reference, not by whatever details happened to be stored on the broken record, since most broken records don't carry enough of that information for that approach to work in practice. It also tells apart a dead copy and a re-grabbed working copy of the same release that happen to share the same name, so the healthy copy is never touched by mistake, and it refuses to clear anything if the Arrs can't be reached or aren't configured, so real breakage is never dropped because of an outage or a misconfiguration. Season packs get the same care at the file level: when only some episodes in a pack were replaced, the pack rightly stays in the repair sweep for its still-used episodes, but the replaced ones are no longer probed and re-flagged as broken on every single repair sweep - they're recognized as replaced and skipped instead, with one log line rather than a repeating false alarm. A manual "Clear replaced" action is also available from the broken entries list, for running this cleanup on demand instead of waiting for the next repair sweep. Also filters orphaned EntryHealth records out of the broken-entries listing: a health record's underlying entry can be deleted by something else - most commonly the repair sweep's own worker pool racing the periodic torrent refresh - without the record itself being cleaned up, since nothing currently locks that read-then-write across the two steps. The listing now checks that a record's name still resolves to a real entry before including it, so a stale record left behind by that race stops showing up as an unexplained zero-file, zero-date broken entry.
When Sonarr or Radarr upgrades a release (a WEB-DL upgraded to a BluRay or REMUX, a PROPER replacing the original), the old NZB entry it leaves behind stays in decypharr forever - still listed in Browse, still scanned by managed repair sweeps, still holding its nzb file, metadata and cache on local disk, despite nothing using it anymore. The same goes for any dangling download no Arr tracks, and for orphaned .nzb and .meta files left on disk with no database entry at all. The Browse screen's nzbs view gains a "Clean up stale NZBs" button. It shows a preview of every usenet entry no Sonarr/Radarr references (matched to the specific copy actually serving each file, so a re-grabbed release's live twin is never mistaken for stale), alongside a second list of orphaned files with no database entry. Each item shows the local disk it would free - actual disk usage, with cache and nzb/metadata reported separately, and no inflated content-size numbers, since the articles themselves live on the news servers, not your disk. A filter-as-you-type search box narrows both lists by name. Every item has a checkbox, so manually added downloads the Arrs never tracked can be unticked and kept; select-all and delete-all are one click, and deletion needs a second confirming click either way. The confirmed selection is a request, not an authorization: the server re-checks every chosen entry at deletion time and skips anything no longer stale - newly referenced by an Arr, too recently added, or still downloading - reporting each skip with its reason. Entries and files touched in the last 24 hours are always excluded, since a fresh download is unreferenced until the Arr imports it. The operation refuses to run at all if the Arr lookup fails or no Arrs are configured, and anything an Arr still references is structurally excluded from the delete path. If a preview ever flags more than half the library as stale, a prominent warning says so. Deleting an entry removes its database record, its nzb and metadata files, and its DFS cache directory; deleting an orphan removes the files. The result reports exactly how much local disk came back. rclone users' VFS cache is deliberately left alone - rclone reclaims it by its own policy once the entries are gone. Both preview and cleanup log per-phase durations and counts at DEBUG and a summary at INFO, and surface failures persistently in the modal rather than a transient toast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Two related lifecycle gaps, both stemming from the same fact: when Sonarr or Radarr replaces a release — a repair re-search, a quality upgrade, a PROPER — the replacement arrives as a new entry, and nothing ever closes the book on the old one.
Broken records never die. A healed broken entry's health record sticks around forever. Every subsequent sweep re-probes the dead release's own missing articles and re-confirms it broken — for something the library stopped using weeks ago. Worse, "fixing" it can fire a pointless blocklist + re-search for a file the Arr already replaced. Season packs are the chronic case: a pack whose broken episodes were individually re-grabbed stays a legitimate sweep candidate (its other episodes are still in use), so its dead episodes get re-probed and re-flagged as broken on every single sweep, permanently.
Stale entries never leave. The old NZB entry an upgrade leaves behind stays in Browse forever, gets scanned by every managed sweep, and holds its nzb file, metadata, and DFS cache on local disk — despite nothing referencing it. Same for dangling downloads no Arr tracks, and for orphaned
.nzb/.metafiles with no database entry at all.Part 1 — supersession awareness in the repair path
Before probing a broken entry, every check now asks: does any Sonarr/Radarr instance still reference this entry's files, backed by this entry's own copies? This one question is answered in the same way everywhere — scheduled sweeps, per-entry rechecks, and the Fix Broken action — so the pointless-blocklist problem from the intro is closed at every entry point, not just the sweep.
The classification is deliberately InfoHash-aware, not name-based. A reference set is built once per run mapping entry → file → the InfoHash currently serving that slot. A broken file is superseded when its slot is unreferenced, or referenced but backed by a different InfoHash — the re-grabbed working copy of the very same release name. This is what tells a dead copy apart from its healthy same-name twin, so the twin is never touched. Truth comes from what the Arrs reference right now, not from whatever details happen to be stored on the broken record — most broken records simply don't carry enough information for the stored-record approach to work.
What flows from that classification:
cleanup_supersededenabled (off by default), the underlying entry is also deleted — keyed by InfoHash, mirroring the identifier and call pattern of the existing, proven post-re-search deletion path.skipped_superseded_files.Part 2 — stale NZB cleanup on the Browse screen
The nzbs view gains a "Clean up stale NZBs" button. It previews two lists: every usenet entry no Arr references (matched to the specific copy serving each file, same InfoHash discipline as above), and orphaned
.nzb/.metafiles with no database entry. Each item shows the actual local disk it would free — cache and nzb/metadata reported separately, no inflated content-size numbers, since the articles live on the news servers, not your disk. A filter-as-you-type box narrows both lists.Every item has a checkbox — manually added downloads the Arrs never tracked can be unticked and kept. Deletion removes the database record, the nzb/metadata files, and (when provably safe — see below) the DFS cache directory, then reports exactly how much disk came back. rclone users' VFS cache is deliberately left alone; rclone reclaims it by its own policy once the entries are gone. Both preview and cleanup log per-phase durations at DEBUG, a summary at INFO, and surface failures persistently in the modal.
Why the deletion is safe
This feature deletes things, so it was designed guard-first. The layers, from classification to the final unlink:
DeleteEntryflow that auto-repair has exercised in production, rather than introducing a second deletion mechanism.cleanup_superseded: true, default off.Testing
This is the most deletion-heavy change in the set, so it wasn't submitted until every path had been exercised live rather than just reasoned about. On my own library (DFS with Sonarr + Radarr):
skipped_superseded_files.go build ./...andgo vet ./...pass clean.Config / UI
cleanup_superseded(bool, default false) on the repair config — controls only whether a fully superseded broken entry's underlying entry is deleted; list-clearing is unconditionalTrying it out
A multi-arch Docker test build with this change (alongside my other in-flight fixes) is available at
ghcr.io/themightybattlecat/decypharr:usenet-improvements, with the merged source on theusenet-improvementsbranch of my fork. The PR is two commits — the repair-path supersession awareness, then the Browse cleanup tool built on the same classification — and I'm happy to split or reshape it if you'd prefer.