Improve content loading and enrich browse sources

README.md

@@ -43,8 +43,9 @@
 
 Opal replaces the stack you'd otherwise juggle — a player, a site for the show,
 a server front-end, a torrent client, a feed. Say what you want (a title, a
-file, a magnet); it searches every source, plays anything, and remembers where
-you left off. One native binary — [Zig](https://ziglang.org) +
+file, a magnet); it searches supported, enabled providers, opens playable media
+and reading results, and remembers where you left off. Personal libraries need
+a configured server; catalog entries can lead to details or a source search. One native binary — [Zig](https://ziglang.org) +
 [dvui](https://github.com/david-vanderson/dvui) + **mpv** — fast and quiet.
 
 <div align="center">
@@ -250,7 +251,7 @@ XDG-compliant:
     <td width="50%" valign="top">
       <img src="assets/screenshots/search.png" width="100%" alt="One query fanned out across every source, ranked" /><br/>
       <b>🔭 One search, every source</b><br/>
-      <sub>Disk, torrents, Jellyfin, Plex, Stremio, anime, YouTube, live TV, TMDB, manga — one ranked, playable list.</sub>
+      <sub>Disk, torrents, Jellyfin, Plex, Stremio, anime, YouTube, live TV, TMDB, manga, novels and visual novels — one sourced list with playback, reading, or detail actions.</sub>
     </td>
   </tr>
   <tr>
@@ -275,7 +276,7 @@ XDG-compliant:
 
 | Instead of… | Opal gives you |
 |---|---|
-| **Stremio / Kodi** + a pile of add-ons | one search across every source, a play button on each row |
+| **Stremio / Kodi** + a pile of add-ons | one search across supported, enabled providers, with playback, reading, and detail actions |
 | **an IPTV / live-TV app** | ~40,000 live channels, searchable as you type |
 | **Jellyfin / Plex** web clients | your own media servers, browsed natively |
 | **Tachiyomi / Mihon** stuck on your phone | manga extensions on the desktop — server bundled, self-managed |

docs/browse-sources.md

@@ -93,3 +93,121 @@ implemented. These remain research entries rather than working source claims.
 Validation for the expansion: native unit/browse suites and the headless build
 passed; the feature suite reported 463 passed, 0 failed, 1 warning and 11
 skipped. The web lifecycle suite reported 29 passed.
+
+## Universal search coverage
+
+Loading improvements dated 2026-10-02. The validation results above describe
+the earlier source expansion, not this set of changes.
+
+Universal search runs independent requests against the selected search sources.
+It does not call a Browse tab's search function or replace that tab's query and
+results. Source buttons show fetching, results, no matches, partial results,
+transport/parse failures, or unavailable configuration separately. Results
+already returned remain usable while slower sources finish.
+
+| Content | Universal search | Result action |
+| --- | --- | --- |
+| Movies and television | TMDB when configured, public Cinemeta otherwise; Cinemeta also backs up failed TMDB catalog requests | Show details or find playback sources; a metadata hit is not a playable stream |
+| Torrents | Installed Nova2 indexes, YTS, configured Torznab/Prowlarr/Jackett, installed EZTV, cached RSS magnets | Resolve the listed magnet or source detail page |
+| Anime | Installed anime connector plus torrent/Stremio providers | Open anime discovery; playback still requires an accessible source |
+| Comics and manga | Installed comic connector and trusted executable source plugins | Open the comics reader |
+| Novels | Independent Wikisource and Internet Archive readable-text searches, plus compatible installed executable plugins | Open the built-in reader by copied work identity |
+| Visual novels | Independent public VNDB lookup with the existing cover filter | Open VNDB details; Opal does not run or download a visual novel from a metadata result |
+| Drama | Movie/TV catalog searches include drama titles; playback uses the normal configured resolver | Catalog details or source discovery; no dedicated drama streaming scraper is built in |
+| YouTube | Independent bounded yt-dlp search | Play the canonical video URL |
+| Live television | Loaded IPTV catalog matched against the query | Play with the channel's required headers |
+| Music, radio and podcasts | Independent public provider requests | Play audio or open a podcast's episode view |
+| Personal video libraries | Connected Jellyfin and Plex server searches | Play the server item; credentials stay out of universal result URLs |
+| Local files | Configured download directory | Play the existing local file |
+| Installed source plugins | Trusted executable plugins implementing the search protocol | Open the supplied stream or resolve the plugin item |
+| OPDS and Audiobookshelf | Their configured Browse libraries remain separate from universal query fan-out | Browse the connected library page |
+| Web | URL navigation in the web browser, rather than a media catalog provider | Open a website |
+
+Catalog results retain real provider covers, descriptions, ratings and genres
+when supplied. Empty fields remain unknown. VNDB's provider rating uses a
+100-point scale; universal cards convert it to the shared ten-point scale.
+Arbitrary executable-plugin scores are not converted into ratings because their
+units are unspecified. Search caches include the enrichment and use a new
+format key so older cache rows cannot decode as new results.
+
+Wikisource, Internet Archive and VNDB queries are bounded to twelve records per
+provider; YouTube discovery requests ten records. These limits prevent a single
+catalog from occupying every universal result slot. Browse pagination remains
+the path for exploring a larger catalog.
+
+## Provider contract research for loading improvements
+
+The [Stremio catalog request contract](https://github.com/Stremio/stremio-addon-sdk/blob/master/docs/api/requests/defineCatalogHandler.md)
+distinguishes search and catalog pagination, and its
+[metadata contract](https://github.com/Stremio/stremio-addon-sdk/blob/master/docs/api/responses/meta.md)
+defines optional artwork, descriptions, release dates and IMDb ratings.
+Cinemeta responses provide those fields directly; a local HTTP inspection
+returned 49 movie records with the expected metadata. This inspection verifies
+the response shape, not playback of all listed titles.
+
+The [MediaWiki search API](https://www.mediawiki.org/wiki/API:Search) provides
+Wikisource titles and snippets without a reader-side search mutation. The
+[VNDB HTTPS API](https://api.vndb.org/kana) provides public visual-novel metadata
+and image flags. Direct request inspections returned two Wikisource search
+records and two VNDB metadata records. These inspections do not replace app
+build checks or interactive verification.
+
+Failure handling now preserves existing movie/TV cards when refresh fails,
+marks cache freshness after successful publication, and distinguishes rejected
+Plex authentication from a genuine empty server search. Independent VNDB detail
+requests preserve the Browse grid and discard superseded detail responses.
+Upstream outages, access restrictions and absent server configuration remain
+reported conditions; they cannot be repaired by adding an unverified mirror.
+
+## Loading and reader promise gaps addressed
+
+| Promise | Change | Practical limit |
+| --- | --- | --- |
+| Useful discovery results | Real covers, authors, years, snippets, summaries and ratings reach native and web views | A missing provider field stays empty |
+| Keep browsing beyond the first page | Wikisource uses the server continuation; comics and audio advance by consumed provider rows | Fixed result buffers still bound a session |
+| Read a manga series | MangaDex has a chapter picker and previous/next navigation; the web reader requests pages that actually finished downloading | 100 chapter rows per feed window, a client navigation bound of 10,000 rows, and 128 pages per chapter |
+| Read novels without losing the selected work | Reader workers copy work identity; older requests cannot replace newer text or clear its loading state | 400 chapters and bounded text per work; truncated text is labelled |
+| Reliable connected reading catalogs | OPDS checks HTTP success and complete Atom envelopes, decodes next links, deduplicates pages, and snapshots request credentials | Unprefixed OPDS 1.x Atom feeds; 300 entries and a 4 MiB response limit |
+| Retain usable television catalogs | IPTV source replacement deletes and inserts in one transaction, rolling back failed replacements | Stream reachability still depends on the broadcaster |
+| Accurate search actions | Catalog entries open details and reading entries open readers; queue actions require a usable playback identity | Metadata does not guarantee an accessible stream |
+| Correct artwork while results change | Shared cover slots suppress images from superseded rows, including rows with no artwork | Failed images use a fallback |
+
+Provider contracts used for these changes include the official
+[MangaDex OpenAPI](https://api.mangadex.org/docs/static/api.yaml),
+[Internet Archive advanced search](https://archive.org/advancedsearch.php),
+[OPDS 1.2 specification](https://github.com/opds-community/specs/blob/master/opds-1.2.md),
+[Radio Browser API](https://api.radio-browser.info/),
+[Audius API](https://docs.audius.org/api/), and
+[Audiobookshelf API](https://api.audiobookshelf.org/).
+Personal-server playback still requires verification against a configured server.
+
+
+Connected Audiobookshelf searches use the upstream
+[library search controller](https://github.com/advplyr/audiobookshelf/blob/master/server/controllers/LibraryController.js)
+and [expanded book search response](https://github.com/advplyr/audiobookshelf/blob/master/server/utils/queries/libraryItemsBookFilters.js)
+contracts: `GET /api/libraries/{id}/search?q=...&limit=6`, with `book` or
+`podcast` matches wrapping `libraryItem` metadata. Up to sixteen permitted
+libraries are searched concurrently; universal search surfaces at most twelve
+items across them. Responses and each request are bounded. Authentication stays
+in headers, and only stable item identities enter universal results.
+
+Opening an Audiobookshelf result fetches expanded file metadata rather than
+passing its download archive to the media player. Native and web views offer
+actual audio tracks or downloaded podcast episodes, with up to 128 selectable
+files and explicit returned/total/truncation information. Single-file books can
+start immediately and use book-wide server resume. Multi-file playback uses
+manual track selection; automatic track advancement and book-wide resume across
+files are not implemented. No connected personal-server search or playback was
+verified during this provider-contract research.
+
+## Checks for the 2026-10-02 loading changes
+
+- Native `zig build` passed after integration, including the refresh retry cooldown.
+- JavaScript syntax checks passed for `catalog.js`, `media.js` and `integrations.js`.
+- Zig formatting and `git diff --check` passed.
+- Regression tests and interactive playback checks were not run for these changes.
+- Personal-server search/playback has not been verified against a configured account.
+
+Remaining functional gaps include independent OPDS server search, automatic
+Audiobookshelf track advancement, and server resume across multi-file books.
+Provider outages and access restrictions remain external availability conditions.

engines/engines/animetosho.py

@@ -0,0 +1,69 @@
+# VERSION: 1.0
+# AUTHORS: Opal
+"""AnimeTosho's documented JSON feed; actual torrent metadata, no DDL scraping.
+
+Contract: https://animetosho.org/about#feeds
+Search uses q; swarm counts and total_size are provider fields, never guesses.
+"""
+import html
+import json
+import re
+from urllib.parse import parse_qsl, unquote, urlencode, urlsplit
+
+from helpers import retrieve_url
+from novaprinter import prettyPrinter
+
+
+class animetosho:
+    url = 'https://feed.animetosho.org'
+    name = 'AnimeTosho'
+    supported_categories = {'all': '', 'movies': '', 'tv': ''}
+
+    def search(self, what, cat='all'):
+        words = unquote(what).strip()
+        if not words or cat not in self.supported_categories:
+            return
+        data = json.loads(retrieve_url(
+            self.url.rstrip('/') + '/json?' + urlencode({'q': words}),
+            unescape_html_entities=False))
+        if not isinstance(data, list):
+            return
+        seen = set()
+        for row in data[:100]:
+            if not isinstance(row, dict) or row.get('status') in ('deleted', 'hidden'):
+                continue
+            digest, title = row.get('info_hash'), row.get('title')
+            if (not isinstance(digest, str) or not re.fullmatch(r'[a-fA-F0-9]{40}', digest)
+                    or not isinstance(title, str) or not title.strip()):
+                continue
+            digest = digest.lower()
+            if digest in seen:
+                continue
+            seen.add(digest)
+            title = ' '.join(html.unescape(title).split()).replace('|', ' ')
+            # Canonical hex BTIH deduplicates Nyaa and other indexes of the same
+            # release. Preserve the feed's tracker URLs without its base32 xt.
+            params = [('dn', title)]
+            magnet = row.get('magnet_uri', '')
+            if isinstance(magnet, str) and len(magnet) <= 4096 and magnet.startswith('magnet:?'):
+                for key, value in parse_qsl(urlsplit(magnet).query):
+                    if key == 'tr' and value.startswith(('https://', 'http://', 'udp://')):
+                        params.append((key, value))
+            link = 'magnet:?xt=urn:btih:' + digest + '&' + urlencode(params)
+            detail = row.get('link', '')
+            if not isinstance(detail, str) or '\n' in detail or '\r' in detail:
+                detail = ''
+            host = urlsplit(detail).hostname or ''
+            if urlsplit(detail).scheme != 'https' or not (host == 'animetosho.org' or host.endswith('.animetosho.org')):
+                detail = ''
+
+            def count(key):
+                try:
+                    return max(-1, int(row.get(key, -1)))
+                except (TypeError, ValueError):
+                    return -1
+
+            prettyPrinter({'link': link, 'name': title,
+                           'size': count('total_size'), 'seeds': count('seeders'),
+                           'leech': count('leechers'), 'pub_date': count('timestamp'),
+                           'engine_url': self.url, 'desc_link': detail})

site/src/pages/features.astro

@@ -6,7 +6,7 @@ const page = pageFor("/features/");
 const groups = [
   {
     title: "Find everything from one search",
-    body: "One query covers local files, torrents, Jellyfin, Plex, Stremio sources, anime, YouTube, live TV and manga — every result labelled with where it came from.",
+    body: "One query covers enabled search providers for local files, torrents, configured Jellyfin and Plex libraries, Stremio sources, anime, YouTube, live TV, manga, novels and visual novels. Results show their source and offer playback, reading or details.",
   },
   {
     title: "Stream torrents like ordinary media",
@@ -67,7 +67,7 @@ const groups = [
       <div>
         <p class="eyebrow">Universal search</p>
         <h2>One query, clearly sourced results</h2>
-        <p class="sub">A query fans out to every enabled provider and comes back as a single ranked list, badged by source.</p>
+        <p class="sub">A query fans out to supported, enabled search providers and comes back as one list, badged by source. Reading-server catalogs have their own browse views.</p>
         <a class="text-link" href="/compare/">Compare Opal with other media apps →</a>
       </div>
       <div class="shot">

src/core/config.zig

@@ -717,23 +717,17 @@ fn applyConfig(key: []const u8, val: []const u8) void {
     } else if (std.mem.eql(u8, key, "abs_connected")) {
         state.app.abs.connected = std.mem.eql(u8, val, "1") and state.app.abs.token_len > 0;
     } else if (std.mem.eql(u8, key, "opds_url")) {
-        if (val.len > 0 and val.len < state.app.opds.server_url.len) {
-            @memcpy(state.app.opds.server_url[0..val.len], val);
-            state.app.opds.server_url_len = val.len;
-        }
-    } else if (std.mem.eql(u8, key, "opds_user")) {
-        var restored_len: usize = 0;
-        if (val.len > 0 and loadSecretValue(key, val, state.app.opds.user_buf[0 .. state.app.opds.user_buf.len - 1], &restored_len)) {
-            state.app.opds.user_buf[restored_len] = 0;
-        }
-    } else if (std.mem.eql(u8, key, "opds_pass")) {
+        @import("../services/opds.zig").setConfiguredServer(val);
+    } else if (std.mem.eql(u8, key, "opds_user") or std.mem.eql(u8, key, "opds_pass")) {
+        var buffer: [128]u8 = undefined;
+        defer @memset(&buffer, 0);
         var restored_len: usize = 0;
-        if (val.len > 0 and loadSecretValue(key, val, state.app.opds.pass_buf[0 .. state.app.opds.pass_buf.len - 1], &restored_len)) {
-            state.app.opds.pass_buf[restored_len] = 0;
+        if (val.len > 0 and loadSecretValue(key, val, buffer[0 .. buffer.len - 1], &restored_len)) {
+            const opds = @import("../services/opds.zig");
+            if (std.mem.eql(u8, key, "opds_user")) opds.setConfiguredUser(buffer[0..restored_len]) else opds.setConfiguredPassword(buffer[0..restored_len]);
         }
     } else if (std.mem.eql(u8, key, "opds_connected")) {
-        // Connected only if a catalog URL was also restored.
-        state.app.opds.connected = std.mem.eql(u8, val, "1") and state.app.opds.server_url_len > 0;
+        @import("../services/opds.zig").setConfiguredConnected(std.mem.eql(u8, val, "1"));
     } else if (std.mem.eql(u8, key, "win_x")) {
         state.app.win_x = std.fmt.parseInt(i32, val, 10) catch 0;
         state.app.win_restore_pending = true;
@@ -852,7 +846,9 @@ fn applyConfig(key: []const u8, val: []const u8) void {
     } else if (std.mem.eql(u8, key, "search_sources")) {
         const resolver = @import("../services/resolver.zig");
         const mask = std.fmt.parseInt(u16, val, 10) catch 0xFF;
-        resolver.source_mask.store(mask, .release);
+        // Legacy "all sources" predates the novel/VNDB search bits. Preserve
+        // custom filters while keeping the all-source preference inclusive.
+        resolver.source_mask.store(if (mask == 4095) resolver.ALL_SOURCE_BITS else mask, .release);
     }
 }
 

src/core/source_config.zig

@@ -253,6 +253,28 @@ pub fn get(id: []const u8, field: []const u8) ?[]const u8 {
     return null;
 }
 
+pub const FieldSnapshot = struct {
+    id: [32]u8 = std.mem.zeroes([32]u8),
+    id_len: usize = 0,
+    value: [512]u8 = std.mem.zeroes([512]u8),
+    value_len: usize = 0,
+};
+
+/// Copy installed endpoints while holding the reload lock. Detached requests
+/// retain owned values rather than slices into a table that can be reloaded.
+pub fn copyFields(field: []const u8, out: []FieldSnapshot) usize {
+    mutex.lock();
+    defer mutex.unlock();
+    var count: usize = 0;
+    for (entries[0..entry_count]) |entry| {
+        if (!std.mem.eql(u8, entry.field[0..entry.field_len], field)) continue;
+        if (count == out.len) break;
+        out[count] = .{ .id = entry.id, .id_len = entry.id_len, .value = entry.val, .value_len = entry.val_len };
+        count += 1;
+    }
+    return count;
+}
+
 /// True if ANY source plugin is installed at all. False is the fresh-install /
 /// post-reset state (Opal ships neutral): every torrent/comics/anime engine is
 /// inert, so searches "run" but can't return source hits — surface that in the