Conversation
`parseNovel` counted `#select-pagination-chapter > option` elements for `totalPages`, while `parsePage` recomputed it from the last numeric option in the whole document, and then mapped each app page onto a site page with `totalPages - page + 1`. The two derivations could disagree, and the reversed mapping made the site's partial page the FIRST page instead of the last: on live novels page 1 returned 2, 6 and 49 chapters instead of 50. Because that partition is anchored at the newest chapter, every added chapter shifted all page boundaries, which the app's per-page chapter cache turns into duplicated or missing chapters. Both `parseNovel` and `parsePage` now derive the chapter count from the largest range end in the site's page select (`ceil(total / 50)` pages), the same shape as the sibling `mynovels` plugin for this CMS. `parsePage` fetches only the site pages whose ranges overlap the requested 50-chapter chunk and slices exactly that chunk, so pages 1..totalPages partition the chapter list: every page but the last holds 50 chapters, and the partial page is last. Anchoring the chunks at the oldest chapter also keeps already-served pages stable when new chapters are added. Locked chapters are filtered after slicing to keep positions stable; name, URL, date and locked-chapter handling are unchanged. Authored with AI assistance.
|
works also |
…ted site popularNovels returned no novels in CI although the catalog markup and the 'a.item' selector are unchanged: the plugin sent no User-Agent at all, and the Cloudflare-fronted site answers such requests with a bot-check page on networks it distrusts (e.g. CI runners), which parses to zero novels. The live check's own HEAD probe (which carries a Mozilla User-Agent) passed, so the failure showed up as FAIL rather than INCONCLUSIVE. Pass browser-like headers (User-Agent, Referer, Accept-Language) on every fetchApi call, following the indraTranslations/dreamyTranslations pattern. Chapter-list pagination behavior is unchanged. Local live check passes: popularNovels 30 novels, searchNovels 1 result, parseNovel (parsePage) 50 chapters, parseChapter 12215 chars. Refs lnreader#2526. Authored with AI assistance.
Route every site request through one helper that throws (carrying the HTTP status) on a non-OK response, so a runner-side block is reported INCONCLUSIVE per docs/testing.md instead of being parsed into a false 'Returned no novels' FAIL. Headers and pagination behavior unchanged. This commit was authored with AI assistance.
|
Pushed v1.1.8 for this head: every request now goes through one helper that treats a refused response as a site block (reported INCONCLUSIVE per docs/testing.md) instead of parsing a block page into an empty catalog. Local |
|
The live check should work now on this head too. The previous failure class is a runner-side block being parsed into an empty result and reported as a plugin failure; every request here now carries browser-like headers and treats a refused response as a site block, which docs/testing.md defines as INCONCLUSIVE rather than a plugin FAIL. Verified against the live site (popular 30, search, 50 chapters per page, chapter text) and driven through the plugin playground in the app UI. The check run is waiting on approval — could you approve it so it can report? |
|
I dont think slicing each page to 50 chapters is a good idea because site's own page width isnt 50 chapters and slicing to 50 per page leads to 2 fetch request for nearly every page and since this site already does cf checks it may cause 429 or 403. Also why is the version bump to 1.1.8 instead of 1.1.6 |
Yes you are right it, it calls for the page 2 times. I tested it internally and it worked out fine managed to fetch the chapters and its content, so the Cloudfare worry might not be true as per my data . But im not so sure about other devices and locals so ill try to find a work around. For the updates it was a chore bump due to many commits should be at 1.16 genuine mistake. |
Replace the uniform oldest-anchored 50-chapter slicing (which straddled two of the site's newest-anchored ranges on nearly every page, costing 2 chapter-pagination ajax requests) with a one-to-one mapping onto the site's own ranges: app page N is the Nth site range in ascending chapter order. The endpoint honors only the page parameter (page_size, per_page, limit, offset, start/end, paginate_by, count, size, length, pageSize, perPage, show all ignored; bookmark_chapter inert; the site's own JS sends only csrfmiddlewaretoken, book_id and page), and the width is fixed at 50 server-side, so the site's own partition is the only single-request design. parseNovel derives totalPages from the same markup, keeping the two in agreement, and requested values come straight from the parsed select so out-of-range clamping can no longer duplicate chapters. Trade-off: the short (remainder) page comes first instead of last, and new chapters shift every page's contents, as before this PR. Set the version to a single 1.1.5 -> 1.1.6 net bump. This commit was authored with AI assistance.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const sitePages = this.parseChapterPagination(parseHTML(rawBody)).sort( | ||
| (a, b) => a.start - b.start, | ||
| ); | ||
| const pageNo = parseInt(page, 10); | ||
| const sitePage = sitePages[pageNo - 1]; |
There was a problem hiding this comment.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
- parseSitePageChapters takes a single params object instead of four positional strings - throw a clear error when the chapter-pagination JSON has no html string instead of parsing undefined into zero chapters silently - let endpoint failures propagate with their original stack; drop the console.error noise and the lossy throw new Error(error) wrapper - drop the parsed-but-never-read ChapterPaginationOption.end field - filter valueless pagination options, comment the non-range skip, drop the duplicated newest-first comment - bump 1.1.6 to 1.1.7 (patch: behavior-preserving robustness fix) Co-Authored-By: firstmate-crewmate (AI agent)
This branch has not been deployed
2 participants
Fix
Closes #2526. Update addressing review feedback (#2527 (comment)):
parsePagenow serves the site's own ranges one-to-one (app page N = Nth site range in ascending chapter order, oldest chunk first) instead of re-slicing uniform oldest-anchored 50s that straddled two site ranges on nearly every page.parseNovelsetstotalPagesto the number of site ranges from the same markup, so the two agree by construction, and requested page values come straight from the parsed select (out-of-range requests used to be silently clamped to the oldest page, duplicating its chapters).1.1.5→1.1.6(single net bump, not one per commit).Why one-to-one and not a page-size parameter: the endpoint only honors
page. Probedpage_size,per_page,limit,offset,start/end,paginate_by,count,size,length,pageSize,perPage,show— all ignored (byte-identical 50-anchor responses) — andbookmark_chapter(inert); the site's own JS (static/book/js/single.js) sends only{csrfmiddlewaretoken, book_id, page}. Page width is fixed at 50 server-side, so no single request can serve an arbitrary oldest-anchored chunk.Verification
Bundled-plugin harness wrapping
fetch, live against the real site: everyparsePagecall issued exactly 1 chapter-pagination request — Turning 1471/1471 chapters over 30 pages[21, 50x29], Best Friend X Swap 202/202 over 5 pages[2, 50x4], Regression Guidelines 453/453 over 10 pages[3, 50x9], Guild Princess 117/117 over 3 pages[17, 50, 50]— zero missing or duplicated chapters; per-page sizes match the site's own ranges exactly.npm run check:plugin -- plugins/english/novelight.tspasses all four steps (30 popular, 1 search result, 43-chapter page, ~12k-char chapter).npm run build:compileclean; the file is clean undereslintandprettier --check(full-repolintstill shows 2 pre-existing errors in unrelated files).Caveats: the short (remainder) page now comes first instead of last, and because the site anchors its partition at the newest chapter, newly added chapters shift every page's contents — same behavior as before this PR; refresh or re-add the novel after updating. No tested chapter is detected as locked, so
hideLockedis a no-op there. AI-authored change, no human review.Playground (real user path,
npm run dev:start+ dedicated browser)check:plugin).turning: finds Turning (WN) plus related titles, all with covers.book/turning: Turning (WN), Ongoing, author 쿠유, genres Romance/Mystery/Fantasy/Adventure/Action, summary present, TOTAL PAGES 30.1 chapter - Character Sheetfirst) — the site's own short oldest page, one ajax request.book/chapter/126140: 9685 chars rendered text ("Perhaps this place needs more of such individuals…"); same path via bundled plugin parses to 11061 chars. First browser attempt transiently showed "Failed to fetch" while the test browser was crashing; retry after relaunch succeeded.On the Greptile P2 (pagination regression coverage)
This repo has no unit-test suite and no place for one to run: no test runner in dependencies, no
testscript inpackage.json, no test files or directories, no unit-test CI workflow (only Lint, Format, Plugin Live Check), anddocs/testing.mdprescribes only the livecheck:pluginplus the browser playground. The live checker requests only page 1 for every page-plugin — a harness-wide property ofscripts/live-check-plugin.js, not something a single-plugin PR can change without touching shared tooling for all ~278 plugins. So the multi-page mapping is verified outside the repo instead: full app-page→site-range mapping checked live on four novels — Turning 30 pages[21, 50x29], Best Friend X Swap 5 pages[2, 50x4], Regression Guidelines 10 pages[3, 50x9], Guild Princess 3 pages[17, 50, 50]— every page exactly one ajax request, zero missing or duplicated, per-page sizes matching the site's ranges exactly. A 28-novel live scan found no chapter count divisible by 50 (exact multiples are rare snapshots); the aligned case needs no separate code path since pages are selected by sorted index from the parsed partition, exact for any remainder. No test framework invented, no dead fixtures added.