fix: stop failing closed on always-empty top views, close orphaned tabs (#22, #9) - #25
Merged
Conversation
Issue #22: _is_untrustworthy_empty_export previously fail-closed on any empty TOP_POPULAR/TOP_RELATED/DYNAMICS export, on the premise that _select_view's pre-download row gate makes an empty CSV structurally impossible for any of the three. Issue #11 disproves that premise for TOP_POPULAR/TOP_RELATED: Wordstat returns them empty every single time, confirmed live, independent of the DOM state before the download click. Since TOP_POPULAR is the first view collected, every full collect() run failed on the very first view and never reached DYNAMICS/REGIONS. The predicate now only fail-closes DYNAMICS, whose emptiness is genuinely anomalous (24 rows across three live runs in #11, corroborated by PR #20). TOP_POPULAR/TOP_RELATED are recorded as normal zero-row exports instead. To keep issue #16's concern intact (a manifest must not read as an unqualified success when its data is empty), CollectionManifest gained a computed empty_views field, and status is now "incomplete" if either missing_views or empty_views is non-empty — so a manifest with two zero-row top views is visibly incomplete in manifest.json, not silently marked complete. Issue #9: collect_many created one CDP tab per CLI invocation via new_page() and never closed it; keep_alive=True correctly protects the user's own Chrome from being closed by session.stop(), but left our own tab orphaned every run. The tab is now closed via session.close_page() in the same finally block, before session.stop() (the CDP handle may be unusable after the session stops), guarded the same way session.stop() already is so a failed close doesn't discard collected results/failures. Verified live against CDP :9223 (not the default :9222): - single-phrase run reaches DYNAMICS/REGIONS; manifest shows status: incomplete, empty_views: [top_popular, top_related], dynamics: 24 rows, regions: 788 rows - --granularity daily collects dynamics_daily.parquet (59 rows) - two-phrase batch: one tab for the whole batch, 4 downloads total - failure path (bad region) after tab creation: tab count unchanged, failure surfaced to the CLI, exit code 1, no traceback - tab count via /json/list: 13 before and after every run above - pytest: 151 passed in ~0.7-0.9s (was 143 passed in 0.68s on main) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JYQKwtowJVXEkZxmsma3Wm
Owner
Author
🔍 Local review (cycle 1) — round 7895f235-f497-401c-bf4f-491e955174d2Reviewed locally (
No FIX verdicts this round. Both configured reviewers (claude, codex) answered. |
Owner
Author
📋 Review summary — all cycles
Totals: 0 FIX, 1 SKIP, 0 UNVERIFIED. Reviewed locally (built-in |
This was referenced Aug 21, 2026
axisrow
added a commit
that referenced
this pull request
Aug 21, 2026
) (#28) * fix: contain downloads to their own directory, keep partial results (#27) Two defects, both from live evidence (CDP :9223, daily/weekly/monthly, single-phrase and 2-phrase batch runs — see PR body for manifests): 1. session.downloaded_files (browser-use) can report a download at a path outside the collector's own downloads_path — observed live under the real ~/Downloads. finalize_raw then tried to replace()/unlink() that path, either crashing on macOS's TCC-protected ~/Downloads (Errno 1) or, worse, silently deleting a file it doesn't own. _resolved_file_snapshot now only admits downloaded_files entries that resolve inside downloads_path; _download_current_view detects an escaped path and raises a new DownloadEscapedError instead of ever touching it. finalize_raw gets a second, independent containment check (belt and suspenders) against run_directory/output_root. Cross-filesystem moves (finalize_raw, and _collect_one's own CSV rescue block) now use shutil.move instead of Path.replace (bare os.rename), which raises OSError across filesystems. Root-caused live: DYNAMICS and REGIONS use structurally identical blob:/target=_self export links, and several full 4-view live runs (in-repo and outside --output-dir, single phrase and batch) all completed cleanly — the escape is an intermittent Chrome-side race, not tied to a specific view or to how downloads_path is configured. Treating it as a loud, contained failure is the correct fix, not a workaround for something fixable upstream. 2. A single failing view (e.g. the above race on regions) used to fail the whole phrase, discarding views already collected and printing "Собрано 0 из 1" even when 3 of 4 were on disk with a valid manifest. _collect_one now catches a per-view failure (except AuthenticationRequiredError, which still aborts the whole batch — the session itself is gone) and returns a partial CollectionResult instead of raising, with a new view_errors field recording why each missing view is missing (including "не пробовался" for views skipped after an earlier one failed, not just the one that actually errored). A phrase where nothing at all was collected still raises, so the CLI can't count a fully failed phrase as a result. CLI output/exit code now key off missing_views + view_errors, not manifest.status (status is "incomplete" whenever top_popular/top_related are empty, which is every live run — see issue #22/#25 — so keying on it would fail every successful run again). Tests: 159 passed (was 151 on main), ~0.5s warm — no regression (#23). ruff clean. Manual mutation checks on finalize_raw's containment guard and _collect_one's view_errors guards/break path (mutmut 3.x would not resolve this project's src-layout package correctly in this sandbox — each mutant was applied and reverted by hand instead, confirmed killed by the relevant test each time). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTU6JGjjt2cZmH5ojqR9w4 * fix: surface same-tick escaped downloads, close resume false-success gap Cycle-review follow-up to PR #28 (issue #27): 1. _download_current_view now checks new_escaped on every poll tick, including the one that finds the legitimate CSV, not only inside the "nothing found yet" branch. session.downloaded_files is session-lifetime and append-only, so a stray escaped path masked by a same-tick success was never merely deferred to the next call — it was permanently absorbed into that call's before_escaped baseline and never surfaced again. The view's own successful download still returns normally (its data is fine); the escape is now reported as a non-fatal warning (CollectionResult.escaped_download_warnings) instead of being silently lost. 2. cli.py's partial-report gate now unions manifest.missing_views with view_errors instead of keying off missing_views alone. missing_views is a computed field over manifest.exports only; under --resume-dir a view can retain a stale export entry (its parquet manually deleted, its re-collection attempted and failed) — view_errors gets an entry for it but missing_views stays empty, so the CLI was reporting a run with a missing parquet as a full success at exit 0. This is the same "врёт о фактическом результате" failure mode issue #27 was about, from the opposite direction. Both were found and verified during this PR's local cycle-review (built-in /review + Codex companion, verdicts posted to the PR). Codex: approve, 0 findings. /review flagged a third, unreachable-in-practice TOCTOU gap in finalize_raw's containment guard — left as SKIP (documented in the PR comment), not worth the complexity for a path that cannot occur in the current call graph. Tests: 163 passed (was 159 before this commit) — mutation-verified by manually reverting each fix and confirming the new guard test fails for the right reason. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTU6JGjjt2cZmH5ojqR9w4 * fix: prune stale manifest export when resume re-attempt fails Cycle-review round 2 follow-up (Codex companion, high confidence): views_to_collect() re-attempts a view whose <view>.parquet is missing from disk even though manifest.exports still has a stale ExportSummary for it (e.g. the file was manually deleted after a prior run). If that re-attempt then failed too, the stale export entry was never removed — missing_views (a computed field over exports only) fell back to falsely reporting nothing missing for that view, letting a programmatic manifest consumer trust a "this view is fine" signal for a file that doesn't exist. Root-fixed by pruning any stale export entries for pending_views up front, before the retry loop runs — not only after a failed retry — so a crash/Ctrl-C mid-retry also leaves an honest manifest, consistent with this method's existing "write after every step" invariant. Other round-2 findings triaged as SKIP/IRRELEVANT/HALLUCINATION (posted to the PR): a cosmetic duplicate string build, a defensive-only rescue-block concern with no live bug, and a false claim about shutil.move vs Path.replace overwrite semantics (both silently replace an existing destination on POSIX — verified empirically). Tests: 164 passed (was 163) — mutation-verified by reverting the prune and confirming the new resume test fails for the right reason (status stayed "complete" with an empty missing_views despite the deleted parquet). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTU6JGjjt2cZmH5ojqR9w4 --------- Co-authored-by: axisrow <axisrow@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Aug 21, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Что и почему
Fixes #22, fixes #9. Порядок — #22 первым, т.к. он блокировал живую проверку самого себя (и #9): пока первый же вид (
top_popular) обрывал прогон,dynamics/regionsне собирались никогда.#22 — fail-closed на всегда-пустом
top_popular_is_untrustworthy_empty_exportсчитала пустой CSV недоверенным дляtop_popular/top_related/dynamicsразом, на основании структурной посылки: раз_select_viewдоказалTABLE_ROW_SELECTOR.length > 0перед кликом «Скачать», пустой экспорт для любого табличного вида — невозможное состояние.Issue #11 эту посылку опровергает фактом:
top_popular/top_relatedпусты у Вордстата всегда, это подтверждено вручную (клик по ссылке экспорта, мимо кода).top_popularидёт первым вVIEW_SELECTORS— значит каждый полный прогон падал на первом же виде, и инструмент не мог закончить сбор целиком.Решение (направление из issue: пометить вид несобранным / развести правила по видам):
dynamics(эмпирически подтверждено — 24 строки в трёх живых прогонах top_popular/top_related возвращают 0 строк данных при --keep-raw, хотя интерфейс показывает данные #11, PR docs: Issue #6 phase 1 live granularity research #20);top_popular/top_relatedзаписываются как обычныйExportSummaryсrow_count: 0— честно, не как «missing».Чтобы не потерять смысл issue #16 (
status: completeпри нулевых данных не должен молча проезжать),CollectionManifestполучил вычисляемое полеempty_views(виды сrow_count == 0), иstatusтеперь"incomplete", если непустоmissing_viewsилиempty_views. Так что манифест с двумя нулевыми топами виден как неполный, а не выдаётся за успех — закрывает и #16 для этих двух видов, отдельно ничего лишнего не трогал.#9 — осиротевшие вкладки
collect_manyсоздавала новую вкладку (session.new_page()) на каждый вызов CLI и никогда её не закрывала —keep_alive=Trueзащищает браузер пользователя, но заодно не даётsession.stop()закрыть и нашу собственную вкладку.Теперь вкладка закрывается через
session.close_page(page)в том жеfinally, доsession.stop()(CDP-хендл может быть непригоден после остановки сессии), с той же защитойtry/except, что иsession.stop()— сбой закрытия не выбрасывает уже собранныеresults/failures.page = Noneобъявлена доtry, чтобыfinallyне пытался закрыть вкладку, еслиsession.start()упал раньше её создания.Проверка — живой прогон, CDP
http://127.0.0.1:9223collector.pyне покрыт юнит-тестами by design; зелёный pytest — не доказательство. Проверено вручную:dynamics— 24 строки,regions— 788 строк. Манифест:top_popular/top_related—row_count: 0, честно видно, неcomplete).--granularity daily:dynamics_daily.parquet— 59 строк, собралось.PhraseFailureдошёл до вывода (Собрано 0 из 1), CLI вернул exit code 1 без traceback, вкладка не осталась.curl -s http://127.0.0.1:9223/json/list: 13 → 13 на каждом из прогонов выше (single run, daily run, batch run). Путь с ошибкой (одиночная фраза) — тоже 13 → 13.allowed_domainsне менялся, порт 9222 не трогался).Тесты
tests/test_collector_view.py: параметризованный тест на все три вида заменён —_is_untrustworthy_empty_exportтеперь fail-closed только дляDYNAMICS; добавлены отдельные тесты, подтверждающие, что пустойtop_popular/top_relatedэтим гейтом не флагуется.Что не расширял
Issue #24 (fail-open месячной динамики) не трогал — следующая отдельная задача, как и указано в брифе.