feat: support daily and weekly dynamics windows - #21
Conversation
|
Follow-up: used the existing The phase-1 doc now states exactly why backfill remains unestablished: the saved 21 Aug file and a second same-day download shared 10 dates with identical values, but no later timestamp existed, so this cannot show whether values grow retroactively. A before/after download on different days is required. |
|
Live CDP evidence (2026-08-21): using the existing --resume-dir path with synthetic completed top_popular/top_related/regions entries collected DYNAMICS only, without weakening empty-export fail-closed. Daily succeeded: requested 2026-06-25..2026-08-19; raw UTF-8 BOM, semicolon, CR-only, actual CSV rows 22.06.2026..18.08.2026 (58 rows), manifest actual_period matches the first/last data rows, output dynamics_daily.parquet. The first attempt exposed a real stale mismatch (daily control/header but monthly rows); the collector now waits for the first table cell shape and rejects gaps. Weekly selector switched successfully, but the live weekly date-pick experiment returned the platform default window (header 29.07.2024..16.08.2026) despite requested dates; this is recorded as unresolved UI behavior rather than presenting requested dates as fact. Backfill remains inconclusive: same-day comparison only, no later timestamp. |
…indow) Live CDP measurements confirm Wordstat's weekly dynamics view silently ignores an explicit --date-from/--date-to and returns its own default ~2-year window instead (107 full weeks, unrelated to what was requested). Since a manifest recording a fabricated 'actual_period' that does not match the caller's intent would be the worst possible outcome, an explicit weekly period is now rejected up front by validate_period() with InvalidPeriodError, rather than collected and silently wrong. Weekly dynamics without an explicit period (Wordstat's default window) remains supported and was confirmed live to behave predictably. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
validate_period() now rejects an explicit weekly period before the browser is ever touched, so _set_period()'s 'week' popup_type and the matching branch in _select_calendar_date() are unreachable dead code. Removing them also collapses the duplicated month_label computation in _select_calendar_date() into one place, leaving a single function that drives both real remaining cases (month, day) instead of three parallel near-identical branches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
This reverts commit 6cdcc89.
…uested window)" This reverts commit 498ecff.
_collect_one always sleeps once per non-REGIONS view (settling for the export blob to catch up with a table repaint) and again on an empty-export retry — both are unconditional real-time pauses against the live UI, not _wait_for-style condition polling with an early return. Tests exercise _collect_one for real against a fake, instantly-responding page, so those sleeps were paid in full: 140 tests took 26.9s instead of well under 1s (root-caused by wordstat-1 from pytest --durations on PR #21). Both pauses are now constructor parameters (settling_seconds=1.0, empty_export_retry_seconds=2.0, matching prior hardcoded behavior) instead of hardcoded literals, so the CLI's real collector keeps its live-tested delays while tests can set them to 0. Full suite: 140 passed in 0.77s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
…eriod
_wait_for_table_granularity only checked the first cell's *format* (e.g.
"looks like a week range"), which the table's pre-existing content
already satisfied before a period change was applied -- the date-range
button showed the newly picked dates while the table itself still held
stale rows, and the exported CSV silently carried Wordstat's default
window instead of the requested one. This was the actual root cause
behind an earlier, incorrect conclusion that Wordstat ignores explicit
weekly periods (it doesn't: a live screenshot of the UI showed a
2018-12-24 explicit weekly period applied correctly by hand); that
conclusion has been reverted.
_set_period now waits for the first cell to actually start at the
expected date via a new _wait_for_period_applied, turning a silently
wrong-period export into a loud InterfaceChangedError if the picker
never catches up. Two related fixes surfaced during live verification:
- The dynamics table's first-cell month name is genitive for
daily/weekly ("22 июня", "24 декабря" -- "of June"/"of December") but
nominative for monthly ("август 2024"). Matching daily/weekly's
genitive text against the existing nominative RUSSIAN_MONTHS list
(used for clicking the calendar popups) never matched, so a first
attempt at this wait ran out its full timeout on every weekly/daily
request. Added RUSSIAN_MONTHS_GENITIVE and match the exact expected
month, not just any month word.
- Weekly's first cell is the Monday of the requested date's week, not
the date itself (confirmed live) -- not a bug, Wordstat's own alignment.
Live-verified against the real Wordstat UI:
- Weekly, requested 2018-12-24..2019-01-13: actual_period
24.12.2018..07.01.2019 (07.01 is the start of the last full week
ending 13.01 -- the period applies exactly, 3 full weeks, no
truncation).
- Daily, requested 2026-07-22..2026-08-20: actual_period
22.07.2026..19.08.2026 (29 of 30 rows -- the same variable trailing-tail
behavior already documented for daily dynamics in phase 1, not a new
defect).
143 passed in 0.68s, ruff clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
Per wordstat-1's decision: keep the default (1.0s, unchanged) but stop the comment from reading like a description of a caught bug. State plainly: introduced without a live timing measurement, no confirmed repro for this specific race, +1s/view (+3s/phrase, +50s on a 50-phrase batch), and why removing it can't currently be verified safe (issue #22 blocks a full CLI run from ever reaching DYNAMICS). Separately note that empty_export_retry_seconds is unlike it: it only fires after a concrete, already-observed empty-export trigger, not preventively. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
… the platform The earlier PR #21 revert removed a section claiming Wordstat ignores an explicit weekly period. That claim was wrong: it came from a live prompt that never actually passed date_from/date_to (requested_period was null in that run's own log), so the default window it observed was default because no period was requested, not because Wordstat rejected one. A user-provided live screenshot of the UI (24.12.2018 -- 13.01.2019, weekly, applied by hand) disproved the claim directly. The real defect was in our own _wait_for_table_granularity: it checked only that the first cell's format matched the granularity, which the table's pre-existing (stale) content already satisfied before a period change had actually been applied -- a silent wrong-period export, not a platform limitation. A first fix attempt had its own bug: comparing the table's genitive month text ("24 декабря") against the nominative RUSSIAN_MONTHS list used for calendar clicks ("декабрь") never matched. Documents, with live verification: - weekly period applies exactly (24.12.2018..07.01.2019 for a requested 24.12.2018..13.01.2019 -- 07.01 is the start of the last full week ending 13.01, not truncation) - daily period applies with the same known variable trailing tail already documented above in this file (29/30 rows, not a new defect) - the live-check method (collect_many + --resume-dir, dynamics-only, since top_popular's fail-closed still blocks a full CLI run) - the lesson: DOM control state and actual table/export content can lag each other on live Wordstat (same class of issue as #3, for period instead of view switching) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk
🔍 Local review (cycle 1) — round 7d7fd1f6-46fe-4894-8bad-5b3280a6428cReviewed locally (
Totals: 2 FIX, 2 SKIP, 1 UNVERIFIED. Finalization is blocked by the UNVERIFIED finding pending user input on how to resolve it. |
Codex review (cycle-review round 1) found that _wait_for_period_applied only confirms the requested window's start (date_from) before downloading; a table that has repainted its first row but not yet its last row could still export and be marked complete with a stale end boundary. Add a containment check in _assert_contiguous_dynamics_rows: exported rows must fall inside the requested window (shorter is legitimate -- the documented variable daily trailing tail -- but a row outside either boundary is not). Weekly's lower bound compares against the aligned Monday of date_from's week, not the raw date, matching Wordstat's own snapping behavior. Separately, a live CDP measurement (this cycle's investigation of a review finding) confirmed issue #6 step 3's "daily granularity outside the 60-day window ... reject with a domain error" is only half implemented: validate_period bounds window length but not how far date_from can be from today. The daily date-range picker's year-select popup only ever offers the current year, so a request like --date-from 2019-01-01 passed pre-flight validation and failed deep inside calendar-click code with an opaque InterfaceChangedError. Add the missing pre-flight bound, expressed relative to `today` (not "only the current year" -- that would wrongly reject legal windows in January) and consistent with the existing 60-day window-length check: PR #20's live-confirmed window 22.06.2026-20.08.2026 remains accepted. Tests: uv run pytest -q (147 passed), uv run ruff check (clean).
Pre-existing bug, unrelated to issue #6 (introduced in 94b1a66, batch mode). /review flagged that _PHRASES_FILE_ENCODINGS tried plain "utf-8" before "cp1251", and a phrases file saved with a UTF-8 byte-order mark decodes successfully under plain "utf-8" too -- just with the BOM character left attached to the first line. Neither resolve_phrases' line.strip() nor the collector's own phrase.strip() removes it (''.isspace() is False), so the first phrase silently carried an invisible leading character into the typed Wordstat search, the run-directory slug, and manifest.json. Fix: try "utf-8-sig" before "cp1251" (it strips a BOM if present and decodes identically to "utf-8" otherwise). Tests: uv run pytest -q (147 passed), uv run ruff check (clean).
🔍 R4 update: live CDP verificationFollow-up on the round-1 triage (previous comment): R4 was marked UNVERIFIED pending live confirmation of whether Wordstat rejects a valid-length daily window whose start is far in the past. Live measurement: the daily date-range picker's year-select popup, opened via CDP against the authenticated live UI, listed only the current year as an option. A request with Verdict revised: UNVERIFIED → FIX. This is also a gap against issue #6 step 3 ("daily granularity outside the 60-day window ... reject with a domain error"), which was only half-implemented — window length was bounded, but not how far date_from can be from today. Fixed in All 3 findings from round 1 are now resolved: C1 (FIX, |
Cycle-review round 2 findings against dc9b16b (round 1's fixes). Codex found the round-1 containment check (2a9689e) sat behind the same len(rows) < 2 early return as the pairwise contiguity loop, but containment only needs dates[0]/dates[-1] -- the same row, for a single-row export -- not a pair. A genuine 1-row daily/weekly export whose single date lay entirely outside the requested window passed _is_untrustworthy_empty_export (which only rejects 0 rows) and was written to Parquet and marked complete, unguarded. Fixed by hoisting the containment check above the row-count early return; the pairwise contiguity loop stays gated at >= 2 rows, since it genuinely needs a pair. Monthly stays exempt from both checks -- its period field is a nominative month name, not a parseable date, so validating its end boundary here would need a separate parser this PR does not add. /review flagged the monthly calendar path (.datepicker-month__years_select / .react-datepicker__month-text) as never live-confirmed, unlike daily/weekly. Live CDP verification against the authenticated Wordstat UI found two real bugs in that path, both now fixed: - month_label.capitalize() ("Январь") never matched the popup's actual lowercase nominative text ("январь"), failing every explicit --granularity monthly --date-from/--date-to request with InterfaceChangedError deep inside the click. - the monthly popup is a single shared range-picker, not two independent day/week-style popups: the intermediate DATE_RANGE_SELECTOR click between picking date_from and date_to closed it instead of reopening it. Skipping that click for the monthly path (day/week keep it -- their two-click flow is live-confirmed working) lets the still-open popup accept date_to directly. Live-verified end to end after both fixes: an explicit --granularity monthly --date-from 2024-01-01 --date-to 2024-06-30 request now collects 6 rows (январь-июнь 2024) with actual_period exactly matching requested_period. /review also re-raised (as RR1) the same weekly-CSV-format claim round 1 correctly SKIPped as R2 -- it recurred only because the doc that disproves it (docs/ISSUE6_PHASE1_RESEARCH.md, PR #20) isn't on this branch. Added an inline comment at the parse site citing that doc's confirmed byte-for-byte weekly CSV format, so a future review pass can check the comment instead of re-raising the same false positive against code alone. Tests: uv run pytest -q (151 passed), uv run ruff check (clean).
🔍 Local review (cycle 2) — round 2 triageReviewed locally (
Totals (round 2): 2 FIX, 2 SKIP, 1 HALLUCINATION. During investigation of the monthly calendar finding, live CDP verification also found the monthly popup is a single shared range-picker (not two independent popups like day/week) -- fixed in the same commit. Live-verified end to end: an explicit monthly window now collects the correct rows with actual_period matching requested_period exactly. Cycle 2 of 3 complete: 2 FIX, 2 SKIP, 1 HALLUCINATION. |
Cycle-review — round 3 (final, cycle 3/3) — round 57e1f8d8-530d-4ac3-b8f1-c35dafdc6a8cReviewers this round: built-in
On the CONFLICTING verdict (C3): this is the third round to raise some form of "monthly dynamics has no post- Round 3 outcome: zero FIX verdicts. No code changes this round. CI: see below. |
|
Decision on C3 (round 3, CONFLICTING): accepting the fail-open monthly behavior as a deliberate, documented tradeoff rather than closing it in this PR. Tracked in #24, and called out explicitly in the PR description under "Known limitation" so it doesn't rely on an inline code comment surviving future edits. No further code change in this PR for this item. |
* docs: record issue 6 live granularity measurements * docs: record variable daily data tail * docs: record phase 2 findings — weekly period rejected, backfill open, dead code removed Adds phase 2 findings to the phase 1 live-measurement document: - dtypes.py guards confirmed updated to real 'август 2024'/'22.06.2026' values; the _NUMERIC regex itself was not touched. - Weekly dynamics with an explicit period is not a partial UI glitch: Wordstat ignores the requested window entirely and returns its own default ~2-year range (107 full weeks) regardless of what was asked. validate_period() now rejects the combination outright. - Weekly dynamics without an explicit period was confirmed live to collect predictably (107 rows, factual period from CSV rows). - The now-unreachable weekly date-picker branch in collector.py was removed rather than left dead, collapsing a duplicated month_label computation into one shared function for month/day popups. - Backfill remains explicitly unestablished: the only comparison done was same-day, so it neither confirms nor rules out later revision of already-exported values. Flagged as an open question for a future time-separated remeasurement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk * Revert "docs: record phase 2 findings — weekly period rejected, backfill open, dead code removed" This reverts commit bc62b8b. * docs: correct phase 1 with the real cause -- our readiness check, not the platform The earlier PR #21 revert removed a section claiming Wordstat ignores an explicit weekly period. That claim was wrong: it came from a live prompt that never actually passed date_from/date_to (requested_period was null in that run's own log), so the default window it observed was default because no period was requested, not because Wordstat rejected one. A user-provided live screenshot of the UI (24.12.2018 -- 13.01.2019, weekly, applied by hand) disproved the claim directly. The real defect was in our own _wait_for_table_granularity: it checked only that the first cell's format matched the granularity, which the table's pre-existing (stale) content already satisfied before a period change had actually been applied -- a silent wrong-period export, not a platform limitation. A first fix attempt had its own bug: comparing the table's genitive month text ("24 декабря") against the nominative RUSSIAN_MONTHS list used for calendar clicks ("декабрь") never matched. Documents, with live verification: - weekly period applies exactly (24.12.2018..07.01.2019 for a requested 24.12.2018..13.01.2019 -- 07.01 is the start of the last full week ending 13.01, not truncation) - daily period applies with the same known variable trailing tail already documented above in this file (29/30 rows, not a new defect) - the live-check method (collect_many + --resume-dir, dynamics-only, since top_popular's fail-closed still blocks a full CLI run) - the lesson: DOM control state and actual table/export content can lag each other on live Wordstat (same class of issue as #3, for period instead of view switching) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCyJr8vFFDW1eSMyCQyNNk --------- Co-authored-by: axisrow <axisrow@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 participant
Implements phase 2 of #6.
Summary
--granularity daily|weekly|monthlyand--date-from/--date-todynamics.parquet; usedynamics_daily.parquetanddynamics_weekly.parquetавгуст 2024and22.06.2026Tests
pytest: 140 passedruff check .: cleanLive verification (CDP
http://127.0.0.1:9223)22.06.2026–18.08.2026, 58 rows, UTF-8 BOM /;/ CR-only."январь", matchesRUSSIAN_MONTHS, not.capitalize()'d), and the shared range-picker popup must not be re-clicked between the two date selections.top_popularis always empty on this platform, and fix: fail closed on empty top exports #19's fail-closed rule correctly refuses to publish it) — validated via a direct collector call bypassing the blocked view, not the full CLI.Known limitation (deliberate, tracked in #24)
Monthly dynamics exports are not validated against the requested window after export — unlike daily/weekly, which reject a stale or truncated table via a post-export containment check. This is a scope decision, not an oversight: closing it needs a small parser for the month-name period field (
"январь 2024"), which #24 tracks. Until #24 lands, a monthly dynamics collection can in principle write a stale/mismatched window ascompletewithout raising. Daily and weekly are not affected.PR #20 is now marked ready for review separately. This PR is intentionally not merged.