Skip to content

feat(velocity): tighten F1+F2+F3 cron to 10-min cadence + emit staleness_seconds - #3175

Open
0motionguy wants to merge 1 commit into
hardening/2026-05-31-wavefrom
feat/F123-cron-tightening-10min
Open

0motionguy wants to merge 1 commit into
hardening/2026-05-31-wavefrom
feat/F123-cron-tightening-10min

Conversation

@0motionguy

Copy link
Copy Markdown
Owner

Summary

Tightens the three velocity fetchers from their previous cron cadence (*/40 and two daily slots) to a 10-min cadence to match TrendShift's live-mentions refresh, and emits staleness_seconds on the star-activity-deltas payload so downstream consumers can drop stale slugs without a side channel.

Fetcher Old New
velocity-refresh (F1) */40 * * * * */10 * * * *
velocity-backfill (F2) 17 2 * * * 5,15,25,35,45,55 * * * *
star-activity-deltas (F3) 30 5 * * * 3,13,23,33,43,53 * * * *

The three writers of star-activity-deltas now run on staggered :03 / :05 / :10 marks within each 10-min window, so they never pile up on the same minute — refresh's re-read-before-merge race window stays the same width it always was.

Why now

Deltas evidence from the 2026-06-15 TrendingRepo crawl pipeline (toolbox PR #241) shows TrendShift's live-mentions board refreshes at ~10-min cadence (impact 4.0). At our prior */40 cadence the homepage velocity board could trail it by up to 40 minutes. Source of truth: https://trendshift.io/live-mentions

Cost honesty

  • velocity-refresh (cheap 1-call REST): 300 calls × 6/hr = 1.8k/hr — trivial for the 20-token GH pool's 100k/hr ceiling.
  • velocity-backfill (GraphQL page-back with PAGE_BUDGET=30000): the existing existingCoversWindows skip-guard already short-circuits repos whose 7d/30d resolve from prior runs, so the deep-walk budget converges to "refresh recent tier only" after the first full pass. Worst-case first day might draw ~5× the prior daily draw against the GraphQL pool; subsequent days settle.
  • star-activity-deltas (Redis-only recompute): 8-worker fan-out over ~5000 keys × 6/hr = 240k Redis ops/hr. Cheap.

staleness_seconds

StarActivityDeltasPayload.staleness_seconds?: number — emitted as Math.round((Date.now() - Date.parse(computedAt)) / 1000) at publish time (≈0). Field is optional for back-compat — pre-2026-06-15 payloads omit it. Downstream readers can recompute the live value from computedAt; this just makes the freshness contract explicit on-wire.

Out of scope

  • WEIGHTS in engagement-composite/scoring.ts — unchanged per task spec.
  • No changes to velocity-seed (the daily expensive walker stays daily).

Test plan

  • cd apps/trendingrepo-worker && npx tsc --noEmit — clean, no errors introduced.
  • vitest run blocked by pre-existing tinypool minThreads/maxThreads conflict in vitest.config.ts (unrelated to this PR; reproduces on base branch). Pure helpers tested by velocity-backfill.test.ts are untouched.
  • Observe one tick of each cron after merge: expect velocity-refresh published at :00,:10,:20,:30,:40,:50, velocity-backfill at :05,:15,:25,:35,:45,:55, star-activity-deltas at :03,:13,:23,:33,:43,:53.
  • Verify star-activity-deltas slug now carries staleness_seconds field (Redis CLI: GET ss:data:v1:star-activity-deltas | jq .staleness_seconds).
  • Monitor the GitHub token pool exhaustion alert for the first 24h after merge — first-pass velocity-backfill cost is bounded by PAGE_BUDGET but the per-day GraphQL draw will be higher than 17 2 * * * baseline until the skip-guard converges.

Refs: toolbox PR #241 (deltas evidence pipeline).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

…ess_seconds

Match TrendShift's live-mentions refresh cadence (2026-06-15 deltas evidence,
impact 4.0) so the velocity board never trails third-party leaderboards by
more than one tick.

- velocity-refresh:      */40    → */10                (300 cheap GH calls × 6/hr)
- velocity-backfill:     17 2 *  → 5,15,25,35,45,55    (interleaved from :05)
- star-activity-deltas:  30 5 *  → 3,13,23,33,43,53    (interleaved from :03)

The three writers of star-activity-deltas now run on staggered :03/:05/:10
marks so they never collide on the shared slug. velocity-backfill's
existingCoversWindows skip-guard keeps the PAGE_BUDGET (30k GraphQL points)
honest at the new cadence — once a repo's 7d/30d resolves, subsequent ticks
just refresh the recent tier instead of re-paging the full history.

Also: each fetcher now emits `staleness_seconds` alongside `computedAt` on the
star-activity-deltas payload so downstream readers carry the freshness
contract on-wire and can drop stale slugs without a side channel. Field is
optional on the interface for back-compat — pre-2026-06-15 payloads omit it.

Evidence: https://trendshift.io/live-mentions
Refs: toolbox PR #241 (deltas evidence pipeline)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant