Skip to content

Fetch and store full article text for content analysis - #70

Open
alyssafuward wants to merge 1 commit into
mainfrom
69-fetch-full-articles
Open

Fetch and store full article text for content analysis#70
alyssafuward wants to merge 1 commit into
mainfrom
69-fetch-full-articles

Conversation

@alyssafuward

Copy link
Copy Markdown
Owner

Closes #69

Summary

  • Adds a fetch-articles command that pulls the full article body (body_html) plus subtitle, tags, wordcount, and audience per post via /api/v1/posts/{slug}
  • Stores an HTML-to-markdown conversion (body_markdown) alongside it in the posts table so the archive is directly queryable/readable
  • Reuses existing SUBSTACK_SID auth and the project's hard-stop-on-429 rate limiting (no retries/backoff)
  • Skips posts that already have a body unless --force is passed

Test plan

  • python check.py — all 9 checks pass
  • Ran python scraper.py fetch-articles end-to-end across both publications — 113/113 posts fetched, ~104K words total
  • Confirmed rate-limit hard-stop triggers and resumes cleanly on rerun
  • Confirmed no article content, cookies, or account info land in the diff (replies.db/config.py stay gitignored)

🤖 Generated with Claude Code

Adds a fetch-articles command that pulls body_html (+ subtitle, tags,
wordcount, audience) per post via /api/v1/posts/{slug} and stores it
alongside a markdown conversion in the posts table, so the full
article archive is queryable locally.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fetch and store full article text for content analysis

1 participant