Substack Broken Link Checker

A fast, async Python tool that finds broken links in your Substack newsletter archive, classifies them by type and produces a report you can act on.

Why This Tool?

Auditing a newsletter archive shouldn't cost $100+/month for tools like Semrush or Ahrefs. But those tools and especially free alternatives can generate a lot of false positives.

The latest updates to this tool fix that, based on measurements from four months of real reports: 49.6% of failures were HTTP 403 and 11.2% were HTTP 429 — bot protection and rate limiting on pages that load perfectly in a browser — against only 12.7% genuine 404s. A report like that is mostly noise and wastes your time.

This tool now sorts failures by whether they are real, and then by the work each one implies:

• Separates signal from noise — every failure is broken, blocked or inconclusive, so a bot-blocking host does not read the same as a dead page
• Turns a report into a work list — triage groups findings by what you would actually do about them
• Works with Substack's bot protection — uses your session cookie to authenticate as a logged-in user
• Handles large archives efficiently — async concurrent checking, 10-20x faster than sequential
• Tracks what you have already checked — incremental scanning, so repeat runs only cover new posts
• Safe to leave unattended — emails you when a scheduled run fails so you're not in the dark

Features

• Failure classification: broken / blocked / inconclusive, so the report distinguishes a dead page from a site that merely refused the checker
• Triage: group a report by the action each finding implies — edit the post, replace the link, re-check later, or ignore
• Fast: async concurrent checking (10-20x faster than sequential)
• Smart caching: same link across multiple posts? Checked once
• Retry logic: exponential backoff for transient failures
• Incremental scanning: track checked posts, only scan new ones
• Domain filtering: skip bot-blocking sites, auto-flag known broken domains
• Failure alerts: email notification when an unattended run dies
• Timestamped reports: each run's results survive rather than overwriting

Quick Start

# Install (provides the `substack-link-checker` CLI)
pip install git+https://github.com/jcddc83/substack-broken-link-checker.git

# Check all posts from 2024
substack-link-checker check --base-url https://YOUR.substack.com --year 2024

# Check posts from a file
substack-link-checker check --base-url https://YOUR.substack.com --url-file posts.txt

Installation

git clone https://github.com/jcddc83/substack-broken-link-checker.git
cd substack-broken-link-checker
pip install -e .

Or directly from GitHub:

pip install git+https://github.com/jcddc83/substack-broken-link-checker.git

This installs the substack-link-checker console command. Equivalent invocations:

• substack-link-checker check ...
• python -m substack_link_checker check ...

Requirements: Python 3.8+

Migrating from v1.0.0

v1.0.0 shipped flat scripts at the repo root. They have been reorganised into a substack_link_checker package with a subcommand CLI.

Old (v1.0.0)	New
python substack_link_checker.py ...	substack-link-checker check ...
python compare_posts.py ...	substack-link-checker compare ...
python import_checked_posts.py ...	substack-link-checker import ...
python fetch_archive_urls.py ...	substack-link-checker fetch-archive ...
python demo_link_checker.py	substack-link-checker demo

The four helper scripts (compare_posts.py, import_checked_posts.py, fetch_archive_urls.py, demo_link_checker.py) are kept at the root as thin back-compat shims, so existing python compare_posts.py ... invocations and the bundled PowerShell scheduled task continue to work. The main substack_link_checker.py script could not be kept as a shim because its name collides with the new package — use substack-link-checker check ... or python -m substack_link_checker check ... instead.

Authentication

If Substack blocks your requests or you need to check paywalled content, use your session cookie:

1. Log into your Substack in a browser
2. Open Developer Tools (F12) → Application → Cookies
3. Find the substack.sid cookie and copy its value
4. Provide it via the SUBSTACK_COOKIE environment variable (recommended) or the --cookie flag:

# Recommended: env var (keeps cookie out of shell history / ps aux)
export SUBSTACK_COOKIE="your-substack-sid-cookie-value"
substack-link-checker check --base-url https://YOUR.substack.com --year 2024

# Alternative: --cookie flag (visible in process listings)
substack-link-checker check --base-url https://YOUR.substack.com --year 2024 \
    --cookie "your-substack-sid-cookie-value"

Security: Treat the session cookie like a password. Prefer the env var so it does not end up in your shell history or in ps aux. See SECURITY.md for full guidance.

Note: Your session cookie might expire after a few weeks. If you start getting 403 errors, get a fresh cookie from your browser.

Usage

Basic Usage

# Check posts from a specific year (uses sitemap)
substack-link-checker check --base-url https://example.substack.com --year 2024

# Check posts from a URL file
substack-link-checker check --base-url https://example.substack.com --url-file posts.txt

# Verbose output with custom report name
substack-link-checker check --base-url https://example.substack.com --year 2024 \
    --verbose --output december_report.csv

Incremental Scanning (Recommended)

Track which posts you've already checked to avoid re-scanning:

# First run: checks all posts, saves history
substack-link-checker check --base-url https://example.substack.com --year 2024 \
    --history-file checked_posts.json

# Subsequent runs: only check new posts
substack-link-checker check --base-url https://example.substack.com --year 2024 \
    --history-file checked_posts.json --only-new

Domain Filtering

# Skip domains that block bots (assumed OK)
substack-link-checker check ... --skip-domains wikipedia.org

# Auto-flag domains as broken without checking
substack-link-checker check ... --broken-domains old.defunct-site.com

Finding Unchecked Posts

# Compare your sitemap against history to find unchecked posts
substack-link-checker compare https://example.substack.com checked_posts.json
# Outputs: unchecked_posts.txt

# Then check just those posts
substack-link-checker check --base-url https://example.substack.com \
    --url-file unchecked_posts.txt --history-file checked_posts.json

Example Output

$ substack-link-checker check --base-url https://example.substack.com --year 2024

Substack Broken Link Checker
==================================================
Base URL: https://example.substack.com
Concurrency: 10
Max retries: 3
Input: Sitemap
Year: 2024
==================================================

Found 45 posts from 2024
[1/45] Processing: https://example.substack.com/p/my-first-post
  Checking 12 links (10 new, 2 cached)...
  3 failed checks (1 genuinely broken) in this post

[2/45] Processing: https://example.substack.com/p/another-post
  Checking 8 links (6 new, 2 cached)...
  0 failed checks (0 genuinely broken) in this post
...

Completed in 34.2 seconds

==================================================
SUMMARY
==================================================
Total links checked: 234
Links skipped (assumed OK): 8
Links auto-flagged broken: 0
Cache hits: 45
Retries performed: 3
Failed checks: 12
  genuinely broken: 5
  blocked (likely fine): 6
  inconclusive: 1

Generating report: broken_links_report.csv
Report generated: 5 broken, 6 blocked, 1 inconclusive

Understanding the report

The tool generates a CSV report with these columns, sorted most actionable first:

• category — how much to trust the finding (see below)
• post_title — title of the post containing the link
• post_url — URL of the post
• broken_link — the link that failed
• error_type — what went wrong (HTTP 404, DNS Failure, SSL Error, …)

Not every failed check is a dead link

A link check failing does not mean the link is dead. Across four months of real reports, 49.6% of failures were HTTP 403 and 11.2% were HTTP 429 — bot protection and rate limiting on pages that load perfectly in a browser — against only 12.7% genuine 404s. Reporting all of those the same way produces a report that is mostly noise and has to be triaged by hand before it is usable.

So every failure is sorted into one of three categories:

Category	Meaning	What to do
broken	The page is genuinely gone	Fix or drop the link
blocked	The server refused us; the link is probably fine	Usually nothing. Consider --skip-domains
inconclusive	A timeout or 5xx — says more about the moment than the link	Re-check later

Anything not positively identified as dead or blocked is inconclusive. The default is deliberately cautious: guessing "dead" would put work in front of you that does not exist.

Error types and how they map

Error type	Category
HTTP 404, HTTP 410	broken
Soft 404 — page loads but its title says otherwise	broken
DNS Failure — domain does not exist	broken
Malformed URL — two links concatenated in the post's HTML	broken
Known broken domain — auto-flagged via --broken-domains	broken
HTTP 401, 403, 406, 429, 451	blocked
Other HTTP 4xx / 5xx	inconclusive
Timeout, SSL Error, Connection Error	inconclusive

Turning a report into a work list

check sorts failures by how much to trust them. triage goes one step further and groups them by the action each one implies, which is a different axis — a dead target and a mangled href are both broken, but one needs a replacement link and the other needs the post edited.

substack-link-checker triage broken_links_report.csv triaged.csv

412 rows -> triaged.csv

  malformed_href       38 links across  21 posts
  retired_host         12 links across   9 posts
  dead_target          49 links across  31 posts
  inconclusive_other   61 links across  40 posts
  blocked             252 links across  88 posts

  actionable total: 99

Classes, most actionable first:

Class	What it means
malformed_href	Two URLs concatenated in the post's HTML. No dead target — edit the post. A suggested fix is included
retired_host	Host is on your --retired-hosts list
dead_target	The page is genuinely gone
inconclusive_other	Timeout or transient failure, not confirmed dead
blocked	Site blocks automated requests; the link is probably fine

--retired-hosts <file> takes hostnames you know are retired, one per line. This matters because a retired subdomain often serves a certificate that does not match, so the check aborts before reading a status and files the row inconclusive — when in fact it is dead. Only you know which hosts those are for the sites you link to.

Wayback and Library of Congress archive URLs legitimately embed a second http:// in their path, so they are not mistaken for concatenations.

Running it on a schedule

run_link_checker.ps1 drives a monthly Task Scheduler run. Copy secrets.ps1.example to secrets.ps1 (gitignored) and fill it in — that file holds your Substack URL, your cookie if you need one, and the alert credentials, so nothing personal ends up in a tracked file.

Set up the failure alerts. A scheduled run that fails silently is worse than no scheduled run, because "no broken links this month" and "the run died on its first post" look identical from the outside. With NOTIFY_EMAIL and NOTIFY_PASSWORD set, a failed run emails you — and calls out an expired session cookie specifically, since that is the failure that recurs. Any SMTP server works via NOTIFY_SMTP_HOST / NOTIFY_SMTP_PORT; it defaults to Gmail, which needs an App Password rather than your account password.

Reports are written with a timestamped filename by default, so each run's results survive rather than overwriting the last.

check Subcommand Options

Flags for the check subcommand. Run substack-link-checker <subcommand> --help for the others.

Option	Short	Description
--base-url	-b	Your Substack URL (required)
--year	-y	Year to check (uses sitemap)
--url-file	-f	File with post URLs (one per line)
--output	-o	Output CSV filename (default: broken_links_report.csv)
--concurrency	-c	Parallel requests (default: 10)
--timeout	-t	Request timeout in seconds (default: 10)
--max-retries	-r	Retry attempts for failures (default: 3)
--history-file	-H	JSON file for tracking checked posts
--only-new		Only check posts not in history
--skip-domains	-S	Domains to skip (assumed OK)
--skip-domains-file		File with domains to skip (one per line)
--broken-domains	-B	Domains to auto-flag as broken
--broken-domains-file		File with domains to auto-flag (one per line)
--cookie	-C	Substack session cookie for authentication
--verbose	-v	Show detailed progress
--limit	-l	Max posts to check

Subcommands

Command	Purpose
substack-link-checker check	Main link checker (the command shown throughout this README)
substack-link-checker triage	Group a report's failures by the work each one implies
substack-link-checker compare	Find posts not yet checked (sitemap vs history)
substack-link-checker import	Import previous results from Excel/CSV into history
substack-link-checker fetch-archive	Extract URLs from the /archive page (fallback when the sitemap doesn't work)
substack-link-checker demo	Self-contained demo against a handful of known-good/bad URLs
run_link_checker.ps1	Windows Task Scheduler automation (PowerShell)

Troubleshooting

Common failure modes and how to fix them:

HTTP 403 Forbidden when fetching the sitemap or post pages

Substack's bot protection is rejecting unauthenticated requests. In order of likelihood:

1. Set SUBSTACK_COOKIE (see Authentication above) so you're requesting as a logged-in user.
2. If you had a cookie set: it has probably expired (Substack rotates session cookies every few weeks). Grab a fresh one from DevTools.
3. If both are current: lower --concurrency (try --concurrency 3) so you look less bot-like.

Sitemap returns no posts for --year YYYY

The year-specific sitemap (e.g. /sitemap-2024.xml) doesn't exist for your Substack — some accounts only expose a single combined sitemap. Fall back to scraping the archive page:

substack-link-checker fetch-archive https://YOUR.substack.com 2024
# Produces archive_urls_2024.txt
substack-link-checker check --base-url https://YOUR.substack.com \
    --url-file archive_urls_2024.txt

DNS Failure or Timeout for links that work in your browser

The target site is rate-limiting or geo-blocking the checker, not actually broken. Add it to --skip-domains so it's assumed OK:

substack-link-checker check ... --skip-domains rate-limited.example.com

For a recurring list, put one domain per line in a file and pass --skip-domains-file path/to/file.txt.

Connection Error: ...ssl:default / SSL Error

The target host is using an old TLS version Python's ssl module no longer accepts by default. Usually the right call is to flag the domain as broken (it really is unreachable from a modern client):

substack-link-checker check ... --broken-domains old-tls.example.com

Many Soft 404 (page title indicates error) results that look fine

The detector matches phrases like "page not found" in the page <title>. If a legitimate post happens to have one of those phrases in its title, it'll be misflagged. Open the report, eyeball the URL, and if it's genuinely live, ignore those rows.

The CSV report file is empty / has only a header

Either no broken links were found (look for "No broken links found!" in the summary) or the run was interrupted before report generation. The tool only writes the CSV on a successful completion of all posts.

--only-new is not skipping anything

Make sure --history-file points at the same JSON file you used on the previous run. The history file is the source of truth for which posts have already been checked; without it --only-new has nothing to compare against.

License

MIT License - see LICENSE file.

Contributing

Issues and pull requests welcome at github.com/jcddc83/substack-broken-link-checker. See CONTRIBUTING.md for guidelines and SECURITY.md for reporting security issues.