Skip to content

📜 Document Cloudflare Pages infrastructure #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lucasrodes merged 1 commit into from
May 19, 2026
Merged

📜 Document Cloudflare Pages infrastructure #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions INFRASTRUCTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Documentation infrastructure

This repo is the **umbrella** of OWID's technical documentation. Its content is small (a landing page plus links into the subprojects). What makes it special is that it owns the custom domain and acts as a **router** in front of subproject Pages projects, so one URL transparently serves docs from multiple GitHub repos.

```
docs-cf.owid.io (later: docs.owid.io)
│
▼
┌──────────────────────────────┐
│ Cloudflare Pages project │
│ owid-docs (this repo) │
│ ───────────────────────── │
│ _worker.js (router) │
└──────────────┬───────────────┘
│
┌──────────────────┼──────────────────────────┐
│ │ │
▼ ▼ ▼
/projects/etl/* /projects/owid-grapher-py/* everything else
│ │ │
▼ ▼ ▼
owid-etl-docs owid-grapher-py-docs this repo's
.pages.dev .pages.dev static assets
(owid/etl) (owid/owid-grapher-py)
```

## Repos involved

| Repo | Pages project | Path prefix |
|---|---|---|
| `owid/owid-docs` (this) | `owid-docs` | `/` (umbrella + router) |
| `owid/etl` | `owid-etl-docs` | `/projects/etl/` |
| `owid/owid-grapher-py` | `owid-grapher-py-docs` | `/projects/owid-grapher-py/` |

Each subproject repo owns:
- a `.github/workflows/deploy-docs-cf.yml` that builds the docs and uploads to its Pages project via `wrangler pages deploy`
- a sed-override of `site_url` in CI so the canonical/sitemap URLs reflect the production path on `docs-cf.owid.io/projects/<short>/`

Per-PR previews still work on each project's own `*.pages.dev` URL — the umbrella router is only invoked for traffic to the custom domain.

## `_worker.js` — the router

Lives at repo root, copied into `site/` by the deploy workflow so CF Pages picks it up at the deployment root. When present, it intercepts every request to the Pages project.

Responsibilities, in order:

1. **Legacy URL rewrite** — strip the `/en/latest` segment baked into old ReadTheDocs links and 301 to the canonical path.
2. **Subproject routing** — match a `SUBPROJECTS` prefix, `fetch()` the same path on the target Pages project, and stream the response back.
3. **Fallback** — `env.ASSETS.fetch(request)` serves this repo's own static assets (umbrella landing page, CSS, etc.).

To add a new subproject, append one line to the `SUBPROJECTS` map.

## Adding a new subproject

Concrete example: a hypothetical `owid/foo-docs` repo serving its docs at `docs.owid.io/projects/foo/`.

1. **Create the CF Pages project** (use CLI — the dashboard's "Upload static files" flow now creates a Worker, not a Pages project):

```bash
npx wrangler pages project create foo-docs --production-branch=master
```

2. **Add a workflow** in `owid/foo` mirroring [`.github/workflows/deploy-docs-cf.yml`](./.github/workflows/deploy-docs-cf.yml) from this repo or from `owid/etl`. Two things to adjust:
- sed-replace `site_url` to `https://docs-cf.owid.io/projects/foo/` (later `docs.owid.io/projects/foo/`).
- Stage the build into `staging/projects/foo/` before `wrangler pages deploy`.

3. **Add repo secrets** in `owid/foo`: `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID` (same values used by the other repos).

4. **Open a PR on this repo** appending the new prefix to `_worker.js`:

```js
const SUBPROJECTS = {
"/projects/etl/": "https://owid-etl-docs.pages.dev",
"/projects/owid-grapher-py/": "https://owid-grapher-py-docs.pages.dev",
"/projects/foo/": "https://foo-docs.pages.dev", // new
};
```

5. **Merge order** — `owid/foo` PR first (so the target Pages project gets its first production deploy), then this repo's worker PR. Reverse order is safe but `/projects/foo/*` will 404 in the interim.

## Common operations

### Cache purge (after risky changes)

CF aggressively caches both static assets and worker responses (including 301s). If something looks stale after a deploy:

CF dashboard → **owid.io zone** → Caching → Configuration → Purge Cache → Custom Purge → enter the specific URL(s) → Purge.

### Swapping the custom domain

Currently `docs-cf.owid.io` is bound to the `owid-docs` Pages project. To re-attach to another project (e.g., during emergency rollback), detach from the source first, wait ~30 s, then attach to the target.

### Production branch

Production deploys on a Pages project are only those uploaded with `--branch=<production-branch>`. The production branch is set when creating the project (`--production-branch=master|main`); changing it later via the dashboard requires re-deploying because CF does not auto-promote the latest deploy on the new branch.

### Required repo secrets

Each subproject repo + this one needs:

- `CLOUDFLARE_API_TOKEN` — custom token scoped to `Account › Cloudflare Pages › Edit` + `User › User Details › Read`.
- `CLOUDFLARE_ACCOUNT_ID` — visible in the CF dashboard URL.

One token + account ID pair works for all repos.

## Related deployments (unchanged today)

- **ReadTheDocs** — still serves `https://docs.owid.io/` and the subprojects. Builds untouched. Will be decommissioned once the CF mirror has been validated and `docs.owid.io` DNS is repointed.
- **GitHub Pages** — `.github/workflows/docs.yml` in this repo still publishes a copy. No active consumer; can be removed after the cut-over.
68 changes: 19 additions & 49 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,62 +1,32 @@
# OWID Documentation

**Work in Progress**
Umbrella site for OWID's technical documentation.

Technical documentation for Our World in Data projects.
Production URL during the parallel-hosting phase: **https://docs-cf.owid.io/**.
The current production URL on ReadTheDocs is unchanged: **https://docs.owid.io/**.

## Development
## What's in this repo

This project uses [uv](https://docs.astral.sh/uv/) for Python dependency management.
| | |
|---|---|
| `docs/` | Source markdown for the umbrella landing page + links into subprojects |
| `zensical.toml` | Site config (Zensical / Material) |
| `_worker.js` | Cloudflare Pages worker — proxies subproject paths to other Pages projects (`/projects/etl/*`, `/projects/owid-grapher-py/*`, …) |
| `.github/workflows/deploy-docs-cf.yml` | Builds + deploys this site to Cloudflare Pages on every push to `main` |
| `.github/workflows/docs.yml` | Legacy GitHub Pages deploy (will be removed after the RtD cut-over) |
| `.readthedocs.yml` | RtD build config (unchanged during the transition) |
| `INFRASTRUCTURE.md` | How the CF deployment fits together + how to add a new subproject |

### Setup
## Local development

```bash
# Install dependencies
uv sync

# Serve documentation locally (when Zensical is ready)
uv run zensical serve

# Build documentation
uv run zensical build
uv run zensical serve # http://localhost:8000
uv run zensical build --clean # output: site/
```

### Current Status

- **Build System**: MkDocs with Material theme (currently active)
- **Migration Target**: Zensical (Material team's new static site generator)
- **Zensical Status**: Alpha (expected stable release early 2026)

### Configuration Files

- `mkdocs.yml` - Current MkDocs configuration (actively used)
- `zensical.toml` - Future Zensical configuration (ready for migration)
- `pyproject.toml` - Python project dependencies

### Migration to Zensical

Zensical is the successor to Material for MkDocs, built by the same team. Key features:

- **5x faster** rebuild times
- **Backward compatible** - can read existing `mkdocs.yml`
- **Modern design** with new Disco search engine
- **Rust-powered** Markdown parser (coming 2026)

#### When to Migrate

- Monitor [Zensical roadmap](https://zensical.org/about/roadmap/)
- Wait for stable release (expected early 2026)
- Test with `zensical.toml` when CLI is available
- Material for MkDocs will be supported for at least 12 months

#### Migration Steps (when ready)

1. Ensure Zensical ≥0.1.0 is installed
2. Test build: `uv run zensical build`
3. Compare output with MkDocs build
4. Update CI/CD to use Zensical
5. Optionally switch to `zensical.toml` for new features
## Deployment

## Documentation
Pushes to `main` deploy to the [`owid-docs`](https://dash.cloudflare.com/?to=/:account/pages/view/owid-docs) Cloudflare Pages project. PRs get per-PR preview URLs at `<branch>.owid-docs.pages.dev`.

Visit the documentation at: https://docs.owid.io/
For the full picture — adding a new subproject, swapping the custom domain, cache-purge ops — see [`INFRASTRUCTURE.md`](./INFRASTRUCTURE.md).
Loading