📊 Scaffold the World Bank PIP September 2026 update (wb/2026-09-22)

[paarriagadap]

Written by Claude Opus 5 — @paarriagadap at the wheel.

Summary

Scaffolding for the September 2026 World Bank PIP release, built ahead of the announced
2026-09-22 publication date so the extraction can start the moment the data lands.

The data does not exist yet. This PR creates the wb/2026-09-22 chains and repoints the
consumers, but the snapshots have not been run and the vintage-dependent metadata still describes
the March 2026 release. CI and the staging build will be red until the extraction runs — that is
expected, not a regression. Do not merge before then.

Three chains bumped → wb/2026-09-22:

Dataset	From	Steps created
world_bank_pip (+ world_bank_pip_legacy)	2026-06-26	2 snapshots, meadow, 2 gardens, grapher
thousand_bins_distribution	2026-03-25	snapshot, meadow, garden
poverty_projections	2026-03-25	snapshot, meadow, garden, grapher

harmonized_national_poverty_lines is deliberately not included: it is a one-off Foster et al.
(2025) paper file with no downstream consumers and no tie to the PIP release cadence.

What to do on release day

Every vintage-dependent field carries a # TODO: block at the top of its .dvc naming exactly what
must change. The full runbook lives at workbench/world_bank_pip/release-runbook.md; the essentials:

1. Read the vintage from the API — do not assume 20260922. The announced release date and the
   vintage string date are not the same: September 2025 shipped as 20250930.
   curl -s "https://api.worldbank.org/pip/v1/versions?format=json" lists them. Current newest is
   20260324_{2021,2017}_01_02_PROD. Cadence is twice yearly, late March / late September;
   2011 PPP stopped at 20240627, so only 2017 and 2021 are still published.
2. GLOBAL_DIST_1000BINS_URL in snapshots/wb/2026-09-22/world_bank_pip.py and the
   url_download in thousand_bins_distribution.dta.dvc are the same file — they must be bumped to
   the same vintage together, or regional percentiles get built from a different release than the
   country data.
3. Percentile file links (_fetch_percentiles) are usually the last thing the WB publishes. If
   they lag, set EXTRACT_COUNTRY_PERCENTILES = False, run, and re-run later with it back on.
4. Poverty lines (poverty_lines_countries) — check the run output's list of countries missing
   from the percentile file and confirm no headcount is below 0.99.

DAG

Both chains are active on purpose. Every consumer now points at 2026-09-22; the outgoing
2026-06-26 / 2026-03-25 blocks are kept, marked # OUTGOING VERSIONS, so the old datasets can
still be rebuilt and diffed against the new extraction —
etlr data://garden/wb/2026-06-26/world_bank_pip --force --only needs a reachable DAG entry, and the
update workflow requires diffing the new garden against a freshly rebuilt old one on the same
upstream catalog. They get removed and archived once the new chain is validated.

The frozen wb/2025-04-14 block (2011/2017 prices) is untouched.

Consumers repointed

data://explorers/wb/latest/world_bank_pip · data://external/poverty_inequality/latest/world_bank_pip
· data://external/poverty_inequality/latest/thousand_bins_distribution ·
viz://chart/wb/latest/{poverty,incomes,gini,palma_ratio}_pip ·
viz://bespoke/poverty_inequality/latest/income_distribution ·
data://garden/poverty_inequality/2026-08-25/harmonized_income_distributions ·
data://garden/poverty_inequality/2025-10-23/historical_poverty ·
data://garden/poverty_inequality/2025-01-22/inequality_comparison ·
data://external/owid_grapher/latest/int_dollar_conversions · dag/growth.yml · dag/covid.yml

Code change beyond the version bump

/check-outdated-practices flagged the @click decorator stack in the PIP snapshot script. Removed
it in favour of the shape the wizard emits — def run(upload: bool = True, live_api: bool = True).
etls inspects the signature and passes upload by keyword, and it never passed --live-api, so
behaviour is identical. Also dropped the module-level LIVE_API = 1, which was dead code (the CLI
flag, now the parameter, is what selects the API).

The other hit — set_index in unpivot_table — is a false positive: it is an intermediate reshape
chained to .stack() on the next line, which the detector's negative lookahead cannot see.

owid-issues

One reminder now covers the whole family. update-wb-world_bank_pip.yml becomes the group
workflow, retitled "Data update: World Bank PIP + 1000 bins + poverty projections", with a
/update-dataset world_bank_pip thousand_bins_distribution poverty_projections pointer in the body.
thousand_bins_distribution previously had no scheduled issue at all.

The separate update-wb-poverty_projections.yml is deleted: its body was empty, so the issue it
opened said nothing, and its annual Oct 20 cron contradicted the dataset's own
update_period_days: 180 — it rebuilds on every PIP vintage, so the March/September schedule is the
right one.

Checks

make check, make unittest (962 passed) and etl d version-tracker are clean. Nothing is run
against data because there is none.

Still open

• Blocked on the release — the entire extraction, every diff and audit, the metadata refresh,
  the indicator upgrade, the archive of the outgoing blocks, and both announcements. Runbook §0–§7.
• Awaiting a decision: the four dated consumers
  (harmonized_income_distributions and inequality_trends_by_reference_year at 2026-08-25,
  historical_poverty at 2025-10-23, inequality_comparison at 2025-01-22) are now dated before
  the data they ingest, which CLAUDE.md's versioning-hygiene rule says should be fixed by bumping
  them. Bumping mints new variable IDs and needs a chart remap, so it is deferred until after the
  extraction rather than done blind.
• Also awaiting sign-off: the PIP cron fires on the 20th of March/September, before the
  observed vintages (20260324, 20250930) — candidate for moving a few days later so the issue
  opens after the release rather than before it. Left as-is for now.
• Nobody has checked: whether the September release changes the data's shape (new PPP version,
  changed poverty lines, restructured percentile files). The scaffolding assumes a drop-in version
  bump; if the release restructures anything, the copied steps are a starting point, not a result.

🤖 Generated with Claude Code

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs	Docs Preview

Login: ssh owid@staging-site-data-pip-update-2026-09

chart-diff: ✅

No charts for review.

data-diff: ✅ no differences

No differences found.

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with etl diff REMOTE data/ --include <dataset> --verbose.

Edited: 2026-09-22 16:46:25 UTC
Execution time: 5.25 seconds