bench: publish 2026-07 refresh results - regenerated benchmarks page + committed artifacts

[igerber]

Summary

• Phase 2 of the benchmark refresh (bench: 2026-07 public benchmark refresh harness with fail-closed publication gates #672 shipped the gated harness): the full timed run's committed results artifact and the regenerated public benchmarks page. All 20 cells (BasicDiD, TWFE, MultiPeriodDiD, CallawaySantAnna, MPDTA real data, SyntheticDiD x {R, pure-Python, Rust}) passed every publication gate with zero flags under a single environment fingerprint; gen_benchmark_tables.py --check verifies docs/benchmarks.rst is byte-exact with the committed artifact.
• Headline results (released diff-diff 3.7.0 wheel vs R 4.5.2 / did 2.5.1 / fixest 0.14.2 / synthdid 0.0.9, Apple M4 Max, medians of warm-started fresh-subprocess replications): CallawaySantAnna 4.6x faster than R at small scale rising to 15.5x at 10k units / 14.8x at 20k - the advantage now GROWS with scale (previously published numbers showed it shrinking); SyntheticDiD's Rust backend (the default install) is 18-55x faster than R at matched 200-replication placebo variance, correcting the invalidated published table that showed Rust slower than pure Python; MPDTA real-data validation ~5x faster with exact ATT match (-0.039951).
• Honest regressions from the fairness fixes are published as-is: with R's byte-compiler JIT warm-started out of the timing window, fixest wins the sub-35ms BasicDiD/TWFE interaction-OLS cells at 10k-20k (0.6-0.9x); at equal placebo-replication counts pure-Python SyntheticDiD is 0.5-4.4x vs R (the old 2.4-16.5x claim reflected a 50-vs-200 replication asymmetry). SDID SE gaps of 3.1-11.5% are placebo Monte Carlo dispersion, documented and gated per the registry note; id-aligned unit/time weights reproduce R at <= 5e-13.
• Prose fully reconciled with the regenerated tables: median/warm-up protocol wording, split BasicDiD vs absorbed-FE TWFE tables plus estimator-mapping row, rewritten Rust-backend and SyntheticDiD notes, honest Key Observations, corrected stale claims (gelsy solver, 14.4x MPDTA, 2-17x BasicDiD, "optional" Rust intro), refresh harness documented as the source of the published tables with the legacy runner labeled, llms.txt Rust line updated to the measured range.
• Internal optimization story committed (results/version_story.{json,md}, repo-internal, not in the docs toctree): CallawaySantAnna runs ~4.9-5.0x faster on the 3.7.0 wheel than 3.5.3 at every measured scale (360k through 2M rows) with identical estimates; BasicDiD/MultiPeriodDiD context rows at 1.15-1.31x. Report headline is data-derived and suppressed unless every CS cell is flag-free.
• CHANGELOG carries the public before/after; the delivered phase-2 TODO row is removed.

Methodology references (required if estimator / math changes)

• Method name(s): No estimator code changes; benchmark artifacts and documentation only. SyntheticDiD placebo variance per Arkhangelsky et al. (2021) Algorithm 4; CallawaySantAnna per Callaway & Sant'Anna (2021).
• Paper / source link(s): Arkhangelsky, Athey, Hirshberg, Imbens & Wager (2021), AER 111(12); Callaway & Sant'Anna (2021), J. Econometrics 225(2); R packages did, fixest, synthdid.
• Any intentional deviations from the source (and why): SyntheticDiD benchmark SE parity gated at the documented 35% Monte Carlo bound (R's placebo permutation is unseeded; REGISTRY.md SyntheticDiD note); all other tolerances standard.

Validation

• Tests added/updated: No test changes in this PR (the 48 gate-regression tests shipped with bench: 2026-07 public benchmark refresh harness with fail-closed publication gates #672 and pass against this artifact).
• Backtest / simulation / notebook evidence (if applicable): Committed benchmarks/refresh_2026_07/results/refresh_results.json - 20 cells, zero flags, all parity gates (ATT/SE/CI per arm, per-(g,t)/event-study/group surfaces at ~5e-11, SDID id-aligned weights <= 5e-13, MPDTA known answer) green; gen_benchmark_tables.py --check clean against the regenerated page.

Security / privacy

• Confirm no secrets/PII in this PR: Yes (committed artifacts carry redacted path tails only; no local usernames)

---

Generated with Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01GPX5Rv8ozQXPdUV23QTfjr

[github-actions]

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings. No estimator, math, weighting, variance/SE, identification, or default-behavior code changes were introduced.

Executive Summary

• Changed surface is benchmark artifacts/docs plus run_version_story.py; no estimator implementation changes.
• SyntheticDiD placebo SE differences are documented in docs/methodology/REGISTRY.md as Monte Carlo-bounded, so the 11.5% max SE gap is not a defect.
• Committed refresh_results.json has 20 cells, zero flags, one environment fingerprint, and SyntheticDiD SE gaps under the documented 35% gate.
• No absolute local paths or usernames were found in the committed JSON artifacts.
• Minor documentation terminology issue: several changed prose lines call SyntheticDiD’s matched placebo-replication counts “bootstrap counts.”

Methodology

• Severity: P3-informational, mitigated/documented
  Finding: SyntheticDiD benchmark SE differences are intentionally Monte Carlo-bounded, not draw-by-draw parity. This is documented in the registry note for SyntheticDiD benchmark SE gates, and the PR’s results stay within that documented gate.
  Impact: No methodology defect; no action required.
  Concrete fix: None required.

Code Quality

No findings.

Performance

No findings. The headline speedups I spot-checked against the artifacts match the committed medians: CallawaySantAnna is ~4.6–15.4x vs R in refresh_results.json, and the version story shows ~4.9–5.0x CS speedup from 3.5.3 to 3.7.0.

Maintainability

No findings.

Tech Debt

No findings. The completed benchmark-refresh TODO row was removed and no untracked deferred correctness issue was introduced.

Security

No findings. The committed JSON artifacts redact Python paths with .../venvs/...; scans found no /Users/..., /home/..., or similar absolute local path leakage.

Documentation/Tests

• Severity: P3
  Finding: SyntheticDiD prose uses “bootstrap counts” even though the documented benchmark method here is placebo variance / placebo replications: CHANGELOG.md:L38-L47, docs/benchmarks.rst:L533-L538, docs/benchmarks.rst:L619-L621.
  Impact: Minor terminology mismatch; the numbers and methodology notes are otherwise consistent, but this can confuse readers because SyntheticDiD also has a distinct bootstrap variance method.
  Concrete fix: Replace “bootstrap counts” with “placebo-replication counts” or “resampling-replication counts” in those lines.

Validation note: git diff --check passed. I could not run python benchmarks/refresh_2026_07/gen_benchmark_tables.py --check in this review container because numpy is not installed, so I verified the artifact consistency by parsing the committed JSON directly.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 483834b209b6ccbc95b73161259be49c3dda9ced

---

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings. The PR changes benchmark artifacts, generated benchmark prose, TODO cleanup, and the internal version-story report; I found no estimator/math/default-behavior changes.

Executive Summary

• Prior P3 terminology issue is resolved: changed prose now says “placebo-replication counts,” not “bootstrap counts.”
• SyntheticDiD SE gaps are covered by the documented registry note: 35% Monte Carlo-bounded gate for independent placebo draws.
• refresh_results.json has 20 cells, zero flags, one environment fingerprint, and headline ratios matching the refreshed prose.
• version_story headline gating now suppresses the CallawaySantAnna headline unless all Callaway cells are clean.
• No user-home absolute paths or secrets found in the changed artifacts; only a non-sensitive /opt/homebrew/... BLAS path is recorded.
• git diff --check passed; gen_benchmark_tables.py --check could not run because numpy is unavailable in this container.

Methodology

• Severity: P3-informational, mitigated/documented
  Finding: SyntheticDiD benchmark SE differences are Monte Carlo-bounded rather than draw-by-draw parity. This is explicitly documented in docs/methodology/REGISTRY.md:L2345-L2361, and the changed docs repeat the 35% gate at docs/benchmarks.rst:L121-L128.
  Impact: No methodology defect; current max SE gap is 11.5%, under the documented 35% bound.
  Concrete fix: None required.

Code Quality

No findings. The new run_version_story.py headline guard aggregates arm flags into cell flags and only emits the headline when every CallawaySantAnna cell is clean: benchmarks/refresh_2026_07/run_version_story.py:L234-L278.

Performance

No findings. The published timing tables are generated from per-arm median fields, consistent with the stated protocol: benchmarks/refresh_2026_07/gen_benchmark_tables.py:L123-L127 and benchmarks/refresh_2026_07/gen_benchmark_tables.py:L261-L287.

Maintainability

No findings.

Tech Debt

No findings. The completed benchmark-refresh TODO row was removed, leaving no new untracked deferred correctness issue: TODO.md:L46-L52.

Security

No findings. The JSON artifacts redact Python executable/package paths with .../venvs/...; scans found no /Users/..., /home/..., or private key/token patterns in the changed benchmark artifacts.

Documentation/Tests

No findings. The previous review’s SyntheticDiD terminology issue is fixed in CHANGELOG.md:L38-L47 and docs/benchmarks.rst:L533-L538, which now distinguish placebo variance / placebo-replication counts from bootstrap counts.

Validation note: git diff --check passed. I could not run python benchmarks/refresh_2026_07/gen_benchmark_tables.py --check because this review environment lacks numpy; I instead parsed the committed JSON artifacts directly and inspected the generator logic.