Skip to content

bench: publish 2026-07 refresh results - regenerated benchmarks page + committed artifacts #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
igerber merged 2 commits into from
Jul 11, 2026
Merged

bench: publish 2026-07 refresh results - regenerated benchmarks page + committed artifacts #673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a153c86
bench: publish 2026-07 refresh results - regenerated benchmarks page …
igerber Jul 11, 2026
483834b
docs(bench): placebo-replication counts, not bootstrap counts
igerber Jul 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
mislabeled cell can never publish; the legacy `--type twfe` no-op on the
BasicDiD scripts now fails loudly (absorbed-FE TWFE has its own dedicated
script pair, and the refresh benchmarks it separately from BasicDiD).
Regenerated page numbers land together with the committed results
artifact once the timed refresh run completes.
The full gated run is committed (`results/refresh_results.json`) and
`docs/benchmarks.rst` is regenerated from it: CallawaySantAnna is
4.6-15.5x faster than R `did` with the advantage GROWING with scale
(15.5x at 10k units / 150k observations, 14.8x at 20k units / 360k
observations; the previously published numbers showed the advantage
shrinking); SyntheticDiD's Rust backend is
18-55x faster than R `synthdid` at matched 200-replication placebo
variance (the previously published table - captured before the current
Frank-Wolfe kernels and at unequal placebo-replication counts - showed Rust
SLOWER than pure Python); MPDTA real-data validation is ~5x faster with
exact ATT match. Honest regressions from the fairness fixes are
published as-is: warmed-up fixest now beats the sub-35ms BasicDiD/TWFE
interaction-OLS cells at 10k-20k (0.6-0.9x), and pure-Python
SyntheticDiD at equal placebo-replication counts is 0.5-4.4x vs R (the old
2.4-16.5x claim reflected a 50-vs-200 replication asymmetry). All
20 cells pass every publication gate with zero flags.

### Fixed
- **`docs/benchmarks.rst` SDID weight-parity claim now machine-verified (and
Expand Down
7 changes: 0 additions & 7 deletions TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,6 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m

### Testing / docs

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|
| Benchmark refresh phase 2 (same PR #672): run the gated timed refresh on an idle machine, commit `benchmarks/refresh_2026_07/results/refresh_results.json`, regenerate the marker-bounded regions of `docs/benchmarks.rst` via `gen_benchmark_tables.py`, and reconcile the remaining pre-refresh prose (protocol bullets "3 replications / mean ± std", combined BasicDiD/TWFE wording, SDID note under the perf table, Key Observations, "Reproducing Benchmarks" section, `llms.txt` speedup cross-references). Row removed by the phase-2 push itself. | `benchmarks/refresh_2026_07/`, `docs/benchmarks.rst` | #672 | Mid | Medium |

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|

---

## Deferred / Documented
Expand Down
Loading
Loading