bench: 2026-07 public benchmark refresh harness with fail-closed publication gates

[igerber]

Summary

• New benchmarks/refresh_2026_07/ harness for a fair, reproducible re-run of the published docs/benchmarks.rst comparisons (BasicDiD, TWFE, MultiPeriodDiD, CallawaySantAnna incl. MPDTA real data, SyntheticDiD vs R fixest/did/synthdid). Two-phase PR by design: this push is the measurement machinery; the timed run's committed results artifact (results/refresh_results.json), the regenerated page tables/environment block, and the remaining protocol-prose reconciliation land together as a second push on this same PR once the run completes on an idle machine. Until then the page's numeric content is intentionally untouched (the marker comments are inert), so the currently published tables remain self-consistent with the protocol prose that produced them.
• diff-diff arm = the RELEASED wheel (pip install diff-diff==3.7.0 in an isolated uv venv), never the dev tree: provenance hard-fails on wheel version/path (the benchmark scripts' sys.path.insert(0, repo_root) made dev-tree shadowing fully functional and invisible), backend-arm mismatch, or unexpected DIFF_DIFF_* knobs reaching the subprocess.
• Fairness enforced, not assumed: untimed in-process warm-up on BOTH sides (R's byte-compiler JIT previously sat inside the Sys.time window on every replication), fresh subprocess per replication, strictly sequential, thread-count env vars stripped and R run under --vanilla, SDID placebo bootstrap at 200 draws on both sides (the legacy orchestrator gave Python 50 vs R's default 200 - unfair to R), median of counted reps published with full distributions retained.
• Fail-closed publication gates, single hard-flag source of truth shared by the runner and gen_benchmark_tables.py (which refuses to render hard-gated, partial, or mixed-environment-fingerprint artifacts into the marker-bounded doc regions): per-rendered-arm ATT/SE/CI-overlap gates; per-period, per-(g,t), event-study, and group effect surfaces vs R; SDID id-aligned unit/time weight identity at 1e-8; MPDTA known-answer (-0.039951); replication determinism and non-finite-output gates (a CV rerun can never mask a correctness failure).
• BasicDiD/TWFE honesty: the legacy path labeled an interaction-OLS cell "BasicDiD/TWFE" while --type twfe was accepted-and-ignored by both arms; the refresh benchmarks BasicDiD and genuine absorbed-FE TWFE as separate gated cells, and the no-op flag now fails loudly everywhere.
• SDID: placebo-SE parity gated at a documented 35% Monte Carlo bound (R's placebo permutation is unseeded, so SEs agree in distribution rather than draw-by-draw; R's rep-to-rep SE values are recorded for audit). Weight comparison aligns by unit/period id - get_unit_weights_df() sorts by descending weight while R emits panel order, so a naive positional comparison fabricates ~1e-2 divergence out of pure ordering; id-aligned, weights reproduce R at <= 5e-13, and the page's weight-parity prose now cites the machine verification.
• run_version_story.py: repo-internal released-wheel 3.5.3-vs-3.7.0 comparison runner (not part of the docs site).

Methodology references (required if estimator / math changes)

• Method name(s): No estimator code changes. Benchmark-policy documentation added to the SyntheticDiD registry section (Arkhangelsky et al. 2021 Algorithm 4 placebo variance; benchmark SE gate is Monte Carlo-bounded; id-aligned weight gates).
• Paper / source link(s): Arkhangelsky, Athey, Hirshberg, Imbens & Wager (2021), "Synthetic Difference-in-Differences", AER 111(12); R packages fixest, did (Callaway & Sant'Anna 2021), synthdid.
• Any intentional deviations from the source (and why): SyntheticDiD benchmark SE parity uses a documented 35% Monte Carlo bound instead of the standard 10% because R's vcov(method="placebo") permutation is unseeded (draw-by-draw parity holds only when the permutation sequence is threaded explicitly, verified at <1e-8 in tests/test_methodology_sdid.py); see the REGISTRY.md SyntheticDiD note.

Validation

• Tests added/updated: tests/test_benchmark_refresh_gates.py (45 regression tests: CI-overlap-cannot-waive-SE, both-arm gating, id-aligned weight comparison, non-finite replication gating, CV-rerun masking, environment-fingerprint mixing, generator refusal, estimator-field wiring, hard/soft flag classification).
• Backtest / simulation / notebook evidence (if applicable): Small-scale end-to-end smoke of every cell (both backends + R): all parity gates pass (MPDTA ATT diff 1.8e-13; TWFE absorbed pair ATT diff 7e-15; per-(g,t)/event-study/group surfaces at ~5e-11; SDID id-aligned weights <= 5e-13); smoke artifacts are local-only (gitignored), committed results follow with the timed run.

Security / privacy

• Confirm no secrets/PII in this PR: Yes (provenance redacts non-allowlisted env values; committed artifacts carry path tails only, no local usernames)

---

Generated with Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01GPX5Rv8ozQXPdUV23QTfjr

[github-actions]

Overall Assessment

⚠️ Needs changes — one P1 methodology-gate issue. No estimator math changes were introduced, and the SyntheticDiD 35% placebo-SE benchmark gate is documented in REGISTRY.md, so that deviation itself is not a defect.

Executive Summary

• P1: SyntheticDiD weight parity is documented as id-aligned and fail-closed, but the comparator can silently fall back to positional comparison when IDs are missing.
• The new ATT/SE/CI gates, hard-flag rendering refusal, environment fingerprint checks, and estimator-field checks are directionally sound.
• No P0 findings found.
• Documentation still contains some pre-refresh BasicDiD/TWFE/protocol wording; P3 only.
• I attempted to run tests/test_benchmark_refresh_gates.py, but pytest is not installed in this environment.

Methodology

Finding M1

Severity: P1
Location: docs/methodology/REGISTRY.md:L2358-L2368, benchmarks/refresh_2026_07/refresh_common.py:L128-L188, benchmarks/refresh_2026_07/run_refresh.py:L363-L376

Impact: The registry says SyntheticDiD unit/time weight gates are id-aligned and fail-closed on key mismatch/duplicates. But compare_weight_vectors() falls back to positional comparison whenever IDs are absent, and run_refresh.py only checks metrics["ok"], not metrics["aligned_by_ids"]. If either benchmark script regresses and stops emitting IDs, publication can still pass by positional order, undermining the documented weight-parity contract.

Concrete fix: Require id alignment for SDID publication gates. In run_refresh.py, fail when not metrics["aligned_by_ids"] as well as when not metrics["ok"], or add a require_ids=True parameter to compare_weight_vectors() and use it for SDID. Add a regression test where weights are positionally identical but IDs are missing and assert a hard sdid_weights_gate_fail.

Code Quality

No additional findings.

Performance

No findings. The warm-up and sequential subprocess design are appropriate for the benchmark goal.

Maintainability

Finding MA1

Severity: P3
Location: benchmarks/refresh_2026_07/refresh_common.py:L831-L853

Impact: ensure_mpdta() uses assert for fixture integrity checks. Those checks disappear under optimized Python (-O), weakening fail-closed behavior.

Concrete fix: Replace the assert statements with explicit if ...: raise RuntimeError(...) checks.

Tech Debt

No blocking findings. The deferred timed artifact/doc refresh is acceptable as long as it lands before publication or is tracked outside PR prose.

Security

No findings. The provenance redaction path avoids committing full local paths and non-allowlisted DIFF_DIFF_* values.

Documentation/Tests

Finding D1

Severity: P3
Location: docs/benchmarks.rst:L52-L67, docs/benchmarks.rst:L86-L107, docs/benchmarks.rst:L323, docs/benchmarks.rst:L956-L967

Impact: The page now mentions the refresh policy but still contains old “3 replications” and combined “BasicDiD/TWFE” prose/tables until the results artifact is generated. This is not a correctness blocker, but it can confuse readers if merged before the second push.

Concrete fix: When committing refresh_results.json, update remaining prose to reflect the split BasicDiD vs absorbed-FE TWFE cells and the new refresh protocol. If intentionally deferred, add a TODO.md entry.

Path to Approval

1. Make SDID weight gates require id alignment, not just matching positional vectors.
2. Add a unit test that missing unit_weight_ids / time_weight_ids hard-fails even when raw weights are positionally equal.
3. Keep or extend the existing out-of-order ID-alignment test to prove valid id-aligned comparisons still pass.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: a30765632c7770bcbf6aba83ca206b9bb539102e

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings.

Executive Summary

• Previous P1 is resolved: SDID weight gates now require ID alignment via require_ids=True.
• The 35% SyntheticDiD placebo-SE benchmark gate is documented in REGISTRY.md, so it is P3 informational, not a defect.
• Hard-gated payloads, mixed environment fingerprints, non-finite replications, and stale flagged cells are now covered by runner/generator checks.
• Phase-2 benchmark artifact/doc reconciliation is tracked in TODO.md, so remaining stale benchmark prose is not a blocker.
• Tests were not run locally because pytest is not installed in this environment.

Methodology

Finding M1

Severity: P3 informational
Location: docs/methodology/REGISTRY.md:L2345-L2368, benchmarks/refresh_2026_07/run_refresh.py:L115-L135, benchmarks/refresh_2026_07/run_refresh.py:L365-L378

Impact: SyntheticDiD benchmark SE parity uses a wider 35% Monte Carlo-bounded gate for placebo SEs, while ATT and id-aligned unit/time weights remain tightly gated. This is explicitly documented with a registry Note and is therefore an accepted documented deviation.

Concrete fix: None required.

Previous P1 check: resolved. compare_weight_vectors() now supports require_ids=True, SDID publication gates pass it for both unit and time weights, and missing IDs fail closed rather than falling back to positional comparison.

Code Quality

No findings. The prior P3 concern about assert in ensure_mpdta() is resolved with explicit RuntimeError checks in benchmarks/refresh_2026_07/refresh_common.py.

Performance

No findings. Warm-up, sequential subprocess execution, CV flagging, and per-arm provenance checks are consistent with the benchmark-refresh goal.

Maintainability

No findings. The runner and table generator share hard-flag classification through refresh_common.hard_flags(), reducing drift between measurement and publication gates.

Tech Debt

Finding T1

Severity: P3 informational
Location: TODO.md:L46-L50

Impact: The timed refresh_results.json artifact and final benchmark prose/table regeneration are deferred to phase 2 of the same PR. This is tracked in TODO.md, so it is not a blocker under the deferred-work rules.

Concrete fix: Complete the tracked phase-2 item: run the gated timed refresh, commit refresh_results.json, regenerate marker-bounded benchmark regions, and remove the TODO row.

Security

No findings. Provenance redaction keeps local path details out of committed artifacts, and unexpected DIFF_DIFF_* knobs are rejected for publication runs.

Documentation/Tests

Finding D1

Severity: P3 informational
Location: docs/benchmarks.rst:L56-L67, docs/benchmarks.rst:L320-L445, TODO.md:L46-L50

Impact: Some benchmark page content remains pre-refresh until the phase-2 artifact lands, but this is explicitly tracked in TODO.md.

Concrete fix: No immediate approval-blocking fix. Follow the tracked phase-2 TODO.

Verification note: I attempted PYTHONDONTWRITEBYTECODE=1 pytest -q -p no:cacheprovider tests/test_benchmark_refresh_gates.py, but pytest is not installed in this environment.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 067d389c13c17491dbc6411aba13a48a76621342

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings.

Executive Summary

• Re-review confirms the prior SDID weight-gate concern is resolved: unit/time weights are emitted with IDs and publication gates require ID alignment.
• The SyntheticDiD 35% placebo-SE benchmark gate is documented in REGISTRY.md, mirrored in docs, and covered by tests, so it is P3 informational.
• Hard publication gates now cover per-arm ATT/SE/CI, detailed effect surfaces, non-finite replications, stale hard flags, and mixed environment fingerprints.
• BasicDiD and absorbed-FE TWFE are now split; the old no-op --type twfe path fails loudly.
• Remaining phase-2 benchmark artifact/prose work is tracked in TODO.md, so it is not approval-blocking.
• Local execution was limited: pytest and numpy are not installed here. Python AST parsing of the new harness/test files passed.

Methodology

Finding M1

Severity: P3 informational
Location: docs/methodology/REGISTRY.md:L2345-L2368, benchmarks/refresh_2026_07/run_refresh.py:L128-L134, benchmarks/refresh_2026_07/run_refresh.py:L317-L329

Impact: SyntheticDiD placebo SE parity uses a documented 35% Monte Carlo-bounded gate because R synthdid placebo permutations are unseeded. This is a documented methodology deviation, not a defect under the review rules.

Concrete fix: None required.

Finding M2

Severity: P3 informational
Location: benchmarks/refresh_2026_07/run_refresh.py:L350-L379, benchmarks/refresh_2026_07/refresh_common.py:L128-L175, benchmarks/python/benchmark_synthdid.py:L128-L145, benchmarks/R/benchmark_synthdid.R:L144-L149

Impact: Prior SDID positional-weight comparison risk is resolved. The Python and R scripts emit IDs, and the publication gate calls compare_weight_vectors(..., require_ids=True) for unit and time weights.

Concrete fix: None required.

Code Quality

No findings. The hard-flag taxonomy is centralized in refresh_common.hard_flags() and shared by the runner and generator, reducing drift risk.

Performance

No findings. Warm-up, sequential subprocess execution, fresh replications, CV rerun behavior, and thread-env stripping are consistent with the benchmark-refresh protocol.

Maintainability

No findings. The generator refuses hard-gated, mixed-fingerprint, or partial publication artifacts by default at benchmarks/refresh_2026_07/gen_benchmark_tables.py:L43-L75 and benchmarks/refresh_2026_07/gen_benchmark_tables.py:L429-L447.

Tech Debt

Finding T1

Severity: P3 informational
Location: TODO.md:L48-L50

Impact: Timed refresh_results.json, regenerated marker-bounded docs regions, and remaining benchmark prose reconciliation are deferred to phase 2 of the same PR and explicitly tracked.

Concrete fix: Complete the tracked phase-2 TODO and remove the row.

Security

No findings. Provenance handling redacts non-allowlisted DIFF_DIFF_* values and slims local path prefixes before committed artifacts: benchmarks/python/utils.py:L18-L62, benchmarks/refresh_2026_07/refresh_common.py:L686-L696.

Documentation/Tests

Finding D1

Severity: P3 informational
Location: tests/test_benchmark_refresh_gates.py:L1-L10, tests/test_benchmark_refresh_gates.py:L242-L267, tests/test_benchmark_refresh_gates.py:L414-L455, tests/test_benchmark_refresh_gates.py:L490-L510

Impact: The new tests cover the key gates: SE mismatch despite CI overlap, documented SDID 35% bound, ID-aligned weights, and generator refusal on hard flags. I could not run them locally because pytest is missing; a direct helper import was also blocked by missing numpy. AST parsing of the new Python files passed.

Concrete fix: Ensure CI runs tests/test_benchmark_refresh_gates.py in the normal dependency environment.