Disaggregated scorecards on the leaderboard (Move 1)

[openclaw-dv]

Context

Surfaced by synthpanel-driven PMF survey (n=70, 4 persona packs, 2026-05-14). All 4 packs independently rejected a single-number leaderboard. Buyers want disaggregated subgroup scorecards — accuracy broken out by age × geography × education × question-type, with explicit error bands.

Q4 from the survey: 61% said they'd switch synthetic-respondent vendors based on a credible 3rd-party benchmark, but the open-ended evidence makes it conditional on seeing the subgroup breakdown.

What to build

For every benchmarked vendor row on synthbench.org, add an expandable calibration matrix:

Dimension	Cells
Age band	18–24, 25–34, 35–49, 50–64, 65+
Geography	Census regions (Northeast / Midwest / South / West) or finer if data supports it
Education	HS-or-less, Some college, BA, Grad
Question type	Numeric, Likert, Multi-choice, Open-ended-summarizable

Each cell: vendor accuracy ± confidence interval, sample size, and a colour gradient.

The headline single-number score stays (people scan), but it's secondary; the subgroup matrix is the lede.

Done when

• Per-vendor expandable matrix renders on synthbench.org/leaderboard/
• Backing data API endpoint returns the matrix data
• At least 2 dimensions are populated for all currently-listed vendors
• Leading indicator instrumentation: track % of leaderboard sessions that expand into per-vendor view

Leading indicator (success signal)

40% of synthbench.org sessions click into the per-vendor disaggregated view within 30 days of launch.

Out of scope (file as follow-ups if relevant)

• User-uploaded validation packs (separate move, see leaderboard issue stream)
• Reproducible run hashes (separate move, vendor-onboarding stream)
• Held-out validation set (separate move, integrity stream)

[openclaw-dv]

Status update (2026-07-14), post-sharpening campaign:

Partial groundwork now exists: per-topic score breakdowns render as expandable leaderboard detail rows (with the expand-bug fixed in #310), and ~16 SubPOP runs carry demographic_breakdown data. A first increment is now in flight: surfacing that existing demographic data as per-vendor scorecards with CIs + n + honest "not yet measured" states, a documented demographic_scorecard field in the leaderboard JSON API, and PostHog expansion events so the leading indicator (% of sessions expanding) is measurable.

What the full matrix still needs (not code): demographic-conditioned runs across all vendors for age×geography×education. Today only SubPOP-derived subgroups exist, on a minority of rows. That's a benchmark-running program (API spend + suite design), not a site feature — suggest treating it as the follow-up milestone once the scorecard UI ships and shows engagement.

[openclaw-dv]

First increment merged in #313: per-vendor demographic scorecards (19 subgroup cells across 7 dimensions from existing SubPOP conditioned runs, honest "Not yet measured" states for the rest), a documented demographic_scorecard field in the leaderboard JSON API (v1.2.0, additive), and PostHog expansion events (leaderboard_row_expanded, vendor_scorecard_viewed, vendor_scorecard_demographics_expanded) so the leading indicator is measurable. Leaving the issue open: the remaining done-when ("≥2 dimensions populated for all vendors") is blocked on running demographic-conditioned benchmarks across vendors — a benchmark-running program (API budget + suite design), not plumbing.

[openclaw-dv]

Second tranche merged in #323: demographic scorecard coverage went 1 vendor / 19 cells → 6 vendors / 69 cells (raw GPT-4o-mini, raw Gemini 2.5 Flash, raw Llama 3.3 70B, SynthPanel GPT-4o-mini, SynthPanel Gemini Flash Lite added; each with CREGION/EDUCATION/INCOME/POLPARTY/SEX × 2 subgroups on subpop, n=100 s=15). Total spend ≤$1.87 of the $20 budget.

Two findings for the remaining scope: (1) AGE is not derivable from SubPOP — demo_AGE.json is empty and AGE isn't among its 8 attributes; the issue's age dimension needs a different dataset (GSS microdata is the obvious candidate). (2) The tranche's first honest schema-v2 files exposed and fixed a validator bug (RAW_RESPONSES_MODE failed ~8% of honest v2 runs at s=15 — sound invariant now in place). Remaining open scope: RACE/RELIG/POLIDEOLOGY dimensions (cheap, same mechanism), age via another dataset, and the >40% expansion-rate leading indicator (PostHog events now measuring).

[the-data-viking]

Coverage tranche 2 merged in #327: 10 new runs for $0.23 metered — globalopinionqa now has second replications for all cheap raw models, Gemini Flash Lite gained its first goqa/subpop rows, and SynthPanel (Llama 3.3 70B) is a new config row across all three datasets. Combined with #326 (first GSS runs: gemini-flash 0.789 / random 0.735 / haiku 0.709 on 75 public-domain questions with fully public per-question artifacts), the dataset×model grid is materially denser. Remaining open scope unchanged (RACE/RELIG/POLIDEOLOGY dims, age via GSS microdata, expansion-rate metric).