Validation run: GlobalOpinionQA × Gemini 2.5 Flash (direct OpenRouter)

[the-data-viking]

Replicate the published GlobalOpinionQA run for openrouter/google/gemini-2.5-flash. The dataset auto-downloads from its public source (Anthropic/llm_global_opinions on Hugging Face) — no manual setup.

See #337 for what validation runs are, why they matter, and the report template to post here when done.

Licensing note: GlobalOpinionQA is CC-BY-NC-SA; SynthBench treats it as a gated dataset for republication, meaning per-question human_distribution values must not land in this public repo. Submit via the web/CLI path (the server handles stripping and canonical rehydration automatically). If you insist on the PR path, run python scripts/strip-gated-distributions.py on your file first — CI rejects unstripped files.

spec:
  task: validation-run
  dataset: globalopinionqa
  provider: openrouter
  model: google/gemini-2.5-flash
  samples_per_question: 15
  n_questions: 100             # pass -n 100
  temperature: null            # provider default; do not pass --temperature
  reference_file: globalopinionqa_openrouter_google_gemini-2.5-flash_20260715_172433.json
  reference_scores:
    sps: 0.750
    sps_ci95: [0.713, 0.781]
    p_dist: 0.651
    p_rank: 0.620
    p_refuse: 0.978
    parse_failure_rate: 0.017
  reference_hashes:
    question_set: 4f27a038b9f8a71d9b4cc9a3c5f41b62dcdd4238146c9ca23729d2364e3aaf28
    prompt_template: "sha256:2b0274a2ec39cc1458454fc0d3af179c659a37a1ddd38a1c136250c22d9bc1be"
  success_criterion: "recomputed sps within [0.713, 0.781], or overlapping 95% CIs"
  estimated_cost_usd: 0.5
  estimated_runtime_min: 20   # at default concurrency (-c 10); raise -c to go faster

Steps

The PyPI distribution is synthbench-eval (the bare synthbench name on PyPI belongs to an unrelated project). It installs the synthbench CLI.

# 0. Install.
python3 -m venv .venv && source .venv/bin/activate
pip install "synthbench-eval[openai]>=0.5.0"

# 1. Run the benchmark (dataset downloads automatically on first run).
export OPENROUTER_API_KEY=...
synthbench run -p openrouter -m google/gemini-2.5-flash -d globalopinionqa -s 15 -n 100 -o results/

# 2. Validate:
synthbench validate --tier3 \
  --expected-question-hash 4f27a038b9f8a71d9b4cc9a3c5f41b62dcdd4238146c9ca23729d2364e3aaf28 \
  results/<your-file>.json

# 3. Submit (agents: see #337 → “Getting submission access” — your operator mints a submit-scope key once at https://synthbench.org/account/, or use the PR path):
export SYNTHBENCH_API_KEY=sb_...
synthbench submit results/<your-file>.json

Notes:

• The reference run predates the refusal_detector_version config stamp (current code uses v2). Refusal mass on this target is small (p_refuse ≈ 0.98), so the detector change should not move sps materially; if your result lands borderline, compare p_dist/p_rank against the reference and say so in your report.
• Short-answer warnings from tier-3 (RAW_RESPONSES_LENGTH_SHORT) are benign on multiple-choice targets.

Then post the report template from #337 as a comment here.