Feat/referee self inconsistency floor

[armaanvgrewal]

Closes #417

Referee self-inconsistency floor

Summary

This PR addresses the referee self-inconsistency floor experiment and hardens the supporting infrastructure used to run it.

The experiment now:

• measures self-inconsistency over explicitly declared answers only,
• tracks undeclared responses separately,
• uses a shared declaration-detection utility in the extraction layer,
• and applies a default Gemini output-token cap to prevent runaway generations from stalling runs.

Changes

Declaration-aware self-inconsistency accounting

Added declaration-aware answer tracking to the referee self-inconsistency experiment.

Per-row outputs now include:

• declared_1
• declared_2

Summary outputs now include:

• declared_pairs
• undeclared_pairs
• undeclared_draws

temp0_self_inconsistency_rate is now computed over declared pairs only.

Shared declaration-detection utility

Added a shared utility in benchmaxxing.extract:

declared_mcq_choice(...)

This makes it possible to distinguish:

• explicit answer declarations,

from

• parser fallback behavior.

The existing parsing behavior remains unchanged.

Gemini output-token limit

Updated GeminiBackend to apply a default:

max_output_tokens = 16384

This limit is merged into existing decoding settings and may still be overridden when needed.

Tests

Added and updated test coverage for:

• declaration detection,
• declared vs undeclared accounting,
• declared-only inconsistency metrics,
• Gemini default output-token behavior.

Validation

Targeted validation

159 passed
1 skipped

including:

• gateway tests,
• extraction tests,
• parse regression tests,
• MedQA parser compatibility tests,
• referee self-inconsistency tests.

End-to-end validation

A full 40-case self-inconsistency run completed successfully and produced:

{
  "n": 40,
  "declared_pairs": 39,
  "undeclared_pairs": 1,
  "undeclared_draws": 2,
  "stable_cases": 39,
  "unstable_cases": 0,
  "temp0_self_inconsistency_rate": 0.0
}

The run completed successfully and in reasonable time under the updated configuration and exercised the new declaration-aware reporting.

Commits

• Track declared answers separately from parser fallbacks
• Limit Gemini output tokens by default

[armaanvgrewal]

@sebasmos @Agastya191 @duckyquang @azzy-chemE @felipeocampoos @PeterLi-jpg

Review requested, thanks!

[sebasmos]

Read 336f78d by hand: extract.py, gateway.py, referee_self_inconsistency.py, and referee_threshold._Cache, which the runner imports and the diff does not show. Full suite keyless on the PR head: 1221 passed, 8 skipped. Ruff clean.

What holds up. declared_mcq_choice mirrors parse_mcq_choice's Heuristic 1 plus the single-letter reply exactly, and parse_mcq_choice itself is untouched, so no committed number moves. On the 38 unique degenerate completions in experiments/referee/results/call_cache.jsonl it flags exactly the 2 conditional declarations and leaves 36 undeclared, which is the audit's count. _Cache.complete keys on (model, temperature, draw, prompt), so draw 1 and draw 2 are two real calls and a 0.0 rate cannot come from a shared cache entry. _Cache builds GeminiBackend with no default_decoding, so the 16384 cap reaches this lane, and config = {**default_decoding, **decoding} does forward it to generate_content. Prompt and temperature are unchanged.

One blocker. The 40-case run in the description is not in the repo: no referee_self_inconsistency.jsonl, no referee_self_inconsistency_summary.json, no referee_self_inconsistency_cache.jsonl. The sibling lane commits all three (referee_threshold.jsonl, its summary, and referee_threshold_requery_cache.jsonl force-added past the *_cache.jsonl ignore, since MedQA is public). Without the rows nobody can check 39/1/2/0 or replay it keylessly, and the results directory is not ignored for the rows or the summary. Please commit the three files.

Two smaller asks. "cache_bypassed": true in the summary is a hardcoded literal and is false: the draws are cached under distinct keys, which is what makes a keyless replay possible. Drop it (test_summary_contains_metadata asserts it, so that test goes with it); new_api_calls_this_run already says what happened. And the PR says Closes #417 while items 4 and 5 are proposed as follow-ups. That split is fine, but please open the two issues before this merges and link them here, so they are not lost when #417 closes.

Test gap, small. The description says the cap "may still be overridden". It can, I checked both default_decoding={"max_output_tokens": 100} at construction and decoding={"max_output_tokens": 7} per call, but no test covers either; one line in test_gateway.py would pin it.

One note, not an ask. declared_mcq_choice does not run is_abstention first, so a refusal that also matches a declaration pattern would come back declared=True with an empty legacy answer. Zero of the 6,134 committed referee rows hit it, so it is a comment, not a defect.

[armaanvgrewal]

Thanks @sebasmos for the detailed review. I've addressed the requested changes and reproduced the relevant artifacts.

What holds up

Agreed and confirmed.

• declared_mcq_choice() continues to mirror parse_mcq_choice()'s declaration detection behavior (Heuristic 1 plus single-letter replies).
• parse_mcq_choice() itself remains unchanged, so existing parser behavior and previously committed metrics are unaffected.
• The declaration audit numbers remain unchanged and consistent with the reported audit counts.
• _Cache.complete() continues to key on (model, temperature, draw, prompt), so draw 1 and draw 2 remain independent cache entries.
• The Gemini output-token cap reaches this lane through the existing backend wiring, while prompt and temperature remain unchanged.

Reproducibility artifacts

Added the full self-inconsistency run artifacts requested for the 40-case validation run:

• experiments/referee/results/referee_self_inconsistency.jsonl
• experiments/referee/results/referee_self_inconsistency_summary.json
• experiments/referee/results/referee_self_inconsistency_cache.jsonl

In addition, I added a provenance bundle under:

runs/referee-self-inconsistency-medqa-n40/

containing:

config.json
versions.json
run_manifest.json
results.json
summary.md

The bundle records:

• run configuration,
• Git SHA,
• environment/library versions,
• RunManifest metadata,
• aggregate results,
• references to the canonical referee artifacts.

The canonical experiment outputs remain the referee result files above.

cache_bypassed

Removed the hard-coded:

"cache_bypassed": true

from:

• implementation,
• tests,
• generated summary artifacts,
• provenance bundle artifacts.

new_api_calls_this_run remains as the runtime-specific indicator of cache usage, as suggested.

Token-cap override test

Added a regression test covering override behavior for:

default_decoding={"max_output_tokens": 100}

and

decoding={"max_output_tokens": 7}

This now pins the documented precedence behavior and ensures the default cap remains overridable.

Validation after the change:

tests/test_gateway.py
22 passed, 1 skipped

Follow-up issues for reviewer items 4 and 5

I agree these are important follow-up investigations and should not be lost when #417 closes. I will open and link two dedicated follow-up issues:

Abstention/declaration note

Acknowledged. I reviewed the note regarding declared_mcq_choice() and is_abstention(). As noted, it is a theoretical edge case but did not occur in any of the reviewed referee artifacts, so I have not made any changes in response to that observation.

@sebasmos @Agastya191 @duckyquang @azzy-chemE @felipeocampoos @PeterLi-jpg: Review requested, thanks!

[armaanvgrewal]

@sebasmos I've opened dedicated follow-up issues for your items 4 and 5 so they'll remain tracked after #417 closes:

• Rescore referee arms from committed caches using declared-only accounting #419
• Audit multi-round prompt contamination from degenerate parser outputs #420