Skip to content

Test-awareness manipulation on the cascade (MedMCQA) #285

Description

@duckyquang

Replicate the Test-awareness manipulation on the cascade (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.

What it measures

Test-awareness as a manipulation on the peer/authority cascade, not just the blind metric.

What to do

Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the medmcqa adapter). Write results under experiments/medmcqa/results/ in the same JSON/JSONL convention as experiments/medqa/results/ (not CSV, so the analysis tooling reads them).

Acceptance

  • Results committed under experiments/medmcqa/ in the MedQA template's format.
  • The same headline statistic the template reports, on MedMCQA.
  • Verified against a real API run (no synthetic data, no smoke-test-only claims).
  • PR opened that closes this issue.

Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions