audit-oe

Audit OpenEvidence MCP citations with parallel PubMed verification and structured accuracy scoring

Install

npx skills add htlin222/audit-oe-skill
npx skills add -g htlin222/audit-oe-skill        # global
npx skills add htlin222/audit-oe-skill --agent claude-code  # specific agent

What it does

This skill orchestrates a rigorous citation audit of OpenEvidence responses:

1. Queries OpenEvidence on any biomedical topic (with BibTeX + Crossref validation)
2. Parses citations and maps each inline reference to its supporting claim
3. Launches parallel verification agents (one per citation) that independently:
   • Verify paper existence via PubMed/bioRxiv
   • Fetch abstracts and full text where available
   • Score claim accuracy on a 1-5 scale
   • Assess evidence strength (study type, sample size, peer review status)
4. Produces a structured audit report with PASS/CAUTION/FAIL grading

Skill structure

audit-oe/
  SKILL.md                              # Skill definition
  README.md                             # This file
  LICENSE                               # MIT
  example/
    glp1-cv/                            # Example: GLP-1 agonists & CV outcomes
      original.md                       # Raw OE response
      assets/
        citations.bib                   # BibTeX from OE (14 citations)
      report.md                         # Completed audit report (score: 3.69/5.0)
  .github/
    workflows/
      release.yml                       # CI: package and release .skill

Example output

From the included example audit of "GLP-1 receptor agonists in reducing cardiovascular events":

Metric	Result
Citations verified	13/13 (100% exist)
Mean accuracy	3.69/5.0
Grade	CAUTION
Critical issue	Stroke claim misrepresented from Galli 2025

Protocol

This skill follows the vercel-labs/skills protocol. Each push to main triggers a GitHub Action that packages the skill as a .skill file and creates a release tagged with the commit SHA.

License

MIT