Truth Crucible

An adversarial multi-agent debate engine for contested questions, grounded in your own corpus with hard citation verification.

Don't ask an LLM "is X right?". Make two LLMs argue it out — using your corpus — and watch which side survives.

1. Why · The Core Idea

One sentence

Throw a contested social/philosophical question into a multi-agent adversarial arena. Make Pro and Con each pull citations from your pre-classified corpus, have an independent judge blind-score anonymized arguments, and emit verdict_class + multi-dimensional scores + a complete audit trail.

The problem with naive prompting

When you ask an LLM directly about contested issues, three failure modes emerge:

Naive prompt	Failure mode
"Which side is right on X?"	Returns the majority position from training data, dressing value disputes as factual conclusions
"Give 5 reasons for each side"	Symmetric enumeration looks balanced but does not differentiate argument strength
"You debate yourself, then conclude"	Same model arguing with itself converges to the median answer (the Degeneration-of-Thought problem)

What Truth Crucible does instead:

▎ Cite-or-lose         Every claim must carry {file, lines, excerpt}; grep failure → dropped
▎ Anonymous blind judging  Judge sees Side_A / Side_B, never knows which is PRO or CON
▎ Three-tier verdict   RESOLVED / CONTESTED_BUT_LEANING / UNRESOLVED — refuses fake binaries
▎ Multi-dimensional scoring   factuality / validity / evidence / responsiveness / novelty / robustness
▎ Self-declared vulnerability   Each claim must state its own weakness; honesty rewarded

What it explicitly does not do

Not a truth oracle — it gives relative scores within a stated framework
Not a bias eliminator — but it makes bias structured, traceable, auditable
Not RL-trained — pure prompts + protocols + tools
Not a chat-loop wrapper — it's an industrial pipeline with state machines, citation verification, and crash recovery

2. How · Architecture

Main flow: a five-stage state machine

flowchart LR
    INIT([INIT<br/>load corpus]) --> PROPOSE
    PROPOSE([PROPOSE<br/>each side files N claims]) --> CONFLICT
    CONFLICT([CONFLICT<br/>judge extracts conflict points]) --> DEBATE
    DEBATE([DEBATE × K rounds<br/>alternating push / rebuttal]) --> SCORING
    SCORING([SCORING<br/>blind multi-dim scoring]) --> REPORT
    REPORT([REPORT<br/>verdict + scores + caveats])

    DEBATE -.convergence check.-> SCORING
    DEBATE -.budget circuit-breaker.-> SCORING

    style PROPOSE fill:#e3f2fd
    style CONFLICT fill:#fff3e0
    style DEBATE fill:#f3e5f5
    style SCORING fill:#e8f5e9
    style REPORT fill:#fce4ec

Roles and tools

flowchart TB
    subgraph orch["Orchestrator"]
        SM["state machine / persistence / circuit-breakers"]
    end

    subgraph pro["PRO Side"]
        PL["Pro Lead<br/>opus/sonnet"]
    end

    subgraph con["CON Side"]
        CL["Con Lead<br/>opus/sonnet"]
    end

    subgraph judges["Judges"]
        JC["Judge: Conflict<br/>opus"]
        JS["Judge: Score<br/>opus"]
    end

    subgraph tools["Tools (in-process MCP)"]
        T1["list_corpus"]
        T2["search_corpus<br/>(zh-Hans/Hant aware)"]
        T3["read_excerpt"]
        T4["submit_claims/conflicts/scores"]
        T5["WebSearch/WebFetch (optional)"]
    end

    subgraph storage["Storage"]
        E[("events.jsonl<br/>event stream")]
        D[("claims.db<br/>SQLite")]
        M["round_K.md<br/>final_report.md"]
        C["cache/*.json<br/>(resume)"]
    end

    SM --> PL
    SM --> CL
    SM --> JC
    SM --> JS

    PL --> T1
    PL --> T2
    PL --> T3
    PL --> T4
    PL --> T5
    CL --> T1
    CL --> T2
    CL --> T3
    CL --> T4
    CL --> T5

    JC --> T1
    JC --> T2
    JC --> T3
    JC --> T4
    JS --> T1
    JS --> T2
    JS --> T3
    JS --> T4

    PL --> E
    CL --> E
    JC --> E
    JS --> E
    PL --> D
    CL --> D
    JC --> D
    JS --> D
    SM --> M
    SM --> C

Data flow: a single claim from production to scoring

sequenceDiagram
    participant A as Lead Agent
    participant T as MCP Tools
    participant V as Verifier
    participant DB as SQLite
    participant J as Judge

    A->>T: search_corpus("background checks")
    T-->>A: hits + line numbers
    A->>T: read_excerpt(file, 12, 35)
    T-->>A: raw text
    A->>T: submit_claims({claim, evidence, logic_chain})
    T->>V: verify each evidence
    V->>V: grep(excerpt) within file:line_start..line_end
    V-->>DB: claim + evidence(verified=T/F)
    Note over DB: all claims complete
    DB->>J: anonymize (PRO/CON → A/B)
    J->>J: 6-dimension blind scoring
    J-->>DB: scores
    DB->>DB: median aggregation + verdict classification

3. What · Key Mechanisms

Mechanism	Why it's needed	How it's done
Cite-or-lose	LLMs invent plausible-looking citations	Each evidence must carry `{file, line_start, line_end, excerpt}`; normalized grep matching against the actual file. Fail → `verified=False` + failure reason recorded
Bidirectional zh-Hans/Hant matching	Chinese corpora often mix simplified and traditional	`zhconv` auto-expands queries (e.g. "管控" / "管控"), normalizes to simplified during verification
Anonymous blind scoring	Position bias / label bias / style bias	SCORING phase randomly maps PRO/CON → Side_A/Side_B; judge sees only the de-labeled claims
Three-tier verdict	Refuses to read a 0.02 gap as "truth"	Verdict classified by mean-score gap: `RESOLVED` (≥1.5), `CONTESTED_BUT_LEANING` (0.5–1.5), `UNRESOLVED` (<0.5)
Self-declared vulnerability	Agents tend to hide weakness, posture as strong	Each claim must populate `vulnerability`. Judge scores the `robustness` dimension by "did the agent flag a real weakness honestly?" Honesty raises the score
Turn budget + circuit breaker	SDK throws max_turns exceptions that swallow content	Even on exception, `capture.value` is preserved; prompts hard-instruct "must call submit before turn 50"
Resume cache	A 2-hour debate crashing mid-way is expensive	Each agent's submit payload is dumped to `cache/<actor>.json` immediately; `--resume <run_id>` skips completed stages
Auto-retry on rate limit	Claude Code's 5-hour window exhaustion crashes the whole run; unsupervised 10-hour debates need to survive limits	SDK exceptions are pattern-matched for rate-limit signatures (rate limit / 5-hour / 429 etc) → `RateLimitError` bubbles to `run_agent` → sleeps 10 min and auto-retries, up to 60 times (10-hour tolerance window). Each retry logs a `rate_limit_hit` event
Position-label inversion	The same source serves different sides depending on the question	The prompt explicitly tells agents: "look at the question first, then choose which classification folder to mine"
WebSearch (optional)	Local corpus may be insufficient	`protocol.enable_web_for_lead` toggle; `source_class=WEB` bypasses local grep, with the URL acting as `file_id`

4. Tech Stack

┌──────────────────────────────────────────────────────────┐
│  Application    orchestrator.py + report.py               │
├──────────────────────────────────────────────────────────┤
│  Agent layer    Claude Agent SDK (Python)                 │
│                 ├── ClaudeAgentOptions                    │
│                 ├── @tool / create_sdk_mcp_server (in-proc)│
│                 └── Uses your Claude Code login — no API key│
├──────────────────────────────────────────────────────────┤
│  Storage        JSONL events  +  SQLite claims.db  +  MD  │
├──────────────────────────────────────────────────────────┤
│  Verification   verify.py: zh normalization + grep anchor │
├──────────────────────────────────────────────────────────┤
│  Data           Pydantic models + zhconv                  │
└──────────────────────────────────────────────────────────┘

Core dependencies

Package	Version	Purpose
`claude-agent-sdk`	≥ 0.97	Agent runtime (uses your Claude Code subscription, no API key needed)
`pydantic`	≥ 2.5	Data models + validation
`pyyaml`	≥ 6.0	Topic + config files
`python-dotenv`	≥ 1.0	(legacy, no longer required)
`zhconv`	≥ 1.4	Chinese simplified/traditional bidirectional matching

Model assignments (default)

Role	Model	Rationale
Pro/Con Lead	`sonnet`	Heavy retrieval + argumentation; price/quality balance
Judge (Conflict + Score)	`opus`	Adjudication quality is paramount
Researcher (Phase 2)	`sonnet` or `haiku`	Document-sharded retrieval can be downgraded

5. Install & Run

Prerequisites

# 1. Install and log into Claude Code (subscription account)
npm i -g @anthropic-ai/claude-code
claude login

# 2. Python 3.10+
python3 --version

Install

git clone https://github.com/we1005/truth-crucible.git
cd truth-crucible
pip install -r requirements.txt

Run the bundled demo topic

# Default topic: "Should the United States enact strict gun control?"
# (uses bundled seed corpus + WebSearch supplement)
python main.py

# Limit rounds (development / debugging)
python main.py --rounds 1

# Validate corpus + CLI only, no API calls
python main.py --dry-run

# Resume a crashed run (cached stages auto-skipped)
python main.py --resume 20260430-120000-abc123

# Run only the topic-neutralization diagnosis (writes runs/<id>/topic_diagnosis.md)
python main.py --diagnose-topic

Visualization (Phase 3)

# 1. Argument attack-graph (open in Gephi / yEd)
python scripts/export_argument_graph.py <run_id>

# 2. Web UI dashboard (read-only, browses past runs + live SSE)
pip install fastapi 'uvicorn[standard]' sse-starlette
cd webui/frontend && npm install && npm run build && cd ../..
python -m webui.backend --serve-static    # single process: http://127.0.0.1:8000
# Or dev mode: python -m webui.backend --reload + cd webui/frontend && npm run dev

Cost reference

Configuration	Wall time	Total tokens	Cost (debits subscription credit)
Demo: 1 round / 3 claims per side	~10 min	~80 K	~$1–3
Full: 4 rounds / 5 claims per side / web enabled	~120 min	~400 K	~$15

6. Outputs (per run)

runs/<run_id>/
├── events.jsonl              # Append-only event stream (replayable, auditable)
├── claims.db                 # SQLite: claims / evidence / conflicts / scores
├── conflicts.md              # Conflict matrix (human-readable)
├── rounds/
│   └── round_<k>.md          # Per-round transcript
├── cache/
│   ├── pro_lead.json         # Each agent's raw submit payload
│   ├── con_lead.json         # for --resume
│   └── ...
├── final_report.md           # Final report: verdict + scores + caveats
├── full_transcript.md        # (optional export) full debate record
├── argument_graph.graphml    # (optional export) argument graph for Gephi/yEd
└── argument_graph.json       # (optional export) same graph as JSON for Web UI/debug

Export the argument graph

python scripts/export_argument_graph.py <run_id>

Outputs argument_graph.graphml + argument_graph.json:

Nodes = claim + conflict
Edges = attacks (claim→claim, from rebuttal chain) + participates_in (claim→conflict)
Claim nodes carry weighted_total and the six dimension scores (median across judges); colour/size by score in yEd

What a single claim looks like in SQLite

-- claims table
INSERT INTO claims (id, side, round, text, vulnerability, confidence, data) VALUES
  ('C-PRO-R0-01', 'PRO', 0,
   'Universal background checks have moderate evidence of suicide-rate reduction...',
   'Effect size varies considerably across meta-analyses depending on methodology...',
   0.75,
   '{"logic_chain": [...], "targets": [], ...}');

-- evidence table (must pass grep verification)
INSERT INTO evidence (claim_id, ord, file_id, line_start, line_end, excerpt, verified) VALUES
  ('C-PRO-R0-01', 0, 'PRO/seed_public_health.md', 32, 38,
   'A 2018 RAND Corporation review found ...', 1);  -- ✅ verified

-- scores table
INSERT INTO scores (claim_id, judge_id, weighted_total, data) VALUES
  ('C-PRO-R0-01', 'judge_score', 3.85,
   '{"factuality": {"value": 4.0, "comment": "..."}, ...}');

7. The Bundled Demo Topic

This public repo ships with a complete, runnable demo:

Should the United States enact strict gun control?

It includes:

config/topic.yaml — Topic definition (PRO position / CON position / context)
examples/sample_corpus/PRO/seed_public_health.md — Public health and comparative international evidence
examples/sample_corpus/CON/seed_constitutional.md — Second Amendment and the right to bear arms
examples/sample_corpus/CRITICAL_OF_BOTH/seed_pragmatic_critique.md — A pragmatist critique of both sides' rhetorical strategies
WebSearch enabled to auto-supplement with public sources (CDC, FBI, Heller opinion, etc.)

The point of Truth Crucible is not "decide who wins". It is to surface — structurally — both sides' strongest N claims, each chain of evidence, and the relative quality across 6 dimensions, so a reader can weigh them. When the system says UNRESOLVED, it is telling you: under this corpus and this scoring framework, this is genuinely undecided. Don't let either side's rhetoric trick you.

8. Use your own topic and corpus

Define a new topic YAML:

# config/my_topic.yaml
topic:
  question: "..."
  pro_position: "..."
  con_position: "..."
  context: "..."

corpus:
  root: "path/to/your/corpus"
  classes:
    PRO:    { dir: "supportive_books" }
    CON:    { dir: "opposing_books" }
    CRITICAL_OF_BOTH: { dir: "third_perspective" }
  analysis_suffix: "_analysis.md"

Run it:

python main.py --topic config/my_topic.yaml

On position-label inversion

The same source plays a different role in different questions. A book supporting movement X is PRO ammunition for "was movement X successful?", but might be CON ammunition for "was movement X externally influenced?". Truth Crucible tells the agent this in the prompt rather than hard-coding any source-to-side mapping — preserving flexibility across topics.

9. Design Principles (non-negotiable)

Doesn't settle academic disputes — emits scores, not truth
Doesn't think for the user — the report mandates a "limitations" section
No silent failures — exceptions all land in events.jsonl; agent_error / no_submit / verify_failure_reason are first-class structured records
Not vendor-locked — the agent layer is a thin abstraction (run_agent()); switching to OpenAI/Gemini is a ~200-line change
No frameworks-over-substance — no LangChain / CrewAI / AutoGen; we wrote the protocol layer ourselves, ~2000 readable lines total

10. Roadmap

Phase 1 — MVP: single judge, single lead per side (stable)

✅ Pro/Con Lead × 1 + Judge × 1
✅ State machine + cite-or-lose + zh bidirectional + resume cache
✅ Three-tier verdict + 6-dim scoring
✅ Exception-path capture preservation (max_turns errors no longer lose work)
✅ Subscription rate-limit auto-retry (10-hour tolerance window)

Phase 2 — Multi-judge + sub-agent clusters (implemented)

✅ N Researcher sub-agents per side (sharded by classification, parallel via ThreadPoolExecutor)
✅ N independent Judges, each with different anonymize seed → median aggregation
✅ Meta-Judge auditing inter-judge consistency + single targeted rescore (hard cap)
✅ Mandatory steelman protocol: round K≥2 first restates opponent at maximum strength
✅ Sycophancy detection (LLM judge with [0,1] score) + position-swap round (PRO argues CON's position)
✅ Cache schema_version: Phase 1 caches auto-invalidate under Phase 2 code

Enable in config/topic.yaml via researcher_count_per_side / scoring.judge_count / scoring.enable_meta_judge / protocol.enable_steelman / protocol.enable_position_swap. Unset defaults to Phase 1 behavior.

Phase 3 — Topic neutralizer + argument graph + Web UI (implemented)

✅ Topic-neutralization preprocessor: 6-dimension bias check; downstream prompts use the neutral version while the original is preserved in events.jsonl. Toggle with protocol.topic_neutralization: off|diagnose|rewrite; CLI --diagnose-topic runs only the diagnosis.
✅ Argument attack-graph: claim + conflict nodes / attacks + participates_in edges. Export with python scripts/export_argument_graph.py <run_id> → argument_graph.{graphml,json}.
✅ Web UI dashboard (FastAPI + React + Vite): run list + 6 tabs (events SSE / claims / conflicts / scores heatmap / react-flow graph / final report). Reuses src/graph.py so the schema cannot drift.
⏭ Cross-model judge cross-check (Gemini / GPT-4) — skipped to avoid external API cost.

11. Repo Layout

truth-crucible/
├── README.md                    ← English (you are here)
├── README.zh-CN.md              ← 中文版
├── pyproject.toml               ← package metadata
├── requirements.txt             ← 5 core + 3 Web UI dependencies
├── main.py                      ← CLI entrypoint
│
├── config/
│   └── topic.yaml               ← default demo topic (US gun control)
│
├── examples/
│   └── sample_corpus/           ← seed corpus for the demo topic
│       ├── PRO/
│       ├── CON/
│       └── CRITICAL_OF_BOTH/
│
├── prompts/
│   ├── pro_lead.md              ← Pro side system prompt
│   ├── con_lead.md              ← Con side system prompt
│   ├── judge_conflict.md        ← Judge: extract conflict points
│   ├── judge_score.md           ← Judge: blind multi-dim scoring
│   └── topic_neutralizer.md     ← Phase 3: 6-dimension bias detector
│
├── src/
│   ├── models.py                ← Pydantic: Claim/Evidence/Score/Verdict
│   ├── config.py                ← YAML loader
│   ├── corpus.py                ← Index + zh-aware search
│   ├── storage.py               ← JSONL + SQLite persistence
│   ├── verify.py                ← cite-or-lose grep verification
│   ├── agents.py                ← Claude Agent SDK glue + MCP tools
│   ├── orchestrator.py          ← state machine main loop
│   ├── sycophancy.py            ← Phase 2: convergence detector + position swap
│   ├── topic_neutralizer.py     ← Phase 3: topic-neutralization preprocessor
│   ├── graph.py                 ← Phase 3: argument-graph builder (CLI + Web UI shared)
│   └── report.py                ← Markdown report generation
│
├── scripts/
│   ├── export_full_transcript.py ← export full debate from SQLite
│   └── export_argument_graph.py  ← export argument graph (GraphML + JSON)
│
├── webui/                          ← Phase 3: read-only dashboard
│   ├── README.md                   ← API routes / start instructions
│   ├── backend/                    ← FastAPI + SSE
│   │   ├── app.py / __main__.py
│   │   ├── security.py             ← run_id regex + file allowlist
│   │   ├── routes/                 ← runs.py + stream.py (SSE)
│   │   └── services/               ← run_index / run_reader / graph_service
│   └── frontend/                   ← React + Vite + react-flow
│       ├── package.json / vite.config.ts
│       └── src/                    ← pages + components (events / claims /
│                                       conflicts / scores / graph / report)
│
└── runs/                        ← one directory per run (gitignored)
    └── <timestamp>-<hash>/
        ├── events.jsonl
        ├── claims.db
        ├── cache/
        ├── rounds/
        ├── conflicts.md
        ├── final_report.md
        └── full_transcript.md   (script-generated)

12. References

This project draws from:

Du et al., ICML 2024 — Improving Factuality and Reasoning through Multiagent Debate
Liang et al., 2023 — Encouraging Divergent Thinking through MAD (the DoT problem)
Irving & Christiano, 2018 — AI Safety via Debate
Anthropic — Building Agents with the Claude Agent SDK
PROClaim — Courtroom-Style Multi-Agent Debate — courtroom role decomposition inspiration
Geoffrey Huntley — Ralph technique — single-agent long loop orchestration (we borrowed the circuit-breaker / exit-detection ideas)

13. License

MIT — see LICENSE.

Truth Crucible v0.1 · Phase 1 MVP

Forge contested questions in the crucible — see whether the truth holds up under adversarial fire.

Navigation Menu

Repository files navigation

Truth Crucible

1. Why · The Core Idea

One sentence

The problem with naive prompting

What it explicitly does not do

2. How · Architecture

Main flow: a five-stage state machine

Roles and tools

Data flow: a single claim from production to scoring

3. What · Key Mechanisms

4. Tech Stack

Core dependencies

Model assignments (default)

5. Install & Run

Prerequisites

Install

Run the bundled demo topic

Visualization (Phase 3)

Cost reference

6. Outputs (per run)

Export the argument graph

What a single claim looks like in SQLite

7. The Bundled Demo Topic

8. Use your own topic and corpus

On position-label inversion

9. Design Principles (non-negotiable)

10. Roadmap

11. Repo Layout

12. References

13. License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
config		config
examples/sample_corpus		examples/sample_corpus
prompts		prompts
scripts		scripts
src		src
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
main.py		main.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

Truth Crucible

1. Why · The Core Idea

One sentence

The problem with naive prompting

What it explicitly does not do

2. How · Architecture

Main flow: a five-stage state machine

Roles and tools

Data flow: a single claim from production to scoring

3. What · Key Mechanisms

4. Tech Stack

Core dependencies

Model assignments (default)

5. Install & Run

Prerequisites

Install

Run the bundled demo topic

Visualization (Phase 3)

Cost reference

6. Outputs (per run)

Export the argument graph

What a single claim looks like in SQLite

7. The Bundled Demo Topic

8. Use your own topic and corpus

On position-label inversion

9. Design Principles (non-negotiable)

10. Roadmap

11. Repo Layout

12. References

13. License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages