Skip to content
BranchesTags

Repository files navigation

ML Research Loop

CI License: MIT MCP Python

MCP-native ML research loop for Codex and Claude.

ML Research Loop fuses the research-planning side of ml-intern with the fixed-budget experiment loop of autoresearch. Codex or Claude acts as the planner; this repository provides the local MCP tools, skills, artifact store, experiment runner, evidence checks, patch guards, and reproducibility signals needed to turn research ideas into bounded ML experiments. The next product layer is a Research Memory Layer: Graphiti for long-lived research relation graphs, Cognee as an optional experimental adapter for semantic retrieval over papers and proof artifacts, and a project-owned memory schema to keep provenance and claim boundaries explicit.

The canonical target architecture is documented in docs/product/target-architecture-cn.md. That document is the source of truth for future development: Codex/Claude is the planner, Skills define workflow policy, MCP executes, Runtime Artifacts remain the factual audit source, and Research Memory provides provenance-backed historical context without replacing proof archives or release gates.

Demo

ML Research Loop launch demo

Watch the 60-second launch demo: MP4 release asset. It shows MCP client acceptance, benchmark readiness, the evidence index, and a Codex-assisted PaperBench review with explicit claim boundaries.

Public Evidence

The current preview includes reproducible local proof artifacts, but does not claim official leaderboard scores. Full details are tracked in the benchmark evidence index and the proof release index.

Track Evidence Current result Public claim boundary
MLE-bench spooky-author-identification bridge smoke result Deterministic bridge smoke test: fake local scorer returns log loss 1.08468 for both baseline and patch rounds (patch content is not graded); proves the workspace/patch/grade/archive plumbing runs end-to-end, not any model score Plumbing smoke test only; no real model score, no real dataset, not a leaderboard claim
PaperBench rice debug harness Official debug split runs through dummy solver + dummy judge with zero failure categories Harness integration proof only; not reproduction quality
PaperBench review Codex-assisted review Codex-assisted rubric review records score 0.0 with evidence gaps Keyless review workflow only; not an official PaperBench score
Real paper pilot Real-paper proof index + release proof archive index MemFlow routing pilot and Adam optimizer pilot both run bounded public mini-slice baseline, ablation, client handoff, guarded iteration, dataset provenance, review report, and committed proof archives with SHA-256 artifact indexes Local public-slice proof with limitations review only; not full paper reproduction or official score
Full reproduction track fastText AG News user trial + release proof archive + proof release index Full AG News CSV + local official fastText binary produced baseline P@1=0.914; one bounded client-style proposal -wordNgrams 2 improved to P@1=0.916; P5 records 2 proposals, 1 failed allowlist rejection, 1 rollback event, and a sanitized downloadable proof tarball with SHA-256 Local reproducible core experiment, controlled patch-loop proof, multi-round failure/rollback evidence, and reviewed release bundle only; not a leaderboard score, not all paper tables, and not arbitrary automatic research improvement
HF external validation track HF external eval track + CP-Bench proof + target shortlist + CP-Bench P18 local proof + CP-Bench P18 manual gate + promotion path audit + DCP-Bench-Open P4 expanded eval + Qwen3-8B local diagnostics CP-Bench 是第一条 Hugging Face 可提交 proof 线。P0-P18 已完成 live verification、dry-run、dependency gate、proposal/submission gate、真实 local evaluator baseline、多轮候选、非 reference client solver 扩容、DCP reverse candidate、竞争力审计、manual submission gate 和一次真实公开上传。P18 本地在 62-row verified submission 上达到 final_solution_accuracy_percent=96.83、61/62 通过、runtime_success=62/62；公开 summary 已落地为 ml_research_loop_p18，final_solution_accuracy_percent=95.24、runtime_success=61/62、公开 result 快照中排名 2/18（来自可复核的公开 result 目录，不是官方认证 leaderboard 名次）。Promotion path audit 当前状态已切到 public_result_available_for_claim_review。由于 CP-Bench 上游已归档，DCP-Bench-Open v0.1.0 P4 扩容候选达到 128 条全运行成功、125/128 通过，完整 164 分母 final_solution_accuracy_percent=76.22，submitted-only 97.66，但 HF 公开组织页当前没有可提交榜单入口。Smol AI WorldCup 仍作为本地诊断和 proposal-loop 训练场。 CP-Bench 现在已有公开 summary，可做有边界的官方结果引用；同时仍需保留本地 proof 与公开结果的区分，仓库 artifact 继续保持 official_scores_claimed=false，避免把本地 96.83 写成公开榜单分数

Try v0.1.0-preview

git clone https://github.com/MagicianDu/ml-research-loop.git
cd ml-research-loop
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
python3 scripts/mcp_client_acceptance.py --python "$(which python3)"
ML_RESEARCH_LOOP_PYTHON="$(which python3)" \
python3 scripts/mcp_golden_path.py --max-experiments 1 --experiment-duration 30
ml-loop demo run --template byte-lm-smoke --runtime-root .demo_runs/byte-lm-smoke --json

Then connect Codex or Claude with ml-loop init-mcp-config and install the workflow skills with ml-loop init-skills. If you try the preview, please share feedback through GitHub Issues. The pinned preview thread is #1 Try v0.1.0-preview and share feedback. The short feedback guide is docs/preview-feedback-cn.md. 高校课程、科研机构和实验室试用请先阅读中文 机构 pilot 指南。指南包含 30 分钟本科实验、 2 小时硕博论文复现 mini lab、1 天实验室 benchmark trial、数据安全注意事项、 预期输入/输出 artifact 和失败反馈流程。Pilot 用户可以通过 pilot feedback issue template 提交结构化反馈。 If a run fails, attach a redacted diagnostics bundle:

ml-loop feedback-bundle \
  --runtime-root .demo_runs/byte-lm-smoke \
  --task-id demo-byte-lm-smoke \
  --output-dir .demo_runs/feedback-bundle

Why This Exists

Most LLM research agents can explain ideas. Fewer can give a strong model a safe, repeatable execution layer for:

  1. gathering paper, dataset, and code evidence;
  2. turning evidence into testable hypotheses;
  3. running local fixed-budget training experiments;
  4. reviewing metrics, logs, dataset state, and code state;
  5. proposing the next parameter or code patch;
  6. preserving artifacts for audit and reproduction.

ML Research Loop keeps those concerns separated:

Layer Responsibility
Codex / Claude Understand the goal, choose tools, inspect state, decide the next move
Skills Encode workflow policy, evidence thresholds, stop rules, and human review boundaries
MCP service Execute retrieval, experiments, review, patch preflight, logs, and artifact management
Research memory Reuse past paper reproduction, experiment, patch, failure, rollback, and proof-bundle experience through a project-owned schema with optional Graphiti/cognee adapters
Runtime artifacts Store tasks, results, workdirs, snapshots, logs, reproduction specs, and grade reports

The server does not implicitly call a service-side LLM. If you want server-side autonomous experiment planning, call run_ai_autoresearch explicitly and provide a configured provider.

Research memory suggestions are advisory only. The client planner still has to inspect provenance, decide the next action, and execute through guarded MCP tools such as run_client_patch_experiment, apply_client_code_patch, or the reproduction/benchmark proof lifecycle.

What Is Different

  • ml-intern capability kept: paper reading, research task planning, dataset/code evidence, provider coverage, source rankings, retrieval diagnostics, cache-aware recovery, and evidence citations.
  • autoresearch capability kept: program.md task instructions, bounded train.py workspaces, hyperparameter search spaces, accept/reject decisions, progress JSON, result JSON, logs, and snapshots.
  • MCP + Skills product shape: tools expose execution; skills teach Codex/Claude how to use the execution layer safely.
  • AIDE/PaperBench patterns absorbed: experiment tree, best-node tracking, loop policy, reproduction readiness, rubric-style grade reports. These are architecture patterns, not required runtime dependencies.
  • Research memory path selected: Graphiti + cognee are the planned optional adapters for long-term relation memory and semantic artifact retrieval, while ML Research Loop keeps the ResearchMemoryCard schema, provenance, privacy, MCP contract, and claim boundaries.
  • Patch execution is guarded: path sandboxing, stale-state checks, workspace-relative diffs, syntax/test preflight, rollback, and post-patch review are part of the public contract.

Quick Start

git clone https://github.com/MagicianDu/ml-research-loop.git
cd ml-research-loop
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

Run the MCP client contract check:

python3 scripts/mcp_client_acceptance.py --python "$(which python3)"

Run a bounded research-to-experiment demo:

ML_RESEARCH_LOOP_PYTHON="$(which python3)" \
python3 scripts/mcp_golden_path.py --max-experiments 1 --experiment-duration 30

Run a stable local template demo:

ml-loop demo list
ml-loop demo run --template byte-lm-smoke --runtime-root .demo_runs/byte-lm-smoke --json

Run the full local release gate:

ml-loop check --json

The full gate runs lint, tests, MCP stdio smoke, client acceptance, golden path, multi-round loop, auto-next, client patch, provider quality, real-data, real-code patch, and reproduction demos.

Inspect public benchmark proof-run readiness without running official evaluations or claiming scores:

ml-loop benchmark proof-plan --json

Inspect the Hugging Face external evaluation target shortlist without uploading or claiming scores:

ml-loop hf-eval shortlist --json
ml-loop hf-eval plan --target-id arguard-b1-binary-classification --output-dir .demo_runs/hf-eval/arguard-b1-plan --json

Write a read-only setup bundle for an external official proof-run environment:

ml-loop benchmark setup-bundle --output-dir .demo_runs/proof-setup --json

Validate future proof-run artifacts before public claims:

ml-loop benchmark publication-bundle \
  --manifest .demo_runs/proof-artifacts/manifest.json \
  --artifact-root .demo_runs/proof-artifacts \
  --output-dir .demo_runs/proof-publication \
  --json

Archive complete proof-run artifacts with hashes:

ml-loop benchmark archive-proof \
  --manifest .demo_runs/proof-artifacts/manifest.json \
  --artifact-root .demo_runs/proof-artifacts \
  --output-dir .demo_runs/proof-archive \
  --json

The same proof lifecycle is also exposed through MCP tools for Codex/Claude: get_benchmark_harness_probe, plan_benchmark_proof_run, write_benchmark_proof_setup_bundle, write_benchmark_proof_publication_bundle, and write_benchmark_proof_archive. MCP write tools enforce allowed roots; set ML_RESEARCH_LOOP_ALLOWED_ROOTS for external proof artifact directories.

Connect Codex Or Claude

Generate a Codex config for the current checkout:

ml-loop init-mcp-config --client codex

Generate a Claude Code config:

ml-loop init-mcp-config --client claude-code --output /tmp/ml-research-loop.mcp.json
claude mcp add-json ml-research-loop "$(cat /tmp/ml-research-loop.mcp.json)"

Install the repository skills so the client model knows the intended workflow:

ml-loop init-skills --client codex
ml-loop init-skills --client claude

Config templates and fresh-checkout onboarding are in examples/mcp/README.md. Full client setup is in docs/mcp-client-setup.md. For a scripted walkthrough, see docs/demo-transcript-cn.md and docs/launch-demo-cn.md.

Proposal Contract 入口

Codex/Claude 的 proposal 规划走 client-side proposal contract。客户端模型 负责读证据和生成 proposal；MCP 服务端负责打包 context、校验 proposal、 执行 guarded experiment、写 reflection，并归档 memory/proof evidence。 MCP 服务端不默认调用大模型，服务端 LLM 只通过显式 opt-in 工具启用。

最短验收路径：

context -> validate -> reflect
build context -> client proposal -> validate -> execute guarded experiment -> reflect -> memory/proof archive

运行本地 smoke/demo：

PYTHONPATH=.:.venv/lib/python3.13/site-packages \
python3 scripts/proposal_contract_smoke.py \
  --fixture-dir examples/proposal-contract/ \
  --output-dir .demo_runs/proposal-contract \
  --json

真实 Smol WorldCup/Qwen3 本地诊断样例在 examples/proposal-contract/smol-qwen3/。 若要把已校验 proposal 直接送入本地诊断轮，可使用：

ml-loop hf-eval smol-worldcup-proposal-round \
  --proposal examples/proposal-contract/smol-qwen3/proposal.json \
  --current-report examples/proposal-contract/smol-qwen3/current-report.json \
  --output-dir .demo_runs/smol-worldcup-proposal-round \
  --model qwen/qwen3-8b \
  --prompt-profile p3-dev-v2 \
  --evaluation-split dev \
  --json

多 proposal 的小规模树搜索可以用 ml-loop proposal search --items ... --branch-budget ... --diversity-max-per-family ... 汇总，输出 best-so-far、Pareto frontier、selected next nodes 和 rollback 分支。ml-loop proposal context 可用 --memory-store 检索历史经验并注入 memory cards；ml-loop proposal reflect --memory-store 可把本轮 reflection 写成可复用的本地 research memory card。Graphiti/cognee 仍是显式 opt-in 的 可选 adapter。

Optimizer/Gate 方法搜索入口

可复用 optimizer/gate 主路径是：

optimizer sources -> MethodSearchTrial -> local eval -> gate -> tell -> GateFeedbackMemory

proposal run-real-benchmark-readiness 会跑 3-5 轮 optimizer candidate race，并把候选材料化到 Smol WorldCup 本地 dev/canary eval。gate result 来自 eval outcome，不需要人工喂 gate-results.json。本地可复跑 smoke：

python docs/evidence/real-benchmark-readiness-smoke-20260627/run_smoke.py

关键证据在 docs/evidence/real-benchmark-readiness-smoke-20260627/README.md。 同一能力也通过 MCP 工具 run_real_benchmark_readiness_run 暴露给 Codex/Claude。当前证据边界仍是 local benchmark readiness； official_scores_claimed=false，不执行外部 submission，也不宣称榜单提升。

相关文档：

验收和宣传都必须保持 official_scores_claimed=false；路线判断看 dev/canary/holdout 一致性。单次本地收益只是诊断证据，不是官方成绩， 也不是稳定产品结论。

Core MCP Tools

Tool Use
get_service_manifest Read contract version, tool contracts, skill contracts, compatibility, and sandbox rules
research_task Build ml-intern-style evidence context from papers, datasets, and code providers
read_paper Read one paper by arXiv ID or URL and return findings and hypotheses
propose_hypotheses Convert evidence into experiment-ready hypotheses
run_hypothesis_experiment Run a bounded autoresearch experiment from a task config and optional task patch
review_research_results Return research review, experiment tree, dataset profile, code plan, and planner actions
run_next_experiment_from_review Execute the next task patch selected from a completed review
run_client_patch_experiment Validate and execute a Codex/Claude single-parameter proposal
apply_client_code_patch Apply a guarded workspace-relative code diff with rollback on failure
build_proposal_context Write a client-side proposal artifact bundle for Codex/Claude planning
validate_client_proposal_contract Validate one structured client proposal before execution
write_proposal_reflection Persist proposal outcome, failure labels, rollback state, and optional memory card
summarize_proposal_search Summarize a small proposal portfolio/frontier without running experiments
run_multi_optimizer_candidate_race Generate LLM/Optuna/TextGrad/DSPy/heuristic/plugin candidates, normalize them into MethodSearchTrials, gate-race them, and feed sampler memory
run_real_benchmark_readiness_run Run 3-5 local Smol WorldCup eval-backed optimizer/gate rounds without official submission or score claims
run_smol_worldcup_proposal_round Validate and run one guarded Smol WorldCup local diagnostic proposal round
run_fasttext_patch_round Execute one allowlisted fastText AG News reproduction-improvement proposal against an archived baseline
write_fasttext_patch_round_proof_bundle Package a completed fastText patch round into a human-reviewed, hash-indexed proof bundle
run_fasttext_multi_proposal_loop Execute several bounded fastText proposals, preserving failed rounds and best-so-far rollback state
write_fasttext_release_proof_bundle Package reviewed fastText proof artifacts into a downloadable tarball with checksum and review checklist
record_research_memory Append provenance-backed research memory cards from reviewed artifacts or explicit card payloads
retrieve_research_memory Search prior paper, dataset, patch, metric, failure, and proof-bundle experience
suggest_from_memory Return advisory next-step suggestions that never execute tools directly
promote_memory_card Append a promoted copy of a reviewed memory card for reuse
audit_memory_trace Explain the artifact hashes and claim boundaries behind memory suggestions
get_experiment_logs Return recent log tails for debugging failed or slow experiments
list_runtime_artifacts Inspect tasks, results, workdirs, snapshots, archive, and known task IDs
archive_runtime_artifacts Move one task's runtime artifacts into archive/
clean_runtime_artifacts Delete one task's runtime artifacts only after confirm=true
run_ai_autoresearch Opt-in server-side LLM loop for unattended runs
cp_bench Stage-dispatched CP-Bench tool (stage: proposal context, local baseline, candidate/proposal rounds, client candidate submission, submission gate)
optimizer_gate Stage-dispatched optimizer/gate pipeline tool (stage: execution plan/preflight/run, scheduler plan/action/loop/handoff, canary runner bundle, promotion review/approval/action/rollback, official submission/claim, public result fetch/verify)
registered_profile Stage-dispatched registered-profile dev/canary gate pipeline tool (stage: build/run execution bundle, build/run canary preflight/execution, canary result gate, outcome schedule)
gate_policy Stage-dispatched gate-policy tool (stage: build input/composition/graph, evaluate policy/graph)
slice_patch Stage-dispatched slice-patch tool (stage: eval matrix, optimizer selection, repair context, canary failure audit, gate evaluation, patch candidate generation/materialization/outcome)
optuna_export Stage-dispatched MethodSearchStudy export tool (stage: sampler, storage, dashboard)
method_search Stage-dispatched Optuna-style ask/tell tool (stage: create_study, ask, tell)
tournament Stage-dispatched direction tournament search (stage: start, status, submit_directions, submit_proposal, step, report, driver_tick, driver_finish) — successive-halving over per-direction greedy chains with deterministic pruning, budgets, stop conditions, and an unattended-driver layer (wakeup budgets, Phase A baseline production, build-once stop notification)

Typical Loop

research_task / read_paper
  -> propose_hypotheses
  -> run_hypothesis_experiment
  -> review_research_results
  -> Codex/Claude reads experiment_state
  -> run_next_experiment_from_review or apply_client_code_patch
  -> review again

The important output for the client planner is experiment_state, including:

  • research_evidence_gate
  • provider_coverage
  • retrieval_diagnostics
  • dataset_profile
  • experiment_tree
  • loop_policy
  • reproduction.readiness
  • code_change_plan.next_experiment_plan
  • planner_actions

Skills

Repository-local skills live under skills/:

Skill Purpose
ml-research-loop-planner Main research and experiment planning workflow
ml-research-loop-reproduction Paper reproduction, required files, rubric, and grade report workflow
ml-research-loop-experiment-optimizer Multi-round metric-aware experiment optimization workflow
ml-research-loop-operator Installation, release check, artifact management, and troubleshooting workflow
ml-research-loop-tournament-driver Unattended per-wake driving of one tournament job (tick, propose, step, finish, notify, deregister)

Setup details are in docs/skills-setup-cn.md.

Product Status

Current status: 0.1.0 preview MCP product.

The public contract is 2026-07-10.preview.v1. Preview means the local service is runnable and release-gated, but stable clients should still check get_service_manifest.contract_version before planning automated loops.

Known boundaries:

  • Live paper, dataset, and GitHub providers can be rate-limited.
  • Offline demos are the deterministic acceptance path.
  • AIDE and PaperBench are pattern sources, not runtime dependencies.
  • Graphiti and Cognee are optional memory adapters, not default runtime requirements; the local JSONL memory baseline is dependency-free.
  • Cognee 仍是 optional experimental adapter；它不阻塞 beta/stable，不进入 默认 release gate。Graphiti/cognee live smoke 只作为 optional integration evidence，其中 Graphiti live 已可通过，Cognee 在本地 gpt-oss 下仍可能在 cognify 阶段超时或返回 PipelineRunErrored。
  • Research memory does not replace runtime artifacts, proof archives, release gates, or human/client-model judgment.
  • Benchmark adapter demos and proof plans do not claim official MLE-bench or PaperBench scores.
  • This checkout includes verified dist/*.whl, dist/*.tar.gz, and dist/SHA256SUMS release artifacts, but stable release still requires frozen contract versions, real external pilot feedback, official/debug benchmark proof, and a tagged clean-checkout validation.

Documentation Map

Governance

About

MCP-native ML research loop for Codex and Claude

Topics

autoresearchclaudecodexexperiment-trackingllm-agentsmachine-learningmcpml-internreproducibilityresearch-agent

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages