Five LLM agents — PDM → Architect → Tech Lead → Coder ⟲ Testing — turn a feature request into reviewed, tested, committed code on a real GitHub repository, under a deterministic orchestrator that never trusts prose: every hand-off passes a quality gate computed from tool evidence, every risky step pauses for a human decision, every rejected attempt is rolled back in git, and every change is measured by a four-layer evaluation strategy before it ships.
Built as a learning project, step by step (the build log, the original brief), into a platform with two environments, CI/CD with a paid evaluation gate, and full observability.
flowchart LR
subgraph you["You"]
B["Board (React, SSE)\n:30086 / :31086"]
GH["GitHub PRs"]
end
subgraph vm["Multipass VM · k3s"]
O["Orchestrator (FastAPI)\nstate machine · gates · HITL\nPostgres · :30084 / :31084"]
subgraph kagent["kagent (A2A)"]
A1[PDM] --- A2[Architect] --- A3[Tech Lead] --- A4[Coder] --- A5[Testing]
end
T["ToolServer (MCP)\nfile/pytest/git tools\n+ /workspace lifecycle API"]
LF["Langfuse\ntraces · cost · datasets"]
R["GitHub Actions runner\n(self-hosted, systemd)"]
end
TR["Target repo\natedgimo/asdlc-sample-calc\nbranch asdlc/<task> → draft PR"]
B <--> O
O -->|A2A| kagent
kagent -->|MCP| T
T -->|"clone · push (accepted only)"| TR
O -->|accept / reset / PR| T
O -.traces.-> LF
kagent -.OTel.-> LF
R -->|deploy staging · eval gate| O
TR --> GH
- A request is filed (board form, markdown file, or API) → the PDM writes a spec, the Architect an ADR, the Tech Lead a plan of bounded work items — each validated by a schema gate and paused for your approval.
- Per work item, the Coder works in the task's own git clone of the target repo (never a shared tree), and the
Testing agent verifies with read-only tools and
git_diffagainst the last accepted commit. - A passed gate commits and pushes; a failed gate resets the clone — rejected work never reaches the remote.
Gated tool calls (e.g.
git_commit) pause the agent mid-run until you (or policy) answer. - On completion the orchestrator opens a draft PR with the spec, work items, every attempt and the trace link.
- What the run taught (reviewer notes, gate pitfalls, accepted decisions) becomes project memory, injected into future prompts — never into evaluation runs.
Prereqs: macOS with Multipass, Docker, kubectl, helm, jq, uv,
gh (logged in), a GCP project with Vertex AI (or the Gemini free tier), and a .env from .env.example.
make up # VM → k3s + Cilium → kagent → hello-agent smoke test
make langfuse-up # self-hosted Langfuse + tracing (optional but everything is better with it)
make vertex-setup CONFIRM=yes # Gemini on Vertex (or stay on the free tier)
make repo-create CONFIRM=yes # the agents' target repo, seeded; then the PAT → .env → see below
make asdlc-secrets ENV=prod # the environment's Secrets from .env
make ci-build ci-import # images at the current sha
make asdlc-deploy ENV=prod # ONE helm install = agents + ToolServer + orchestrator + board
make git-smoke # a real coding task: clone → gate → push → draft PR (≈ $0.02)No cluster, no keys? make board-demo + make board-dev runs the whole engine with canned agents on localhost, and
make eval-demo runs the entire evaluation harness for free.
Two releases of one umbrella chart (charts/asdlc) share the cluster, kagent and Langfuse:
| namespace | board | deployed by | Langfuse env | |
|---|---|---|---|---|
| staging | asdlc-staging |
:31086 | CI, on every push to main |
staging |
| prod | asdlc |
:30086 | make promote TAG=sha-… CONFIRM=yes |
default |
GitHub Actions runs the free layers (unit tests ×3, board e2e, demo eval, chart render, secret guard) on cloud runners,
and the deploy + the paid eval gate on a self-hosted runner inside the VM — the gate first deploys the PR's own
sha to staging, runs the regression dataset through the real agents, and blocks the merge below the thresholds, with a
diff against the baseline posted on the PR. Switch it with make eval-gate-on|off.
Four layers — unit tests, the offline demo eval, online evals on staging, observation in prod — with dataset discipline,
a pinned LLM judge, per-item regression diffs and retry-on-flake. The strategy and how to see each layer:
docs/evaluation-strategy.md. Current baseline: experiments/runs/baseline-1 (43 items,
$0.31) and the thresholds in experiments/thresholds.yaml.
| Path | What |
|---|---|
| orchestrator/ | the deterministic core: engine, gates, HITL, workspace client, project memory, API, its chart |
| toolserver/ | the agents' MCP tools + the git-backed workspace lifecycle API |
| charts/asdlc, envs/ | one environment per helm install; agent prompts live here (files/agents/) |
| ui/ | the board: seven screens over one SSE stream |
| experiments/ | datasets, evaluators, the LLM judge, run reports, the diff |
| infra/ | Makefile-driven scripts and the step-by-step build log (infra/README.md) |
| .github/workflows | cloud CI, the deploy job, the eval gate |
| docs/ | the evaluation strategy |
Vertex gemini-2.5-flash for everything: a feature end-to-end ≈ $0.03–0.05, the full eval set ≈ $0.33, the PR gate
≈ $0.05 (+ a candidate deploy). Cloud CI ≈ 10 min of the free 2,000/month; the self-hosted runner is free.
This is a single-operator lab: NodePorts (board, orchestrator, Langfuse) are open on the LAN without auth; the
self-hosted runner executes this private repo's workflows only — do not make the repo public without hardening that
first. LLM-written code runs in the ToolServer pod (non-root, resource-limited, no cluster credentials — the only
secret there is the repo-scoped GitHub PAT). Known limitations, accepted deliberately: single-replica invariants
(orchestrator engine and ToolServer locks are per-process; scaling either corrupts state), one target repo per
environment (the spec has a project field but no repo routing yet), eval step names resolve against the feature
pipeline first, the Postgres store has no optimistic concurrency (last full-document writer wins), one model label
prices all agents' generations, and the LLM judge is variance-prone by nature (mitigated by deterministic scores,
retries and diffs — see the strategy doc).
MIT © 2026 Moti Atedgi. Built with kagent, Langfuse, FastMCP/kmcp, k3s + Cilium — and with Claude Code.