🤖 AgentForge

The same agent, built ten different ways.
A hands-on comparison of modern AI agent frameworks — with runnable examples, a shared result contract, and a reproducible benchmark harness.

Quick Start · Capability Matrix · Find an Example · Benchmarks · Contributing

---

What is AgentForge?

Choosing an agent framework usually means reading ten sets of marketing docs and guessing. AgentForge exists so you can read code instead of claims: the same capabilities, implemented across ten frameworks, plus a harness that measures them under identical conditions.

Who it's for

• Engineers evaluating which framework to adopt.
• Developers who know one framework and need to translate a pattern into another.
• Anyone learning agent concepts — tools, memory, RAG, multi-agent, MCP — who wants a minimal working example rather than a tutorial series.

What you get

🧩 ~90 runnable examples	Across 10 frameworks, each focused on one concept
📊 A capability matrix	Which framework has an example of what — see below
⚖️ A fair benchmark harness	Same model, temperature, prompts and tool code across frameworks
🔌 One result contract	Every comparison agent returns the same AgentResult, success or failure
🧪 Tests that run without credentials	pytest on the shared code, mocked network

Honesty note. This repository ships the benchmark harness, not benchmark scores. Any numbers you see were produced by a specific run, on a specific day, against a specific model — reproduce them yourself before trusting them.

---

🚀 Quick Start

git clone https://github.com/Olwtelet/AgentForge.git
cd AgentForge

Each framework is an independent project with its own pinned dependencies — they deliberately do not share one environment (see Why separate environments). Pick one and set it up:

cd llama-index
uv sync

cp .env.example .env

Fill in the keys listed in that .env, then run any example:

uv run python 00_hello_world.py

That's it. Every framework directory follows the same three steps: uv sync, cp .env.example .env, uv run python <example>.py.

Don't have uv?

Install it once (see the uv docs):

pip install uv

Or use a plain virtual environment instead — every module has a pyproject.toml, so pip works too:

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

---

🧭 Repository Architecture

AgentForge/
├── ag2/  agno/  autogen/  crewai/  google-adk/       # framework examples,
├── langgraph/  llama-index/  openai-agents-sdk/      # one self-contained
├── pydantic-ai/  smolagents/                         # project each
│
├── study-agents-differences/        # the cross-framework comparison module
│   ├── agent_contract.py            #   AgentResult / TokenUsage — the shared contract
│   ├── *_agent.py                   #   the same agent, once per framework
│   ├── shared_functions/            #   identical tool code given to every framework
│   ├── prompts.py                   #   identical system prompt for every framework
│   ├── benchmarks/                  #   scenarios, datasets, runner, report
│   ├── knowledge_base/              #   documents used by the RAG scenario
│   ├── tests/                       #   pytest suite (unit tests need no credentials)
│   └── agent-ui.py                  #   Streamlit UI to chat with any agent
│
├── .github/workflows/ci.yml         # lint · compile · unit tests · secret scan
├── ruff.toml                        # repository-wide lint rules
└── res/                             # logos and images

Why separate environments

Frameworks pin conflicting versions of openai, pydantic, langchain-core and friends. Forcing them into one environment would mean unpinning them, which destroys reproducibility — the opposite of the point. So each directory owns its pyproject.toml and uv.lock, and you install only what you're studying.

---

🤖 Frameworks Included

Framework	Examples	Docs	Repo
AG2	ag2/	Docs	GitHub
Agno	agno/	Docs	GitHub
AutoGen	autogen/	Docs	GitHub
CrewAI	crewai/	Docs	GitHub
Google ADK	google-adk/	Docs	GitHub
LangGraph	langgraph/	Docs	GitHub
LlamaIndex	llama-index/	Docs	GitHub
OpenAI Agents SDK	openai-agents-sdk/	Docs	GitHub
Pydantic AI	pydantic-ai/	Docs	GitHub
smolagents	smolagents/	Docs	GitHub

---

📊 Capability Matrix

This table describes what this repository demonstrates, not what each framework is capable of. A ❌ means "no example here yet" — almost always the framework itself supports the feature. Contributions welcome: see CONTRIBUTING.md.

Framework	Hello World	Tools	Structured Output	Streaming	Memory	HITL	Multi-Agent	RAG	MCP	Tracing	Evals
AG2	✅	✅	✅	❌	❌	✅	✅	❌	❌	❌	❌
Agno	✅	✅	⚠️	✅	✅	✅	⚠️	✅	❌	❌	❌
AutoGen	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌
CrewAI	✅	✅	✅	✅	✅	⚠️	✅	✅	❌	❌	❌
Google ADK	✅	✅	✅	❌	✅	❌	✅	⚠️	✅	❌	✅
LangGraph	✅	✅	❌	❌	✅	✅	✅	✅	❌	❌	❌
LlamaIndex	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌
OpenAI Agents SDK	✅	✅	✅	✅	❌	❌	✅	❌	❌	✅	✅
Pydantic AI	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌
smolagents	✅	✅	❌	❌	❌	❌	✅	❌	❌	❌	❌

Legend — ✅ a dedicated example exists · ⚠️ shown incidentally inside another example, not on its own · ❌ no example in this repository yet.

Why the ⚠️ marks

• Agno / Structured Output — Pydantic response models appear inside 6_workflow_example.py, not as a standalone example.
• Agno / Multi-Agent — 6_workflow_example.py orchestrates steps rather than demonstrating agent-to-agent delegation.
• CrewAI / HITL — a human-input tool exists in crewai-simple-examples/agents.py but is wired to a Chainlit UI, not a standalone script.
• Google ADK / RAG — 12_grounding.py grounds answers in Google Search results; there is no local vector-store example.

---

🔎 Find an Example

Jump straight to the code for the concept you care about.

Framework	Hello World	Tools	Structured Output	Streaming	Multi-Agent
AG2	0_sample_agent.py	1_agent_with_tools.py	2_structured_outputs.py	—	4_multi_agent.py
Agno	1_simple_agent.py	4_parallel_tool_calling.py	6_workflow_example.py	3_streaming.py	6_workflow_example.py
AutoGen	0_hello_world.py	1_tools.py	3_structured_outputs.py	2_streaming_and_metrics.py	5_multi_agent_teams.py
CrewAI	0_hello_world.py	1_tools.py	4_structured_outputs.py	3_streaming.py	10_multi_agent_collaboration.py
Google ADK	00_hello_world.py	01_tools.py	08_structured_outputs.py	—	04_multi_agent_systems.py
LangGraph	langgraph_agent.py	langgraph_agent.py	—	—	customer-support.ipynb
LlamaIndex	00_hello_world.py	01_tools.py	02_structured_outputs.py	04_streaming.py	09_agent_delegation.py
OpenAI Agents SDK	0_hello_world.py	1_tools_and_metrics.py	2_structured_outputs.py	4_handoffs_and_streaming.py	5_agents_as_tools.py
Pydantic AI	00_hello_world.py	01_tools_and_metrics.py	04_structured_outputs.py	03_streaming.py	07_agent_delegation.py
smolagents	simple-agent.py	simple-agent.py	—	—	multi-agent.py

Framework	Memory	Human-in-the-Loop	RAG	MCP	Tracing / Evals
AG2	—	3_human_in_the_loop.py	—	—	—
Agno	agno_agent.py	5_human_in_the_loop.py	agno_rag_api_agent.py	—	—
AutoGen	7_memory.py	4_human_in_the_loop.py	autogen-project/	—	—
CrewAI	7_memory.py	crewai-simple-examples/	9_knowledge.py	—	—
Google ADK	07_memory.py	—	12_grounding.py	11_mcp_tools.py	14_evaluation.py
LangGraph	langgraph_agent.py	customer-support.ipynb	langgraph-project/	—	—
LlamaIndex	03_memory.py	06_human_in_the_loop.py	10_agentic_rag.py	mcp/	12_observability.py
OpenAI Agents SDK	—	—	—	—	8_tracing.py · 7_llm_as_a_judge.py
Pydantic AI	06_message_history.py	10_human_in_the_loop.py	—	—	—
smolagents	—	—	—	—	—

---

⚖️ Benchmarks

The study-agents-differences/ module implements the same agent in every framework and measures them under identical conditions.

What is measured

• Latency — wall-clock time for one chat() call.
• Token usage — prompt, completion and total, as reported by each framework.
• Tool calls — where the framework exposes the count.
• Failure rate — errors are recorded as results, not swallowed.

What is held constant

Model · temperature · system prompt · user prompts · the actual Python code behind each tool · knowledge base · memory setting · iteration count · timeout · retries · result shape.

The shared tool implementations live in shared_functions/ and are registered into every framework, so no framework gets a smarter tool than another.

Running a benchmark

cd study-agents-differences
uv sync
cp .env.example .env

uv run python -m benchmarks.runner --list

uv run python -m benchmarks.runner --scenario web_search --iterations 5

Results are written as JSONL — one object per run:

{"framework": "langgraph", "scenario": "web_search", "prompt_id": "ucl_2024_explicit",
 "iteration": 1, "model": "gpt-4o-mini", "provider": "azure", "temperature": 0.0,
 "elapsed_seconds": 2.43, "input_tokens": 100, "output_tokens": 80,
 "total_tokens": 180, "tool_calls": 2, "success": true, "error": null}

Turn them into Markdown tables:

uv run python -m benchmarks.report benchmarks/results/run-*.jsonl -o REPORT.md

The report is generated from the recorded runs — no hand-copied numbers.

Limitations — read before drawing conclusions

Benchmark results depend on the model, network conditions, provider load, library version, prompt wording, tool implementation and the specific run. A framework being faster here does not make it faster for your workload.

Two things are deliberately not equalised, because they are intrinsic to each framework: the retrieval pipeline (each uses its own vector store and chunking) and the agent loop (ReAct vs function-calling vs graph agents take different numbers of model round-trips). Details in benchmarks/README.md.

Current status: not executed — API credentials required. This repository ships the harness and datasets. Figures collected by the original author with an earlier, less controlled setup are preserved in study-agents-differences/README.md and labelled as historical.

Interactive UI

Chat with any of the comparison agents side by side:

cd study-agents-differences
uv run streamlit run agent-ui.py

---

🧪 Tests

Unit tests run with no credentials and no network — every HTTP call is mocked.

cd study-agents-differences
uv run pytest

Integration tests (which call real APIs and spend tokens) are opt-in:

uv run pytest -m integration

CI runs lint, byte-compiles every example, runs the unit tests and scans for secrets on every pull request. Integration tests only run on manual dispatch.

---

📚 Further Reading

• CONTRIBUTING.md — add a framework, add an example, update benchmarks
• SECURITY.md — reporting vulnerabilities and handling credentials
• study-agents-differences/README.md — the comparison module in depth
• study-agents-differences/benchmarks/README.md — benchmark methodology

---

⚠️ A Note on Credentials

Every module reads its configuration from a local .env file that is never committed. Copy .env.example, fill it in, and keep it local. If you believe a credential has been exposed, see SECURITY.md.

(back to top)