a high performance LLM Inference framework

---

⚡ Overview

AFramework is designed for building arbitrarily complex llm inference systems while abstracting away the underlying mechanisms required for high-performance execution. It provides a robust foundation for multi-agent orchestration, tool execution, and verifiable reasoning.

🚀 Installation

Recommended: use uv for install and sync (builds the C++ logger extension and service binary):

cd AFramework
uv sync

Python code is used from the source tree (editable). When you change C++ under AFramework/logger/logger_cpp/, run uv sync again to rebuild.

Alternatively, with pip:

pip install -e .

🛠️ Quick Start

Starting the Service

AFramework runs as a daemon that listens on a Unix socket (default /tmp/afr.sock). You do not need to start it manually for normal client use: the first time a client needs the control plane (BaseAgentCoreClient.execute_task / execute, cancel_task, get_health, or TaskCoreClient.submit), the library calls start_agent_framework() if the socket is not already responsive. Plugin modules are still registered lazily on that same path (only the modules your spec needs).

For a long-lived or explicitly managed daemon, you can still run AgentFramework yourself or call start_agent_framework() up front—see Usage Examples.

Client Usage

Use BaseAgentCoreClient for the default agent core. BaseAgentCore requires both a model backend and a verifier (see USAGE_EXAMPLES); BaseVerifier needs the same backend config nested under model_backend in the verifier spec.

import asyncio
from AFramework.agent_core import BaseAgentCoreClient

async def main():
    model_backend = {
        "model_backend_class_name": "VLLMBackend",
        "model": "Qwen/Qwen3-30B-A3B-Thinking-2507",
        # Optional: defaults to http://0.0.0.0:8005/v1 if omitted
        # "base_url": "http://127.0.0.1:8005/v1",
    }
    client = BaseAgentCoreClient(agent_id="my_agent")
    client.register_model_backend(**model_backend)
    client.register_verifier(
        verifier_class_name="BaseVerifier",
        model_backend=model_backend,
        max_verification_steps=20,
    )

    session_id = await client.execute_task(
        messages=[{"role": "user", "content": "Hello world"}],
        stream=True,
    )
    print(f"Task started with Session ID: {session_id}")

if __name__ == "__main__":
    asyncio.run(main())

Redis configuration

AFramework uses Redis for resource control (rate limiting, circuit breaker) and the ToolPool. For non-default Redis:

Environment variables:

export REDIS_HOST=my-redis.example.com
export REDIS_PORT=6379
export REDIS_DB=0

Or pass explicitly to AgentFramework:

from AFramework.aframework import AgentFramework

service = AgentFramework(
    redis_host="my-redis.example.com",
    redis_port=6379,
    redis_db=0,
)

Defaults: 127.0.0.1:6379 db=0 when not specified.

📚 Documentation

• API Reference – Comprehensive API documentation for clients, model backends, verifiers, TaskCore, batch processing, and more.
• Usage Examples – Patterns and examples for agentic tasks, MCP tools, batch processing, error handling, and configuration.
• Architecture Decision Records (ADRs) – Why we chose key designs; start with ADR 0001 — Phase 2.0 multi-tenant resources (tool names, resource keys, policies, MCP deferrals).
• Logging – Logger architecture and configuration.
• Stochastic Router – Router design and deferred features.
• ResourceManager – Resource control and future enhancements.

✨ Features

• Stochastic Task Router: Near-optimal routing via expected-latency minimization, task profiling (BOCPD), worker telemetry, and per-worker queues. See implementation_plan_router.md for design details.
• Multi-Process Architecture: True parallelism with isolated worker processes.
• Agentic Core: Flexible AgentCore abstractions for custom agent behaviors. The worker passes a canonical keyword bundle into register_task; model_backend, verifier, and messages may be omitted on the wire (None at dispatch). BaseAgentCore still requires those for its default loop; custom cores define their own contract (see Plugin Registration in the API reference).
• MCP Support: First-class support for Model Context Protocol (MCP) servers.
• Verifiers: Integrated verification loops for robust reasoning and reduced hallucinations.
• Modular Backends: Swappable LLM backends (vLLM, OpenAI, Anthropic).

Error handling semantics

• Per-task fail-fast: Task and agent cores raise exceptions instead of returning {"error": ...} blobs. Workers catch these per task, update Redis with task_status=FAILED and a structured task_error, and re-raise so only the individual coroutine fails while the worker process continues.
• Structured task_error payloads: Stored in Redis as:
  • {"type": "<task_execution|internal_error|...>", "message": "<human readable>", "details": {...}, "traceback": "<optional traceback>"}.
• Client behavior:
  • BaseAgentCoreClient.get_task_result(session_id):
    • Raises TaskExecutionError when the task failed (based on task_status and task_error).
    • Raises RuntimeError when the task is not yet completed or the result is missing.
    • Returns the normal result only when task_status == COMPLETED.
• Durability / late readers: Results and errors remain queryable via AgentRunStore until cleaned up (no special TTL). If a client dies, another client can later call get_task_result(session_id) and will see the same success result or raised TaskExecutionError.

📋 TODOs

Deferred and optional work. Full specs and rationale are in the linked docs; the canonical checklist is also in todos.md at the repo root.

MCP (multi-tenant transport)

• MCP tenant integration — Per-tenant MCP URLs/headers/env, registry client keys, control-plane contract, tests. MCP_TENANT_INTEGRATION_TODO.md

Stochastic router (optional / future)

Background: Stochastic_router_future.md — routing stack is production-ready; items below are enhancements.

• Await shutdown_response per worker before supervisor join, or extend the drain window for large deployments.
• Round-robin among fallback workers when retrying after enqueue failure (today uses others[0]).
• AFW_CIRCUIT_COOLDOWN_S environment variable for circuit cooldown without code changes.
• Prometheus metrics for circuit breaker / workers in cooldown (get_metrics_for_pull / PrometheusMetricsBackend).
• Optional: clearer documentation that cancel is best-effort for in-flight work.

ResourceManager (optional / future)

Background: ResourceManager_future.md

• TTL on Redis keys (afw:res:*, afw:circuit:*) with refresh on activity.
• Redis authentication (password/username) and env vars + worker redis_config broadcast.
• Redis URL as a single config string (redis://...) with precedence over host/port.
• Prometheus label escaping for resource keys containing quotes or backslashes.
• Wire queue_stats into MetricsBackend.format_snapshot and Prometheus (per-worker queue depth / backlog).
• Per-resource metrics backend override on ResourcePolicy (e.g. OTel vs Prometheus per key).
• Sliding-window rate limiter as an alternative to the token-bucket algorithm.

Observability / metrics (optional / future)

• Denial metrics with tenant — Expose rate-limit / policy denial counters labeled by tenant where resource keys are tenant-qualified (e.g. acme.openai:gpt-4), for per-tenant dashboards. Would extend ResourceManager / MetricsBackend / Prometheus paths as needed.

---

Production grade LLM inference engine.