Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,622 changes: 80 additions & 1,542 deletions CHANGELOG.md

Large diffs are not rendered by default.

43 changes: 11 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
**Ship AI agents with real-time budget, policy, and human-approval gates.**

Zero-refactor cost control, tool policy enforcement, and audit trail for any
LLM-powered agent - works with OpenAI, Anthropic, LangGraph, CrewAI, AutoGen,
LlamaIndex, and your own stack.
LLM-powered agent — works with any LLM SDK that uses `httpx`, plus your own stack.

[Quickstart](https://docs.nullrun.io/getting-started/onboarding/) · [Docs](https://docs.nullrun.io) · [Examples](https://github.com/nullrunio/nullrun-examples)

Expand Down Expand Up @@ -63,7 +62,7 @@ Existing observability tools tell you **after** the fact. NullRun enforces **bef
|---|---|
| **Hard & soft budget gates** — atomic Redis-enforced | **Tool policy enforcement** — block dangerous tools before execution |
| **Human-in-the-loop approvals** — pause agent and await `approval_resolved` via WS push | **Immutable audit trail** — every decision, every tool call, every cent |
| **Zero-code instrumentation** — `nullrun.init()` patches `httpx` once for any vendor | **LangGraph, CrewAI, AutoGen, LlamaIndex** — first-class integrations |
| **Zero-code instrumentation** — `nullrun.init()` patches `httpx` once for any vendor | **No vendor lock-in** — works with any LLM SDK that uses httpx |
| **Memory-safe streaming** — 16 MiB response body; full body for usage extraction | **Lightweight** — no LLM-key storage, no proxy required |
| **Server-authoritative cost** — server-minted execution IDs | **MCP support** — expose tools to agents via Model Context Protocol |

Expand Down Expand Up @@ -211,33 +210,17 @@ def my_agent(prompt: str) -> str:

```

### Framework adapters — auto-detected
If you call `@protect` *before* `init()`, the SDK lazy-initializes
the runtime from `NULLRUN_API_KEY` on the first decorated call. You can
write your agent code with the decorator first and the init second — or
skip `init` entirely if your environment is already configured.

NullRun auto-detects installed frameworks and instruments them automatically
when `init_or_die()` runs (or when `@protect` first fires). You don't need
to choose an extra; if a framework is already in your environment, it gets
patched in place.
For CLI scripts that want fail-fast on missing config, pass
`fail_on_exit=True` — the SDK prints a four-line developer report and
exits with code 1 instead of raising. `nullrun.shutdown()` is
auto-registered via `atexit` inside `init()`, so a clean WS close on
process exit happens without any explicit call.

| Framework | What gets patched | Trigger |
|---|---|---|
| **LangGraph** (`Pregel.invoke` / `stream` / `ainvoke` / `astream`) | `NullRunCallback` injected per call | auto on `init_or_die()` |
| **LangChain** (`BaseCallbackManager`) | `NullRunCallback` registered | auto on `init_or_die()` |
| **OpenAI Agents** (`Runner.run` / `run_streamed`) | `RunHooks` / `RunStreamedHooks` instrumented | auto on `init_or_die()` |
| **LlamaIndex** (`get_dispatcher`) | `LLMChatEndEvent` / `FunctionCallEvent` handlers | auto on `init_or_die()` |
| **CrewAI** (event bus + `usage_metrics`) | `Agent` / `Task` / `Crew` lifecycle | auto on `init_or_die()` |
| **AutoGen** (`Agent.run` / `a_run`) | message-streaming hooks (HTTP path is httpx-based) | auto on `init_or_die()` |

**HTTP-level coverage is the foundation** — `httpx` (and `requests`) are
patched once by `init_or_die()` regardless of vendor. Token counts and
model info are extracted from response bodies for OpenAI, Azure, Anthropic,
Mistral, Gemini, Cohere, and Bedrock without those vendor SDKs needing to
be installed. If you use the raw `httpx.Client` API directly, you get
cost tracking out of the box.

If you call `@protect` *before* `init_or_die()`, the SDK auto-triggers
instrumentation lazily on the first decorated call. You can write your
agent code with the decorator first and the init second — or skip `init`
entirely if your environment is already configured via `NULLRUN_API_KEY`.
---

## How NullRun compares
Expand Down Expand Up @@ -336,10 +319,6 @@ and `src/nullrun/transport.py::consume_approval`.

Runnable, copy-pastable examples live in a separate repo so you can adapt without cloning the SDK source:

- **[LangGraph](https://docs.nullrun.io/how-to/langgraph/)** — multi-node agent with budget + approval
- **[CrewAI](https://docs.nullrun.io/how-to/crewai/)** — multi-agent crew with shared budget
- **[AutoGen](https://docs.nullrun.io/how-to/autogen/)** — group-chat agent with policy gating
- **[LlamaIndex](https://docs.nullrun.io/how-to/llama-index/)** — RAG pipeline with cost-per-query enforcement
- **[Custom tools](https://docs.nullrun.io/how-to/fastapi/)** — register your own tools for policy
- **[Multi-agent](https://docs.nullrun.io/how-to/multi-agent/)** — shared budget across sub-agents

Expand Down
66 changes: 0 additions & 66 deletions docs/errors/NR-B003.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/errors/NR-C000.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@ to override.

- `NR-C001` — `nullrun.init()` called with no api_key.
- `NR-C003` — `get_org_status()` called before the runtime is bound.
- `NR-C004` — `nullrun.status()` called before `nullrun.init()`.
- `NR-C004` — `nullrun.get_runtime()` (or `runtime.status()`) called before `nullrun.init()`.
12 changes: 6 additions & 6 deletions docs/errors/NR-C001.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ call from the environment, so `init()` does not need to be called —
but the env var must be present by the time the runtime is asked to
gate a call.

If the developer chose to call `init()` or `init_or_die()`
explicitly, the same code surfaces earlier (at the explicit init
call, not at the first `@protect`). Both paths produce the same
typed exception.
If the developer chose to call `init()` explicitly with
`fail_on_exit=True`, the same code surfaces earlier (at the explicit
init call, not at the first `@protect`) and the SDK prints a
four-line developer report and `sys.exit(1)` instead of raising.

## Why this raises (instead of falling back)

Expand Down Expand Up @@ -64,7 +64,7 @@ try:
result = my_agent(prompt)
except NullRunConfigError as exc:
if exc.error_code == "NR-C001":
# Show the user the dashboard link inline. With `with nullrun.handle():`
# Show the user the dashboard link inline. With `with nullrun.guard():`
# around the call, the SDK will print the four-line developer report
# automatically — you only need this catch for custom error UI.
return render_onboarding(api_key_help_url=exc.user_action)
Expand All @@ -75,4 +75,4 @@ except NullRunConfigError as exc:

- `NR-A001` / `NR-A002` / `NR-A003` — key provided but rejected.
- `NR-C003` — runtime bound, but no `org_id` available for `get_org_status()`.
- `NR-C004` — `nullrun.status()` called before the runtime is bound.
- `NR-C004` — `nullrun.get_runtime()` / `runtime.status()` called before the runtime is bound.
30 changes: 20 additions & 10 deletions docs/errors/NR-C004.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
# NR-C004 — `nullrun.status()` called before `nullrun.init()`
# NR-C004 — `get_runtime()` / `runtime.status()` called before `nullrun.init()`

> **0.18.4 note:** the top-level `nullrun.status()` wrapper was removed
> in 0.18.4; reach the snapshot via `nullrun.get_runtime().status()`
> instead. `get_runtime()` raises `NullRunConfigError(NR-C004)` when
> no runtime has been bound. This doc retains the same shape; only the
> entry-point wording has changed.

| Field | Value |
|---|---|
| **Code** | `NR-C004` |
| **Category** | Configuration |
| **Exception class** | `NullRunConfigError` |
| **Retryable** | No |
| **Default `user_action`** | "Call `nullrun.init(api_key='nr_live_...')` before calling `nullrun.status()`. The snapshot only makes sense once the SDK has a runtime bound to the API key." |
| **Default `user_action`** | "Call `nullrun.init(api_key='nr_live_...')` before requesting the runtime snapshot. The snapshot only makes sense once the SDK has a runtime bound to the API key." |

## When

Raised by `nullrun.status()` when the runtime has not been initialised
yet. `status()` returns a snapshot of the runtime's account state —
without an active runtime, there is nothing to snapshot.
Raised by `nullrun.get_runtime()` when the runtime has not been
initialised yet. `runtime.status()` returns a snapshot of the
runtime's account state — without an active runtime, there is
nothing to snapshot.

This is distinct from `NR-C001` (no api_key at all): the call to
`init()` was simply never made, or the runtime was shut down with
Expand All @@ -22,29 +29,32 @@ This is distinct from `NR-C001` (no api_key at all): the call to

- Forgot to call `nullrun.init()` at process startup.
- Called `nullrun.shutdown()` (or `runtime.shutdown()`) at module
unload and then `nullrun.status()` from a signal handler or
unload and then `runtime.status()` from a signal handler or
finalizer that ran afterwards.
- Calling `status()` from a test fixture that did not auto-init.
- Calling `runtime.status()` from a test fixture that did not
auto-init.

## How to fix

1. Add `nullrun.init(api_key=...)` (or read from `NULLRUN_API_KEY`
env var by passing `api_key=None`) at the top of the entrypoint.
2. If the runtime was intentionally shut down, skip the snapshot
call rather than re-initialising after shutdown.
3. In tests, use the `nullrun_test_runtime` fixture from
3. In tests, use the `make_runtime` fixture from
`tests/conftest.py` instead of constructing one manually.

## Catch pattern

```python
from nullrun.breaker.exceptions import NullRunConfigError
from nullrun import get_runtime

try:
snap = nullrun.status()
rt = get_runtime()
snap = rt.status()
except NullRunConfigError as exc:
if exc.error_code == "NR-C004":
log.error("status() called before init: %s", exc.user_action)
log.error("runtime not bound: %s", exc.user_action)
return None
raise
```
Expand Down
3 changes: 1 addition & 2 deletions docs/errors/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The codes follow a `NR-<CATEGORY><NNN>` pattern:
| `NR-C000` | Generic config error (default on `NullRunConfigError`; subclasses override) | [NR-C000](NR-C000.md) |
| `NR-C001` | `nullrun.init()` called with no api_key (no param, no env) | [NR-C001](NR-C001.md) |
| `NR-C003` | `get_org_status()` called before the runtime is bound to an org | [NR-C003](NR-C003.md) |
| `NR-C004` | `nullrun.status()` called before `nullrun.init()` | [NR-C004](NR-C004.md) |
| `NR-C004` | `nullrun.get_runtime()` (or `runtime.status()`) called before `nullrun.init()` | [NR-C004](NR-C004.md) |

### Authentication (NR-A)

Expand All @@ -44,7 +44,6 @@ The codes follow a `NR-<CATEGORY><NNN>` pattern:
|---|---|---|
| `NR-B001` | Network error: timeout, ConnectError, DNS failure | [NR-B001](NR-B001.md) |
| `NR-B002` | 5xx from the NullRun backend | [NR-B002](NR-B002.md) |
| `NR-B003` | `@sensitive` failed to extract a `BusinessImpact` envelope | [NR-B003](NR-B003.md) |
| `NR-B004` | Budget exhausted | [NR-B004](NR-B004.md) |
| `NR-B005` | Local circuit breaker tripped | [NR-B005](NR-B005.md) |

Expand Down
64 changes: 10 additions & 54 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "hatchling.build"
name = "nullrun"
# Full release history lives in CHANGELOG.md; only the current version
# is pinned here.
version = "0.18.1"
version = "0.18.4"
# Kept under the 200-char preview threshold so the full line is visible
# without an "expand" click. The headline is the canonical §1 statement
# from positioning.md — "runtime decision layer for tool-using AI agents"
Expand Down Expand Up @@ -71,50 +71,12 @@ dependencies = [
]

[project.optional-dependencies]
# OpenTelemetry is an observability standard (not a vendor framework);
# auto-detected at runtime when installed.
opentelemetry = [
"opentelemetry-api>=1.26.0,<2.0",
"opentelemetry-sdk>=1.26.0,<2.0",
]
langgraph = [
"langgraph>=0.2.0,<1.0",
]
# Framework auto-instrumentation dependencies.
#
# These extras install framework SDKs whose event systems NullRun
# subscribes to. NullRun's HTTP-level instrumentation
# (``patch_httpx`` + ``patch_requests`` + 5 URL-keyed extractors)
# covers OpenAI / Anthropic / Mistral / Gemini / Cohere / Bedrock
# WITHOUT requiring their vendor SDKs — all of those vendors route
# through httpx, and NullRun parses the response body by URL host.
# The vendor SDK packages are NOT imported anywhere in
# ``src/nullrun/``, so extras like ``[openai]`` / ``[anthropic]`` /
# ``[mistral]`` / ``[gemini]`` / ``[cohere]`` / ``[bedrock]`` would
# be dead weight for the SDK.
#
# What NullRun DOES import from these framework SDKs:
# - ``agents`` → ``agents.Runner`` (openai-agents tracing model)
# - ``langchain`` → ``langchain_core.callbacks.BaseCallbackManager``
# + ``langchain_core.language_models.BaseChatModel``
# - ``langgraph`` → ``langgraph.pregel.Pregel``
# - ``llama-index`` → ``llama_index.core.instrumentation``
# - ``crewai`` → ``crewai.Crew`` + ``crewai.events``
# - ``autogen`` → ``autogen_agentchat.agents.BaseChatAgent`` +
# ``autogen_ext.models.openai.OpenAIChatCompletionClient``
#
# Each ``patch_*`` wraps its framework import in
# ``try/except ImportError`` so ``nullrun.init()`` never crashes when
# the optional package is missing. Auto-detection: NullRun activates
# an adapter when the package is installed — the user does NOT need
# to choose which framework extra to install; installing any one of
# them auto-enables its adapter.
agents = ["openai-agents>=0.1,<1.0"]
langchain = ["langchain-core>=0.3,<1.0"]
llama-index = ["llama-index-core>=0.10.20,<1.0"]
crewai = ["crewai>=0.80,<2.0"]
autogen = [
"autogen-agentchat>=0.4,<1.0",
"autogen-ext[openai]>=0.4,<1.0",
]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.23",
Expand Down Expand Up @@ -463,22 +425,16 @@ line-length = 100
select = ["E", "F", "I", "UP", "B", "S"]
ignore = [
"S101",
# Pre-existing violations in master / wip/working-tree: tracked
# for follow-up cleanup in a dedicated PR rather than blocking CI.
# Categories:
# S110 (try/except/pass) - 14 sites; needs logging, not blanket
# noqa
# E501 (line too long) - 13 sites; long descriptive comments
# F841 (unused variable) - 6 sites; one is the timestamp var
# in the legacy-code fallback path
# E402 (import order) - 5 sites; TYPE_CHECKING blocks
# F401 (unused import) - 2 sites
# Pre-existing violations in master. Tracked for follow-up cleanup
# in a dedicated PR rather than blocking CI. Categories:
# S110 (try/except/pass) - blanket noqa would hide bugs; needs
# logging, not blanket noqa
# E501 (line too long) - long descriptive comments
# E402 (import order) - TYPE_CHECKING blocks
"S110",
"E501",
"F841",
"E402",
"F401",
# S311 (suspicious random) - 1 site, in circuit_breaker jitter.
# S311 (suspicious random) in circuit_breaker jitter.
# random.uniform is correct for jitter (we want non-cryptographic
# randomness to spread reconnection timing across workers).
"S311",
Expand Down
Loading
Loading