Argus

LLM observability and evaluation platform.

Argus traces LLM requests across OpenAI, Anthropic, and Gemini, capturing latency, token usage, cost, and quality metrics for every request. Events are streamed through Kafka into ClickHouse for fast analytics and visualized in Grafana.

It also includes an evaluation pipeline that scrapes documentation, generates benchmark datasets with an LLM judge, and replays them against target models.

Built with FastAPI, Kafka, ClickHouse, Docker, and Grafana. Deployable to Google Cloud Run with GitHub Actions.

Architecture

Your App
    │
    │  Argus SDK
    │  (provider wrapper -> tracer -> emitter)
    ▼
 HTTP
    │
    ▼
Collector API (FastAPI)
    │
    ▼
 Kafka
    │
    ▼
 Consumer
    │
    ▼
ClickHouse
    ▲
    │
Eval Runner (scrape -> generate -> judge -> replay)
    │
    ▼
 Grafana + /v1/* API

Why HTTP by default?

Applications send events to the collector over HTTP by default (ARGUS_EMITTER=http), so they do not need a Kafka client.

The collector publishes events to Kafka, and the consumer batches writes into ClickHouse.

Other emitter modes:

• http (default)
• kafka (publish directly)
• noop (disable telemetry)

Event emission is best effort and never raises exceptions inside the host application.

---

Quick Start

No API keys are required. The demo seeds synthetic data so the dashboards have data immediately.

docker compose up -d --build

docker compose run --rm api python -m scripts.init_clickhouse

docker compose run --rm api python -m scripts.seed_demo --hours 72 --rate 40

Grafana

http://localhost:3000

Anonymous access is enabled. Default admin credentials are admin/admin.

Collector API

http://localhost:8000/docs

---

Instrumenting LLM Calls

from argus.providers.openai_provider import OpenAIProvider

llm = OpenAIProvider(default_model="gpt-4o-mini")

response = llm.complete(
    "Summarize the theory of relativity in two sentences."
)

print(response.text)

Every request automatically records:

• latency
• token usage
• estimated cost
• refusal detection
• JSON validation

Switching providers only requires changing the provider class.

AnthropicProvider(...)
GeminiProvider(...)

Use prompt_version to compare prompt revisions.

llm.complete(
    prompt,
    prompt_version="v2"
)

---

Evaluation Pipeline

from argus.eval.scraper import scrape_sync
from argus.eval.benchmark import build_benchmark, save_benchmark
from argus.eval.runner import run_benchmark
from argus.providers.anthropic_provider import AnthropicProvider

judge = AnthropicProvider(default_model="claude-sonnet-4")

docs = scrape_sync([
    "https://en.wikipedia.org/wiki/Observability"
])

items = build_benchmark(
    judge,
    docs,
    per_doc=5
)

save_benchmark(
    items,
    "benchmarks/observability.jsonl"
)

target = AnthropicProvider(
    default_model="claude-haiku-4"
)

summary = run_benchmark(
    target,
    items,
    judge_provider=judge,
    benchmark="observability"
)

print(summary.pass_rate)
print(summary.per_category)

Evaluation results are stored alongside trace data in ClickHouse.

---

Collector API

Method	Endpoint	Description
POST	/v1/traces	Ingest a TraceEvent
POST	/v1/evals	Ingest an EvalEvent
GET	/v1/metrics/cost?hours=24	Cost by provider and model
GET	/v1/metrics/latency?hours=24	p50, p95 and p99 latency
GET	/v1/metrics/volume?hours=24	Request volume and errors
GET	/v1/metrics/quality?hours=24	Refusal, error and JSON validation rates
GET	/v1/drift?metric=cost_usd	Drift analysis
GET	/healthz	Health check

---

Drift Detection

argus/analytics/drift.py compares a recent window against a baseline using:

• Population Stability Index (PSI)
• Mean Z-score shift

Drift levels:

PSI	Status
< 0.10	Stable
0.10 - 0.25	Moderate
> 0.25	Significant

The demo dataset intentionally injects latency and cost drift for OpenAI so the dashboard shows a realistic example.

---

Pricing

config/pricing.yaml contains placeholder pricing values used for local development.

They are not guaranteed to match current provider pricing.

Model lookup uses:

1. Exact model match
2. Longest prefix match

Unknown models are recorded with zero cost and generate a warning.

---

Project Layout

argus/
├── analytics/
├── collector/
├── eval/
├── ingest/
├── providers/
├── config.py
├── emitter.py
├── events.py
├── pricing.py
├── quality.py
└── tracer.py

scripts/
├── init_clickhouse.py
└── seed_demo.py

grafana/
deploy/

Dockerfile
docker-compose.yml
.github/workflows/ci.yml

---

Local Development

Install the core package:

pip install -e ".[dev]"

Run tests:

pytest

Install everything:

pip install -e ".[all]"

---

Deployment

The GitHub Actions workflow:

• runs Ruff
• runs tests
• builds the Docker image
• deploys to Cloud Run on main

Manual deployment:

export GCP_PROJECT_ID=...
export GCP_REGION=us-central1
export AR_REPO=argus

./deploy/deploy.sh

The collector runs on Cloud Run.

Kafka and ClickHouse should be available separately and configured through:

• ARGUS_KAFKA_BOOTSTRAP
• ARGUS_CH_HOST

---

License

MIT