enterprise-multi-agent-knowledge-platform

A multi-agent retrieval service that answers questions over a 6,000-document knowledge corpus using FAISS + BM25 hybrid fusion behind a LangGraph supervisor, cutting p50 query latency by 91.9% under concurrent load (ANN index + cache vs exact search, concurrency 16).

What this solves

• Pain: single-signal retrieval misses queries that combine a rare name with a common topic, and exact vector search collapses under concurrent load on CPU-only hardware.
• Mechanism: dense (FAISS) and sparse (BM25) rankings fused with reciprocal rank fusion, served through a guardrailed agent graph with an IVF ANN index and an in-process LRU cache.
• Measured result: precision@5 0.863 hybrid vs 0.839 for the best single retriever (+2.86%), and p50 latency 32.2 ms vs 396.5 ms at concurrency 16 (91.9% lower, 2 vCPU).

Executive summary

An internal support or engineering team searching a knowledge base of several thousand documents loses time in two ways: queries that name a specific system inside a broad topic ("settlement disputes at swiftpay") return near-misses, and the search service itself slows to hundreds of milliseconds once a handful of users query concurrently on shared CPU hardware. On this project's labeled evaluation set, the best single retriever left 16.1% of the top-5 slots wrong, and exact brute-force search served a median of 396.5 ms per query at concurrency 16.

The service treats retrieval as two independent signals and merges them mechanically. A BM25 index anchors on rare lexical tokens (entity names); a dense LSA index (TF-IDF + truncated SVD into FAISS) anchors on topic semantics; reciprocal rank fusion combines both rankings without any trained reranker. Requests flow through a LangGraph supervisor graph: a guardrail agent validates and routes (empty, oversized, and prompt-injection-pattern queries are rejected with HTTP 422), a retriever agent runs the hybrid search, and a synthesizer agent produces an extractive answer with citations. An IVF ANN index replaces exact search and an in-process LRU/TTL cache absorbs repeated queries; both are env-configurable and the cache degrades to recompute on any failure.

On the committed 6,000-document corpus with 200 labeled queries (ground truth known by construction, generator committed), hybrid RRF reached precision@5 0.863, recall@10 0.772, and MRR@10 0.9617, against 0.839 / 0.771 / 0.9492 for BM25 alone and 0.810 / 0.724 / 0.9442 for dense alone. Under an asyncio load generator at concurrency 16 (200 requests, one uvicorn worker, 2 vCPU, 4GB shared container), the production configuration (IVF + cache) served p50 32.2 ms vs 396.5 ms for exact search without cache, a 91.9% reduction; observed cache hit ratio during the run was 78.9%.

Architecture

flowchart LR
    C[Client] --> API["FastAPI /query"]
    subgraph AG["Agent graph (failure boundary: any node exception returns HTTP 500, request isolated)"]
        G[Guardrail agent] -->|route=retrieve| R[Retriever agent]
        G -->|route=reject, HTTP 422| X[Reject terminal]
        R --> S[Synthesizer agent]
    end
    API --> G
    subgraph RET["Hybrid retrieval (failure boundary: missing or corrupt artifacts trigger rebuild at startup)"]
        R --> D["FAISS dense index (IVF or flat)"]
        R --> B["BM25 sparse index"]
        D --> F["RRF fusion (author built)"]
        B --> F
    end
    subgraph CA["Cache (failure boundary: any cache error falls back to recompute, never fails the query)"]
        API -.get/set.-> L["LRU+TTL in-process, optional Redis"]
    end
    S --> API
    API --> M["/metrics and /healthz"]

Loading

Tech stack

Technology	Role	Why chosen here
FastAPI + uvicorn	HTTP service, validation	Async endpoints let one worker overlap I/O during load tests; Pydantic schemas reject malformed queries before the graph runs
LangGraph	Agent graph runtime	Conditional-edge supervisor pattern; adding a worker agent is one node plus one route value, no rewiring
faiss-cpu	Dense vector index	Supports both exact (flat) and ANN (IVF) on CPU, which is exactly the comparison this repo benchmarks
rank-bm25	Sparse lexical index	Pure-Python BM25 is enough at 6,000 documents and keeps the sparse path dependency-light
scikit-learn	Embeddings (TF-IDF + truncated SVD)	CPU-only deterministic embedding space; no GPU or model download needed for clone-run-verify
Author-built modules	RRF fusion, LRU/TTL cache, metrics registry, guardrail, corpus generator, extractive LLM adapter	Core logic is original and unit tested; libraries provide index structures and graph execution only
pytest + ruff + GitHub Actions	Tests, lint, CI gate	CI fails under 80% coverage; measured coverage is 95%
Docker + Kubernetes manifests	Packaging and deploy skeleton	Compose and k8s YAML are schema-validated in CI (this environment has no Docker daemon, so they are not integration tested here)

Quickstart

Requires Python 3.10+ and about 2 GB of free RAM.

git clone https://github.com/panchalvedant13/enterprise-multi-agent-knowledge-platform.git
cd enterprise-multi-agent-knowledge-platform
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python scripts/build_index.py --docs 6000 --queries 200   # about 10 s on 2 vCPU
.venv/bin/uvicorn emkp.api.app:app --app-dir src --port 8000 &
sleep 5
curl -s http://127.0.0.1:8000/healthz
curl -s -X POST http://127.0.0.1:8000/query -H "Content-Type: application/json" \
  -d '{"query": "settlement and chargeback issues reported by swiftpay", "top_k": 5}'

The second curl returns an extractive answer with citations, passages, and an agent trace. To run the test suite: .venv/bin/pip install -r requirements-dev.txt, then .venv/bin/pip install -e . --no-deps, then .venv/bin/pytest (58 tests). To reproduce the benchmarks: benchmark/retrieval_eval.py, then benchmark/build_flat_index.py and two benchmark/load_test.py runs (flags in each script's docstring).

Performance under load

Methodology: an asyncio load generator (committed at benchmark/load_test.py) drives the running service with seeded traffic drawn from the labeled queries, 200 requests per concurrency level after 20 warmup requests, identical traffic for both conditions. Conditions: exact flat FAISS with cache disabled vs IVF ANN (nprobe=8) with the in-process LRU cache. Hardware: 2 vCPU, 4GB shared container, one uvicorn worker, localhost.

xychart-beta
    title "IVF + cache latency vs concurrency (ms); series top to bottom at x=32: p99, p95, p50"
    x-axis "concurrency" [4, 16, 32]
    y-axis "latency (ms)" 0 --> 900
    line [162.1, 336.5, 846.9]
    line [119.9, 190.2, 614.6]
    line [53.1, 32.2, 150.9]

Loading

Condition	Concurrency	p50 ms	p95 ms	p99 ms	Throughput rps
flat, no cache	4	102.2	137.2	186.8	36.7
flat, no cache	16	396.5	480.2	513.3	37.9
flat, no cache	32	813.2	859.1	876.7	38.4
IVF + cache	4	53.1	119.9	162.1	62.2
IVF + cache	16	32.2	190.2	336.5	234.0
IVF + cache	32	150.9	614.6	846.9	134.5

Raw JSON: benchmark/results/load_flat_nocache.json, benchmark/results/load_ivf_cache.json. Honest degradation: at concurrency 32 the p99 advantage nearly disappears (846.9 ms vs 876.7 ms) because cache misses queue behind a single CPU-bound worker, so the cache mostly protects the median, not the tail.

Architecture decisions

Two ADRs in docs/adr/:

• ADR-001: hybrid RRF fusion instead of cross-encoder reranking under a CPU-only latency budget.
• ADR-002: local FAISS + BM25 instead of a managed vector database for this deployment profile.

Intentionally out of scope

• Neural embedding models (sentence-transformers): adopt when the corpus contains real paraphrase variation that LSA cannot capture and a GPU or embedding API budget exists.
• Cross-encoder reranking: adopt when precision@5 requirements exceed what fusion delivers and the latency budget grows past roughly 100 ms per query (see ADR-001).
• Distributed index sharding: adopt when the corpus approaches millions of documents or the index no longer fits in one node's RAM.
• Authentication and multi-tenancy: adopt before exposing the service beyond a trusted internal network.
• Generative (non-extractive) answer synthesis: adopt when a hosted or local generative model is available; the LlmAdapter interface in src/emkp/llm/adapter.py is the seam.

Security and Compliance

• All secrets arrive via environment variables (EMKP_REDIS_URL is the only secret-bearing setting); nothing secret is committed.
• Production path: inject env vars from Vault or a cloud secret manager (AWS Secrets Manager / GCP Secret Manager) at deploy time; the k8s manifests read from a Secret resource.
• Never logged: raw query text (only a salted hash via hash_query), document contents, cache values. Logs are structured JSON with request IDs.
• Guardrail agent rejects prompt-injection patterns, empty queries, and queries over EMKP_MAX_QUERY_CHARS before any retrieval runs.

Failure Modes

Failure	Detection	Behavior	Recovery
Index artifacts missing or corrupt	Startup load raises; error logged with type	Service rebuilds the index from documents.jsonl before serving	Automatic at startup; /healthz stays 503 until ready
Corpus file missing	Startup raises FileNotFoundError with the path	Fail fast, no half-ready service	Run scripts/build_index.py, restart
Cache down (Redis unreachable, or get/set throws)	Connection error at startup or per-request exception, logged as warning	Falls back to in-process LRU at startup; per-request errors recompute the answer	Automatic; no query ever fails because of the cache
Malformed or hostile query	Pydantic validation and guardrail agent	HTTP 422 with reason; injection patterns logged by hash only	None needed; request isolated
Disk full	Index save at startup raises OSError	Startup fails loudly with the exception (query path itself does no disk writes)	Free space, restart; serving resumes from existing artifacts

Hardest problem solved

The corpus generator computes labeled queries by sampling (topic, entity) pairs that have enough relevant documents to score against. The original code used a hard floor of 10 documents per pair. On corpora under roughly 800 documents (exactly what the fast test suite builds), entities are spread thin enough that no pair qualified, the eligible list came back empty, and query sampling crashed with ZeroDivisionError from eligible[qid % len(eligible)], a modulo by zero two lines away from the real cause. The fix (commit 8d834f6) scales the floor with corpus size, max(3, min(10, n_docs // (len(TOPICS) * 10))), and raises a ValueError naming the actual constraint when nothing qualifies, so the failure now reports the cause instead of a symptom. The regression test builds a small corpus and asserts both the successful path and the loud failure path.

Future work

• Swap LSA for a small ONNX sentence-embedding model and re-run benchmark/retrieval_eval.py to quantify the quality delta against the same labels.
• Add a SQL worker agent for structured questions; the supervisor routing map is the single integration point.
• Shard-aware index loading so multiple uvicorn workers share memory-mapped FAISS segments.
• Prometheus histogram export from the metrics registry instead of the current text summary.
• Rerun the load benchmark on dedicated hardware to separate scheduler noise from queueing effects at concurrency 32.

License

MIT. Corpus is synthetic (generator committed at src/emkp/corpus/generator.py); all numbers above were measured by executing the committed benchmark code, raw outputs in benchmark/results/.