RAG Chat Backend

A production-style Retrieval-Augmented Generation (RAG) chat API built with FastAPI, Groq (Llama 3.3), ChromaDB, and Postgres. Upload PDF documents, then ask questions grounded strictly in their content — with persistent, multi-turn conversation memory.

Note on Implementations:

• main branch (Current): Built entirely from scratch with custom RAG logic and persistent Postgres database storage.
• langchain-version branch: A refactored implementation utilizing LangChain for orchestration and in-memory storage for rapid prototyping.

Features

• Document ingestion — upload PDFs, automatically chunked (paragraph/sentence-aware), embedded locally, and stored in a vector database.
• Grounded Q&A — answers are retrieved from your documents, not the model's general knowledge. The system explicitly refuses to guess when context doesn't cover a question.
• Multi-turn conversations — history is stored server-side in Postgres, keyed by conversation_id, not managed by the client.
• Follow-up aware retrieval — recent conversation turns are folded into the retrieval query, so ambiguous follow-ups ("what about after that?") still search correctly.
• Free to run locally — embeddings run on CPU via sentence-transformers, ChromaDB is a local folder, Groq's free tier powers generation.
• Self-hostable — designed to be handed off and run entirely on someone else's infrastructure; no external API dependency besides Groq.

Architecture

main.py                  # FastAPI app, wires routers together
config.py                 # Centralized settings (.env-driven)
routers/
  chat.py                 # /chat endpoint
  documents.py             # /upload-doc endpoint
models/
  schemas.py               # Pydantic request/response models
services/
  groq_client.py           # Async Groq client + logging setup
  embedding_service.py     # Local sentence-transformers embedding model
  vector_store.py           # ChromaDB client, query + upsert logic
  document_service.py       # PDF loading + chunking
  rag_pipeline.py            # Retrieval, prompt construction, DB-backed history
  database.py                # SQLAlchemy async engine + models (Conversation, Message)

How a /chat request flows

1. Client sends { question }. No history, no context, no token limits — all server-controlled. The conversation ID is tracked automatically via HTTP cookies.
2. Server fetches the last N turns for that conversation from Postgres (HISTORY_WINDOW, default 3).
3. Those turns + the current question are concatenated into a retrieval query and embedded.
4. ChromaDB returns the top matching document chunks.
5. A system prompt wraps those chunks with explicit grounding instructions ("answer only from context, say so if you don't know").
6. The full message list (system + recent history + question) is sent to Groq (async, non-blocking).
7. Both the new question and the model's answer are saved to Postgres.
8. Response includes { response, conversation_id } and a Set-Cookie header — the client's browser automatically passes the cookie back on the next call to continue the same conversation.

Document upload flow

1. PDF is loaded page-by-page via pypdf.
2. Text is split using RecursiveCharacterTextSplitter (tries paragraph breaks first, then lines, then sentences — avoids cutting mid-sentence).
3. Chunks are batch-embedded locally (all-MiniLM-L6-v2).
4. Any existing chunks tagged with the same filename are deleted first (via ChromaDB metadata filtering), then new chunks are inserted — so re-uploading a file cleanly replaces the old version instead of duplicating it.

Setup

pip install -r requirements.txt

.env:

GROQ_API_KEY=your_key_here
DATABASE_URL=postgresql+asyncpg://postgres:yourpassword@localhost:5432/rag_chat
HISTORY_WINDOW=3
DEFAULT_MAX_TOKENS=500

Postgres (via Docker):

docker run --name rag-postgres -e POSTGRES_PASSWORD=yourpassword -e POSTGRES_DB=rag_chat -p 5432:5432 -d postgres
python create_tables.py

Run:

uvicorn main:app --reload

Design decisions worth knowing

• max_tokens and context are never client-supplied — both are computed/controlled server-side. This prevents cost abuse and keeps retrieval logic out of the client's hands.
• History windowing (last N turns) doubles as the free alternative to LLM-based query rewriting — instead of paying for an extra LLM call to disambiguate follow-ups, recent turns are folded directly into the retrieval query.
• Server-side conversation storage, not client-managed — matches how real products (ChatGPT, etc.) work, and avoids the client having to resend a growing history payload on every request.
• Metadata-based document replacement — re-uploading a file with the same name cleanly replaces its old chunks rather than duplicating or colliding on IDs.

Future Roadmap & Enhancements

• Advanced Query Rewriting: Implementing dedicated LLM-based query rewriting to complement the current cost-effective history-folding approach.
• Enhanced Security: Adding encryption-at-rest and strict access-control hardening for enterprise-grade conversation storage.
• Traffic Management: Introducing rate-limiting and API quota management for public-facing deployments.
• Summarization Workflows: Extending the architecture with dedicated endpoints designed for broad document summarization, alongside the existing high-precision fact retrieval.