A production-grade, cost-aware fraud decisioning service for Indian payment infrastructure.
Data: PaySim Synthetic Dataset (Kaggle) — a synthetic mobile-money simulator. Results reflect synthetic data characteristics. The model is designed for real-time payment fraud detection and will require retraining and recalibration on live transaction data before production use. Near-perfect metrics (PR-AUC 0.9999) are expected on PaySim because the dataset encodes fraud via deterministic accounting anomalies — real-world fraud is noisier.
Track: AI Risk Manager — Track 02, Razorpay Buildathon 2026
Author: Sayantan Mandal, Gati Shakti Vishwavidyalaya
RazorSentry scores every transaction in real time, explains why, and writes a tamper-evident blinded audit record — in under 60ms.
- Cost-aware threshold — threshold chosen by rupee net savings, not F1
- Three-tier policy — BLOCK / REVIEW / APPROVE with SHAP reason codes
- PII blinding — HMAC-SHA256 hashes identifiers before audit log storage
- Live monitoring — EWMA spike detector + PSI drift detector on the dashboard
- Async scoring path — Redis Queue decouples burst acceptance from processing
- Razorpay webhook — POST /webhook/razorpay accepts payment.failed events natively
For known limitations, honest failure modes, and future scope see LIMITATIONS.md
For full architecture details see ARCHITECTURE.md.
For the development journey, decisions, and what broke see DEVLOG.md.
Trained and evaluated on PaySim synthetic data. These numbers reflect synthetic data — see the note at the top.
| Metric | Value |
|---|---|
| PR-AUC | 0.9999 |
| Precision @ threshold | 1.000 |
| Recall @ threshold | 0.9998 |
| Operating threshold | 0.35 (isotonic calibration) |
| False positives | 0 on test set |
| False negatives | 1 on test set |
| Net savings (test set) | ₹195.36 Cr across 101,643 transactions |
| False positive cost assumption | ₹150/txn |
The flat curve reflects PaySim's deterministic fraud encoding. On real data this curve has a clear peak — the cost model still applies and will select the optimal threshold on live data.
Isotonic calibration moved the threshold from 0.02 to 0.35 — scores are now interpretable probabilities.
| Component | Technology | Purpose |
|---|---|---|
| API Server | FastAPI + 4 uvicorn workers | Parallel request handling |
| Sync scoring | POST /score | Sub-60ms real-time decisions |
| Async scoring | POST /score/async + Redis RQ | Burst traffic decoupling |
| Database | PostgreSQL 16 (Docker) | Append-only audit log with connection pooling |
| Queue | Redis 7 (Docker) | RQ job queue for async scoring |
| Privacy | HMAC-SHA256 PII blinding | Account IDs never stored in plaintext |
| Monitoring | GET /dashboard | Live fraud ops dashboard, auto-refresh 10s |
| Drift detection | PSI monitor | Alerts when incoming distribution shifts from training |
100 concurrent users · 60 seconds · MacBook Air M1 · docker compose up (4 workers + PostgreSQL + Redis)
| Metric | /score [legit] | /score [fraud] | /health |
|---|---|---|---|
| Requests | 9,440 | 2,412 | 1,174 |
| Failures | 29 (0.31%) | 7 (0.29%) | 0 |
| p50 latency | 48ms | 48ms | 12ms |
| p95 latency | 540ms | 580ms | 280ms |
| p99 latency | 870ms | 860ms | 600ms |
| Throughput | 159.4 req/s | 40.7 req/s | 19.8 req/s |
Total: 219.9 req/s sustained · 99.7% success rate
Bottleneck: SHAP TreeExplainer adds ~10-15ms per request. Production mitigation: async SHAP annotation after the decision is logged. Horizontal scaling grows throughput linearly.
Prerequisites: Docker Desktop installed and running. That is all. No Python install needed. No database setup. One command starts everything.
# 1. Clone the repo
git clone https://github.com/Sayantan181222/RazorSentry.git
cd RazorSentry
# 2. Copy environment variables
cp .env.example .env
# Edit .env and add your GROQ_API_KEY (optional — service works without it)
# PII_SALT is pre-filled with a demo value — change it for any real use
# 3. Start all services (PostgreSQL + Redis + RazorSentry + RQ worker)
docker compose up --build -d
# 4. Wait ~30 seconds then verify everything is healthy
docker compose ps
# All four services should show as healthy or running
# 5. Open the live dashboard
open http://localhost:8000/dashboard
# 6. Run the demo
bash scripts/demo_curl.sh
# 7. Inspect the PostgreSQL audit log
make db-shell
# Inside psql: SELECT transaction_id, decision, score FROM decisions LIMIT 5;Total time from clone to running dashboard: under 10 minutes on a standard connection.
No PaySim download needed for the demo.
data/sample_transactions.csvis included for quick testing. For full model training, download PaySim from Kaggle and place it atdata/PaySim.csv, then runpython src/data_loader.pyandpython src/train.py. The trained model is already committed tomodels/lgbm_model.pklso training is not required to run the service.
If Docker is not available:
pip install -r requirements.txt
# Use SQLite instead of PostgreSQL
export DATABASE_URL=sqlite:///razorsentry.db
export MODEL_PATH=models/lgbm_model.pkl
export THRESHOLD_PATH=models/threshold.txt
# Start the service (single worker, no Redis queue)
uvicorn src.service:app --reload --port 8000
# In a separate terminal, test it
curl -s http://localhost:8000/health | python3 -m json.toolNote: Without Docker, async scoring (/score/async) is unavailable as it requires Redis. All other endpoints including /score, /dashboard, /monitor/spike, and /monitor/drift work normally.
The 5-minute pitch video covers:
- Live scoring demo via
bash scripts/demo_curl.sh - Async scoring via POST /score/async with job polling
- Dashboard at
http://localhost:8000/dashboardshowing live decisions - Data drift simulation via
python scripts/simulate_drift.py— dashboard flips from ✅ Stable to 🔴 DRIFT ALERT (PSI > 0.2) in real time - Metrics walkthrough: PR curve, cost curve, confusion matrix, top FP cases
- Architecture explanation and what broke
Video link: [To be added after recording]
| Method | Path | Description |
|---|---|---|
| POST | /score |
Sync scoring — decision in under 60ms |
| POST | /score/async |
Async — returns job_id, poll for result |
| GET | /score/result/{job_id} |
Poll async result |
| POST | /batch |
Score a list of transactions |
| POST | /webhook/razorpay |
Ingest Razorpay payment.failed events |
| GET | /decisions/{decision_id} |
Retrieve decision by UUID |
| GET | /monitor/spike |
EWMA fraud spike status |
| GET | /monitor/drift |
PSI feature drift status |
| GET | /dashboard |
Live monitoring dashboard |
| GET | /health |
Service liveness |
| GET | /ready |
Readiness — 503 until model and DB are live |
| GET | /health/pool |
PostgreSQL connection pool stats |
LightGBM makes every money decision. Groq LLaMA (llama-3.1-8b-instant) drafts a 2-line analyst note for REVIEW-queue items only — after the decision is already final. The LLM never touches the scoring path. The EWMA spike monitor and PSI drift detector are deliberately LLM-free: pure statistics, fast, no hallucination risk.
python scripts/simulate_drift.pySends 50 normal transactions then pauses. Open the dashboard. Press Enter to send 50 drifted transactions simulating a festival-season CASH_OUT fraud ring (₹4.5L-9.5L). The Feature Drift Monitor flips from ✅ Stable to 🔴 DRIFT ALERT (PSI > 0.2) in real time.
Temporal leakage — velocity features computed before the train/test split inflated PR-AUC. Fixed by computing all features strictly within split boundaries.
LabelEncoder at inference — re-fitting on a single row gave wrong type encoding. Fixed with a hardcoded TYPE_ENCODING dict.
drain_flag on zero-balance accounts — amount >= 0.9 * 0 is always True. Fixed with a non-zero guard on oldbalanceOrg.
CI/CD: ModuleNotFoundError — pytest could not find src in GitHub Actions. Fixed by adding sys.path.insert to conftest.py directly.
PII blinding not firing — 2006 old records stored raw account IDs before privacy.py was wired in. Confirmed fix: SENSITIVE_ACCOUNT_99999 stored as 2e10c17cfbb44f3d in PostgreSQL.
Load test 89.6% failures — all 429s from the per-IP rate limiter. Fixed by adding LOAD_TEST_MODE env flag to bypass rate limiting for benchmarking.
| Field | Detail |
|---|---|
| Training data | PaySim (Kaggle) — synthetic — 500k legit + all fraud rows |
| Fraud rate (train) | 0.98% |
| Fraud rate (test) | 4.18% — fraud concentrates in later time steps |
| Features | balance_error_orig, balance_error_dest, drain_flag, zero_orig_after, type_encoded, amount_log, orig_txn_count_1h, orig_txn_sum_1h, dest_in_degree_1h, high_amount_flag |
| Model | LightGBM + Isotonic calibration (CalibratedClassifierCV) |
| Threshold | 0.35 — chosen by rupee net-savings maximisation, not F1 |
| Not intended for | Deployment on real data without retraining · High-value (>₹10L) decisions without human review · Offensive fraud research |