Improve gateway metrics and reliability

.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+          cache: true
+
+      - name: Vet
+        run: go vet ./...
+
+      - name: Race detector
+        run: go test -race ./...
+
+      - name: Internal package coverage
+        shell: bash
+        run: |
+          go test ./internal/... -coverprofile=coverage.out
+          go tool cover -func=coverage.out
+          coverage_value="$(go tool cover -func=coverage.out | awk '/^total:/ {gsub("%", "", $3); print $3}')"
+          awk -v coverage="$coverage_value" 'BEGIN { if (coverage < 65) { print "coverage " coverage "% is below 65%"; exit 1 } }'
+
+      - name: Upload coverage profile
+        uses: actions/upload-artifact@v4
+        with:
+          name: coverage
+          path: coverage.out

Makefile

@@ -1,8 +1,18 @@
-.PHONY: test run compose-up compose-down load-baseline load-rate-limit load-cache load-failure
+.PHONY: test test-cover test-race benchmark run compose-up compose-down load-baseline load-rate-limit load-cache load-failure load-scale load-all
 
 test:
 	go test ./...
 
+test-cover:
+	go test ./internal/... -coverprofile=coverage.out
+	go tool cover -func=coverage.out
+
+test-race:
+	go test -race ./...
+
+benchmark:
+	go test -run '^$$' -bench BenchmarkGatewayUncached -benchmem ./internal/proxy
+
 run:
 	go run ./cmd/gateway -config deploy/docker/gateway.yaml
 
@@ -23,3 +33,8 @@ load-cache:
 
 load-failure:
 	k6 run loadtests/upstream_failure.js
+
+load-scale:
+	k6 run loadtests/scale.js
+
+load-all: load-baseline load-cache load-rate-limit load-failure load-scale

README.md

@@ -6,6 +6,7 @@
 ![Prometheus](https://img.shields.io/badge/Prometheus-metrics-E6522C?logo=prometheus&logoColor=white)
 ![Grafana](https://img.shields.io/badge/Grafana-dashboards-F46800?logo=grafana&logoColor=white)
 ![k6](https://img.shields.io/badge/k6-load_tested-7D64FF?logo=k6&logoColor=white)
+[![CI](https://github.com/Arnavsharma2/Distributed-API-Gateway/actions/workflows/ci.yml/badge.svg)](https://github.com/Arnavsharma2/Distributed-API-Gateway/actions/workflows/ci.yml)
 
 A distributed API gateway in Go — adds rate limiting, caching, retries, circuit breaking, and Prometheus observability to backend services across multiple replicas without changing application code.
 
@@ -17,8 +18,9 @@ A distributed API gateway in Go — adds rate limiting, caching, retries, circui
 | Go backend engineering | Idiomatic Go: interfaces, goroutines, structured JSON logging, YAML config parsing |
 | Redis patterns | Atomic sliding-window rate limiter implemented as a Lua script to eliminate race conditions under concurrent replicas |
 | Resilience engineering | Circuit breaker, exponential-backoff retry, and fail-open degradation when Redis is unavailable |
-| Observability | 7 Prometheus counters/histograms, provisioned Grafana dashboard, structured per-request logs |
-| Performance testing | k6 load tests across 5 scenarios; p95/p99 latency measured at up to 1,000 RPS with active circuit breaking and retries |
+| Observability | 16 low-cardinality Prometheus metric families, RED/dependency dashboard, recording rules, and actionable alerts |
+| Performance testing | Isolated uncached, cache, rate-limit, failure, and scale workloads with p95/p99, failure-rate, and dropped-iteration gates |
+| Quality gates | GitHub Actions runs vet, the race detector, and a 65% internal-package coverage floor (67% current) |
 | Container orchestration | Multi-service Docker Compose stack: Nginx load balancer, 3 gateway replicas, Redis, Prometheus, Grafana |
 
 ## Architecture
@@ -43,17 +45,51 @@ flowchart LR
 
 ## Benchmark Results
 
-| Scenario | Replicas | RPS | p95 latency | p99 latency | Notes |
+The baseline now uses `/api/benchmark/products`, an intentionally uncached and
+unlimited route. This isolates proxy overhead instead of accidentally measuring
+Redis cache performance. At a fixed 2,000 RPS arrival rate, adding an Nginx
+upstream keep-alive pool removed per-request TCP setup and source-port exhaustion:
+
+| Single-replica baseline | Actual RPS | HTTP failures | p95 latency | p99 latency | Dropped iterations |
+|---|---:|---:|---:|---:|---:|
+| Before upstream keep-alive | 1,527.12 | 5.90% | 215.82ms | 251.41ms | 13,803 |
+| After upstream keep-alive | **1,992.24** | **0.00%** | **0.97ms** | **1.64ms** | **228** |
+
+That is 30% more completed requests per second, no HTTP failures, and a 99.5%
+reduction in p95 latency under the same workload. These are fixed-arrival load
+tests, not claims about an absolute hardware ceiling.
+
+The isolated scenario suite now reports:
+
+| Scenario | Replicas | Target / actual RPS | p95 | p99 | Scenario signal |
 |---|---:|---:|---:|---:|---|
-| Baseline smoke | 3 | 18.69 | 8.73ms | 18.38ms | Sanity pass |
-| Rate-limit smoke | 3 | 200.01 | 6.42ms | 11.75ms | Global sliding-window active |
-| Baseline products | 1 | 244.99 | 6.02ms | 46.96ms | Single-instance ceiling |
-| Baseline products | 3 | 500.01 | 4.40ms | 8.31ms | **2× throughput, 5× p99 improvement** vs. 1 replica |
-| Cache pressure | 3 | 999.94 | 2.20ms | 6.24ms | ~1,000 RPS via Redis cache |
-| Rate-limit pressure | 3 | 200.02 | 6.11ms | 12.17ms | Global enforcement across all replicas |
-| Upstream failure | 3 | 100.02 | 7.13ms | 13.68ms | Retries and circuit breaking active |
+| Uncached proxy | 1 | 2,000 / 1,992.24 | 0.97ms | 1.64ms | 0% HTTP failures |
+| Uncached proxy | 3 | 2,000 / 1,981.87 | 1.21ms | 2.73ms | 0% HTTP failures |
+| Redis cache | 3 | 1,000 / 996.70 | 1.33ms | 2.92ms | 99.85% hit rate |
+| Global rate limit | 3 | 200 / 200.00 | 1.40ms | 1.99ms | Exactly 40 of 6,001 requests admitted |
+| Sustained upstream failure | 3 | 100 / 100.02 | 1.52ms | 3.19ms | 99.50% rejected by open circuits |
+
+The single upstream is shared by all replicas, so the three-replica result is a
+resilience check at this arrival rate, not a throughput multiplier. The global
+limiter test also verifies that Redis sorted-set members include a random
+replica ID; timestamp-plus-local-counter members previously collided across
+replicas and admitted 43 requests against a limit of 40.
 
-Scaling from 1 → 3 replicas doubled throughput and cut p99 latency from 46.96ms to 8.31ms on the products route.
+Reproduce the current baseline with:
+
+```bash
+docker compose up -d --build --scale gateway=1
+docker run --rm -i \
+  -v "$PWD/loadtests:/scripts" \
+  grafana/k6:0.54.0 run \
+  -e BASE_URL=http://host.docker.internal:8080 \
+  -e RATE=2000 /scripts/baseline.js
+```
+
+Measurements were collected on 2026-08-10 using Docker Desktop and k6 0.54.0.
+The test ran for 30 seconds after the stack was warm. Results will vary by host,
+so the scripts enforce service-level thresholds and report load-generator drops
+separately from HTTP failures.
 
 ## Features
 
@@ -64,7 +100,7 @@ Scaling from 1 → 3 replicas doubled throughput and cut p99 latency from 46.96m
 - Retry policy with exponential backoff for transient upstream failures.
 - Circuit breaker that opens after repeated failures and recovers after cooldown.
 - Structured JSON logs with request ID, route, status, cache status, rate-limit status, and latency.
-- Prometheus metrics and a provisioned Grafana dashboard.
+- Prometheus RED and dependency metrics, recording rules, alerts, and a provisioned Grafana dashboard.
 - k6 scripts for baseline, rate-limit, cache, failure, and scale experiments.
 
 ## Quickstart
@@ -93,6 +129,11 @@ curl -s http://localhost:9090/metrics | grep gatekeeper
 # Unit tests
 go test ./...
 
+# Race detector, coverage, and microbenchmark
+make test-race
+make test-cover
+make benchmark
+
 # Load tests
 k6 run loadtests/baseline.js
 k6 run loadtests/cache.js
@@ -143,14 +184,29 @@ If Redis is unavailable, the gateway fails open — rate limiting and cache are
 | Metric | Description |
 |---|---|
 | `gatekeeper_requests_total` | Request count by route and status |
-| `gatekeeper_request_duration_seconds` | Latency histogram (p50 / p95 / p99) |
-| `gatekeeper_rate_limited_total` | Rate-limited requests by route |
+| `gatekeeper_requests_in_flight` | Current concurrency by route and method |
+| `gatekeeper_request_duration_seconds` | End-to-end latency by route, method, and status class |
+| `gatekeeper_response_size_bytes` | Response-body size distribution |
 | `gatekeeper_cache_events_total` | Cache hits and misses |
-| `gatekeeper_upstream_errors_total` | Upstream error count |
-| `gatekeeper_retries_total` | Retry attempts by route |
-| `gatekeeper_circuit_state` | Circuit breaker state (0 = closed, 1 = open) |
-
-The Grafana dashboard visualizes request rate, p95/p99 latency, cache hit rate, upstream errors, and circuit state in real time.
+| `gatekeeper_rate_limit_decisions_total` | Allowed, limited, and fail-open decisions |
+| `gatekeeper_upstream_requests_total` | Upstream attempts by outcome |
+| `gatekeeper_upstream_request_duration_seconds` | Upstream latency by route and outcome |
+| `gatekeeper_redis_operations_total` | Redis operations and errors by operation |
+| `gatekeeper_redis_operation_duration_seconds` | Redis dependency latency |
+| `gatekeeper_retries_total` | Retry attempts by route and reason |
+| `gatekeeper_circuit_state` | One-hot closed, open, and half-open state |
+| `gatekeeper_circuit_transitions_total` | Circuit state changes |
+| `gatekeeper_circuit_rejected_total` | Requests rejected by open circuits |
+
+The request histogram starts at 0.5ms, so Prometheus can resolve the low-millisecond
+percentiles produced by the local benchmark. The Grafana dashboard covers request
+rate, 5xx ratio, p50/p95/p99 latency, concurrency, cache effectiveness, upstream
+latency, Redis latency, rate-limit decisions, retries, and circuit behavior. Its
+route variable supports both focused debugging and fleet-wide views.
+
+Prometheus also provisions five alerts: target down, high 5xx ratio, high p95
+latency, Redis errors, and an open circuit. Three recording rules precompute the
+5-minute request rate, 5xx ratio, and p95 latency used for SLO monitoring.
 
 ## Failure Modes
 
@@ -159,4 +215,4 @@ The Grafana dashboard visualizes request rate, p95/p99 latency, cache hit rate,
 | Redis unavailable | Fails open — rate limiting and cache bypass; requests continue to upstream |
 | Upstream intermittent 5xx | Retries with exponential backoff; records retry and error metrics per attempt |
 | Upstream sustained failure | Circuit breaker opens after threshold; returns `503` until cooldown half-open trial |
-| Multiple gateway replicas | All replicas share Redis state — limits are enforced globally, not per process |
+| Multiple gateway replicas | All replicas share Redis state — limits are enforced globally, not per process |

deploy/docker/gateway.yaml

@@ -6,6 +6,52 @@ redis:
   url: redis://redis:6379
 
 routes:
+  # Dedicated uncached route for measuring proxy overhead. Keeping benchmark
+  # traffic away from Redis avoids conflating gateway, cache, and rate-limit cost.
+  - name: benchmark-products
+    path_prefix: /api/benchmark/products
+    upstream_url: http://mock-api:3000/products
+    rate_limit:
+      enabled: false
+    cache:
+      enabled: false
+    retry:
+      enabled: false
+    circuit_breaker:
+      enabled: false
+
+  # Dedicated cache workload: cache behavior is measured without unrelated
+  # rate-limit rejections changing the result.
+  - name: benchmark-cache
+    path_prefix: /api/benchmark/cache/products
+    upstream_url: http://mock-api:3000/products
+    rate_limit:
+      enabled: false
+    cache:
+      enabled: true
+      ttl_seconds: 30
+    retry:
+      enabled: false
+    circuit_breaker:
+      enabled: false
+
+  # Dedicated rate-limit workload backed by a stable upstream.
+  - name: benchmark-rate-limit
+    path_prefix: /api/benchmark/rate-limit
+    upstream_url: http://mock-api:3000/products
+    rate_limit:
+      enabled: true
+      key: header
+      header: X-User-ID
+      limit: 40
+      window_seconds: 60
+    cache:
+      enabled: false
+    retry:
+      enabled: false
+    circuit_breaker:
+      enabled: false
+
   - name: products
     path_prefix: /api/products
     upstream_url: http://mock-api:3000/products