Observability: Add structured logging, metrics, and tracing

[OneByJorah]

Observability Gap

Current State: Basic logging with Python standard logging. No metrics, no distributed tracing, no structured logging.

Required:

1. Structured Logging:

   • Replace basic logging with structlog or python-json-logger
   • Add correlation IDs for request tracing
   • Structured log format (JSON) for log aggregation
   • Log levels: DEBUG, INFO, WARN, ERROR with consistent fields

2. Metrics (Prometheus):

   • Add prometheus-client
   • Expose /metrics endpoint
   • Metrics: HTTP request duration, request count, error rate, active connections, WebSocket connections, DB query duration, recording duration, stream health
   • Custom business metrics: active streams, recordings count, events per type

3. Distributed Tracing:

   • Add OpenTelemetry
   • Trace HTTP requests, DB queries, external calls
   • Export to Jaeger or Tempo

4. Health Checks:

   • Enhance /health endpoint with detailed dependency checks
   • Add readiness/liveness probes for Kubernetes

5. Frontend Observability:

   • Add error boundary with Sentry or similar
   • Add performance monitoring (Web Vitals)
   • Log frontend errors to backend

Priority: HIGH for production