Sentinel is a multi-provider AI Gateway designed to sit between applications and LLM providers, providing a single API interface while handling reliability, performance, cost, and traffic-management concerns.
Instead of an application communicating directly with a single LLM provider, requests are sent to Sentinel through a unified /chat endpoint. Sentinel decides how the request should be handled and applies a series of infrastructure layers before communicating with the selected provider.
The current implementation supports Groq as the primary provider and OpenAI as a secondary provider/fallback.
Client
│
▼
Sentinel /chat
│
├── Request Queue / Backpressure
│
├── Exact + Semantic Cache
│
├── Token Bucket Rate Limiter
│
├── Circuit Breaker
│
├── Provider Routing
│
├───────────────┐
▼ ▼
Groq OpenAI
Primary Fallback
│ │
└───────┬───────┘
▼
Response
Applications interact with a single POST /chat endpoint rather than integrating separately with each LLM provider.
Requests can specify a provider, while Groq is used as the default provider.
Sentinel currently integrates:
- Groq : primary provider
- OpenAI (
gpt-4o-mini) : secondary provider and automatic fallback
The provider layer is designed so additional providers can be introduced later without changing the gateway's public API.
Sentinel uses an in-memory token bucket to control request bursts.
The current configuration allows an initial burst of 5 requests and refills at 1 token per second.
Requests that exceed the available tokens are rejected with HTTP 429 Too Many Requests.
Responses are cached using a SHA-256 hash derived from the provider and prompt.
Identical requests can therefore be served directly from memory without contacting the LLM provider again.
Sentinel extends exact-match caching with embedding-based semantic similarity.
Prompts are converted into embeddings using all-MiniLM-L6-v2, and cosine similarity is used to identify semantically similar requests.
A similarity threshold of 0.80 was selected based on testing.
This allows requests with different wording but similar meaning to reuse previously generated responses.
Sentinel implements a three-state circuit breaker for the primary Groq provider:
CLOSED → OPEN → HALF_OPEN → CLOSED
Repeated Groq failures open the circuit and prevent further requests from being sent to the unhealthy provider until the recovery timeout expires.
When Groq fails, Sentinel automatically falls back to OpenAI rather than returning the provider failure directly to the client.
This provides provider-level resilience and allows Sentinel to continue serving requests during an upstream outage.
Sentinel includes an in-memory request queue that limits the number of requests being processed concurrently.
When the configured capacity is reached, additional requests are rejected with HTTP 503 Service Unavailable rather than allowing uncontrolled request growth.
Request cleanup is guaranteed using try/finally so queue capacity is released even when request processing fails.
A Python load-testing client using asyncio and httpx generates concurrent requests against Sentinel and records:
- HTTP status code
- Cache status
- Failover status
- Individual request latency
- Aggregate statistics
This provides a reproducible way to measure Sentinel's behavior under burst traffic and provider failures.
Benchmarks below are observed results from local load tests using 10 concurrent requests. They are experimental measurements rather than universal performance guarantees.
Using the same 10-request workload:
| Metric | Result |
|---|---|
| Cache hit rate on repeated workload | 100.00% |
| Average latency with cache hits | 424.89 ms |
| Average successful-request latency with cache disabled | 4407.02 ms |
| Observed latency reduction | ~90.4% |
The cache-enabled workload served repeated requests without contacting the LLM provider.
The cache-disabled comparison required provider calls for the successful requests, demonstrating the latency benefit of reusing cached responses.
Semantic-cache testing also verified that sufficiently similar prompts could reuse cached responses at a configured similarity threshold of 0.80.
A burst of 10 concurrent requests was tested with the token bucket enabled and disabled.
| Configuration | Successful | Rejected |
|---|---|---|
| Limiter disabled | 10 | 0 |
| Limiter enabled | 5 | 5 |
With the limiter enabled, 50% of the burst was throttled with HTTP 429 Too Many Requests.
This confirmed that the token bucket correctly limits burst traffic according to its configured capacity.
Groq was deliberately made unavailable during testing to simulate an upstream provider failure.
| Metric | Observed Result |
|---|---|
| Failover attempts | 5 |
| Successful fallback requests | 5 |
| Failover success rate | 100.00% |
| Average Groq failure detection time | 367.15 ms |
| Average failover transition time | 0.30 ms |
| Average OpenAI fallback duration | 3253.45 ms |
| Average total recovery time | 3253.76 ms |
The results show that Sentinel detected Groq failures quickly and transitioned to OpenAI almost immediately. Most of the recovery time was spent waiting for the fallback provider to generate its response.
The 100% failover success rate refers to requests that were admitted by the rate limiter and actually entered the Groq → OpenAI failover path.
The request queue was tested with a deliberately small capacity and simulated long-running requests.
The test verified that:
- Requests within queue capacity were accepted.
- Requests arriving after the queue reached capacity were rejected with HTTP
503 Service Unavailable. - Queue capacity was released correctly after requests completed.
try/finallycleanup prevented requests from remaining permanently stuck in the queue after failures.
Sentinel currently provides:
- Unified LLM API
- Groq + OpenAI provider routing
- Token bucket rate limiting
- Exact-match caching
- Semantic caching
- Circuit breaker
- Automatic provider failover
- In-memory request queue and backpressure
- Async concurrent load testing
- Measured performance and reliability benchmarks
git clone https://github.com/Ritesh90256/Sentinel.git
cd SentinelWindows PowerShell:
python -m venv .venv
.venv\Scripts\Activate.ps1pip install -r requirements.txtCreate a .env file in the project root:
GROQ_API_KEY=your_groq_api_key
OPENAI_API_KEY=your_openai_api_key
RATE_LIMIT_ENABLED=true
uvicorn app.main:app --reloadThe API will be available at:
http://127.0.0.1:8000
Interactive API documentation is available at:
http://127.0.0.1:8000/docs
With Sentinel running in one terminal, open another terminal, activate the virtual environment, and run:
python tests/load_test.pyThe load-testing script sends concurrent requests to Sentinel and reports response status, cache behavior, failover behavior, and latency.
Sentinel is a personal project focused on building an infrastructure-oriented AI Gateway that sits between applications and multiple LLM providers. Instead of requiring applications to handle provider-specific APIs, failures, traffic limits, and caching independently, Sentinel provides a unified /chat endpoint and manages these concerns centrally.
The project currently supports Groq as the primary provider and OpenAI as the fallback provider.
Sentinel combines several backend infrastructure patterns:
- Unified provider routing through a single API endpoint
- Token-bucket rate limiting to control burst traffic
- Exact-match caching using hashed request keys
- Semantic caching using embeddings and cosine similarity
- Circuit breaking to stop repeatedly calling an unhealthy provider
- Automatic Groq → OpenAI failover when the primary provider fails
- In-memory request queuing and backpressure to prevent uncontrolled request growth
- Asynchronous load testing using Python,
asyncio, andhttpx
The project was also tested rather than only implemented. Controlled experiments measured cache effectiveness, burst throttling, and provider failure recovery. The observed results demonstrated a 100% cache hit rate on the repeated workload, approximately 90.4% lower measured latency for cached responses, 5 of 10 burst requests throttled by the configured token bucket, and a 100% successful fallback rate among requests that entered the Groq → OpenAI failover path.
The load-testing and failure experiments also helped identify where time was actually being spent. In the provider-failure test, Sentinel detected Groq failures in an average of 367.15 ms and transitioned to OpenAI in approximately 0.30 ms, while most of the recovery time came from the fallback provider's response itself.
Overall, Sentinel evolved from a simple LLM API wrapper into a small but complete AI infrastructure gateway, combining performance optimization, traffic management, and provider resilience behind a single interface.