Semantic Cache

A configurable semantic caching and request-coalescing proxy for LLM providers. It sits between API clients and Ollama (or other LLM providers), reducing latency and cost by serving semantically similar responses from cache and preventing duplicate LLM calls through request coalescing.

Core capabilities

• Exact cache lookup — SHA-256 keyed by query + model + agent + namespace + tenant
• Semantic similarity lookup — cosine or Euclidean embedding comparison with configurable threshold
• Embedding generation — through configured embedding providers (Ollama, deterministic test providers)
• Response validation — reject empty, error-indicating, or malformed responses before caching
• Request coalescing (single-flight) — concurrent identical queries share one LLM execution
• Redis-backed coordination — distributed lock-based coalescing across multiple instances
• Local in-memory operation — runs without external dependencies for development and testing
• Deterministic test mode — in-memory stores and deterministic providers for reproducible testing
• Ollama integration — LLM generation and embedding via Ollama API
• API-key authentication — optional header-based key verification
• Metrics and observability — Prometheus endpoint, structured JSON logs, request IDs
• Configurable cache TTL — per-entry expiry, eviction strategies (TTL, LRU, LFU, size, hybrid TTL+LRU)
• Concurrency control — semaphore-based LLM request throttling with queue timeout
• Cache warm/invalidate endpoints — pre-populate or selectively evict cache entries

Prerequisites

• Python 3.12
• pip
• Docker and Docker Compose (for Redis and containerized deployment)
• Ollama (for LLM and embedding models)
• VS Code with Dev Containers extension (optional)

Quick start

# Clone
git clone git@github.com:uncle-voh-max/semantic-cache.git && cd semantic-cache

# Create and activate virtual environment
python3 -m venv .venv && source .venv/bin/activate

# Install with dev dependencies
pip install -e ".[dev]"

# Start Redis (required for distributed coalescing — optional for in-memory mode)
docker compose up -d redis

# Start Ollama and pull models
ollama pull llama3.2:3b
ollama pull nomic-embed-text

# Run the API
uvicorn src.main:app --reload --host 127.0.0.1 --port 8000

# Check health
curl http://localhost:8000/health/live

# Submit a query
curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What is semantic caching?", "agent": "default", "metadata": {"tenant": "local"}}'

By default the application uses in-memory cache stores (no Redis required). Add CONFIG_FILE=config/local.yaml to enable Redis.

Docker Compose

# Start full stack (API + Redis)
docker compose up -d

# Start only Redis
docker compose up -d redis

# View logs
docker compose logs -f

# Stop all services
docker compose down

# Remove volumes
docker compose down -v

Container networking: Inside Docker, the API container connects to Redis via the Docker Compose service name redis:6379 and to host Ollama via host.docker.internal:11434. The config/local.yaml uses host.docker.internal for Ollama and redis://redis:6379/0 for Redis. The config/production.yaml uses ollama:11434 for a containerized Ollama.

How a request flows

sequenceDiagram
    participant Client
    participant API as FastAPI
    participant Middleware as Middleware
    participant Service as QueryService
    participant Exact as Exact Cache
    participant Embed as Embedding Provider
    participant Semantic as Semantic Cache
    participant Coalescer as Request Coalescer
    participant LLM as LLM / Agent

    Client->>API: POST /v1/query
    API->>Middleware: RequestID, Metrics, APIKey
    Middleware->>Service: execute_query(request)
    Service->>Exact: get(exact_key)
    alt exact hit
        Exact-->>Service: CacheEntry
        Service-->>Client: exact_hit response
    else miss
        Service->>Embed: embed(normalized_query)
        Embed-->>Service: embedding
        Service->>Semantic: search(embedding)
        alt semantic hit
            Semantic-->>Service: (entry, score)
            Service-->>Client: semantic_hit response
        else miss
            Service->>Coalescer: execute(key, factory)
            Coalescer->>LLM: execute(key=..., factory) (leader only)
            LLM-->>Coalescer: GenerationResult
            Coalescer-->>Service: CoalescingResult(value, role) (all callers)
            Service->>Service: validate response
            Service->>Exact: set(entry)
            Service->>Semantic: store(entry)
            Service-->>Client: generated response
        end
    end

Loading

Configuration

Configuration sources (in priority order):

1. CONFIG_FILE environment variable pointing to a YAML file
2. config/local.yaml (auto-detected if present)
3. Inline defaults

Environment variables override YAML values using double-underscore nesting:

export CACHE__SIMILARITY_THRESHOLD=0.9
export OLLAMA__BASE_URL=http://my-ollama:11434
export SECURITY__ENABLED=true
export SECURITY__KEYS='["sk-abc123"]'

See config/local.yaml for all options. Key settings:

Setting	Default	Description
OLLAMA__BASE_URL	http://localhost:11434	Ollama server URL
OLLAMA__LLM_MODEL	phi4-mini	Model for text generation
OLLAMA__EMBEDDING_MODEL	nomic-embed-text	Model for embeddings
CACHE__ENABLED	true	Master cache switch
CACHE__SIMILARITY_THRESHOLD	0.85	Semantic match threshold
CACHE__TTL_SECONDS	3600	Entry time-to-live
CACHE__EVICTION_STRATEGY	ttl_lru	One of: ttl, lru, lfu, size, ttl_lru
CACHE__SEMANTIC_CACHE_ENABLED	true	Enable semantic lookup
CACHE__NAMESPACE	semantic-cache	Logical cache partition
REDIS__URL	redis://localhost:6379/0	Redis connection
COALESCING__ENABLED	true	Request coalescing switch
AGENT__MAX_CONCURRENT_LLM_REQUESTS	5	Semaphore limit
SECURITY__ENABLED	false	API-key auth switch
METRICS__ENABLED	true	Prometheus metrics

API usage

All routes are defined in src/api/routes/routes.py.

POST /v1/query

Submit a query for generation. Returns a cached response if available.

curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Explain semantic caching",
    "conversation_id": "conv-123",
    "user_id": "user-456",
    "agent": "default",
    "metadata": {"tenant": "local"},
    "cache": {"enabled": true, "bypass": false, "refresh": false}
  }'

Response:

{
  "request_id": "a1b2c3d4-...",
  "response": "Semantic caching is ...",
  "cache": {
    "status": "miss",
    "similarity_score": null,
    "matched_query": null,
    "entry_age_seconds": null,
    "estimated_time_saved_ms": null
  },
  "execution": {
    "provider": "ollama",
    "model": "phi4-mini",
    "latency_ms": 1234.5,
    "llm_latency_ms": 1200.0,
    "embedding_latency_ms": 34.5,
    "coalesced": false
  }
}

Cache status values: exact_hit, semantic_hit, miss, bypass, disabled, refresh, error.

GET /health/live

Simple liveness probe.

GET /health/ready

Readiness probe that checks LLM, embedding, and agent health.

GET /metrics

Prometheus metrics in plain-text format.

GET /v1/cache/stats

curl http://localhost:8000/v1/cache/stats

POST /v1/cache/invalidate

curl -X POST http://localhost:8000/v1/cache/invalidate \
  -H "Content-Type: application/json" \
  -d '{"namespace": "semantic-cache", "confirmation": "CONFIRM"}'

POST /v1/cache/warm

curl -X POST http://localhost:8000/v1/cache/warm \
  -H "Content-Type: application/json" \
  -d '{
    "entries": [
      {"query": "What is AI?", "response": "Artificial intelligence is ..."}
    ]
  }'

Cache behaviour

• Exact hit — SHA-256 key match on query+model+agent+namespace+tenant+user+metadata
• Semantic hit — embedding cosine similarity >= threshold after exact miss
• Miss — no cache entry; LLM is called
• Bypass — cache.bypass=true skips cache lookups and writes
• Disabled — cache.enabled=false skips all cache operations
• Refresh — cache.refresh=true forces LLM call and overwrites cache
• TTL — entries expire after cache.ttl_seconds from creation
• Tenant isolation — semantic search filters by namespace, tenant, model, agent, and optionally user
• Cache key — SHA-256 of configurable dimension set (default: query, model, agent, namespace, tenant)
• Uncached responses — responses failing validation (empty, error indicators, too short) are not written
• Write failures — logged; cache writes are fail-open (response still returned to client)

Request coalescing

• Leader/follower — the first caller for a key becomes leader and executes the LLM call; concurrent callers become followers and await the same result
• Signature — coalescer.execute(*, key: str, coroutine_factory: Callable[[], Awaitable[T]], timeout: float | None = None) -> CoalescingResult[T]
• Result — CoalescingResult has .value (the actual result) and .role ("leader" | "follower"); .coalesced is True for followers
• Local coalescing — InMemoryRequestCoalescer uses asyncio.Future sharing; process-local only
• Distributed coalescing — RedisRequestCoalescer uses Redis SET NX PX for lock-based leader election; local followers share the Python future; remote followers raise RemoteLeaderCompleted and should re-read shared cache
• Timeout — followers can specify per-call timeout via the timeout kwarg; timed-out followers raise TimeoutError without affecting the leader or other followers
• Cancellation — leader cancellation propagates CancelledError to all local followers
• Exception propagation — leader exceptions propagate to all local followers
• Lock TTL — leader lock has a configurable TTL; no lock renewal is implemented; if a leader runs past the TTL, a new leader may start
• Fail-open — if Redis is unavailable and fail_open=true (default), coalescing degrades to local-only

Running tests

# Full suite
python3 -m pytest tests/

# One test file
python3 -m pytest tests/test_coalescing.py -v

# One test function
python3 -m pytest tests/test_coalescing.py::TestInMemoryCoalescing::test_concurrent_requests_same_key -v

# With coverage
python3 -m pytest tests/ --cov=src --cov-report=term

# Verbose
python3 -m pytest tests/ -v --tb=short

# Async tests use pytest-asyncio (auto mode)

All 147 tests pass with Python 3.12. Tests use deterministic providers and in-memory stores so no external dependencies are needed.

Load tests

Located in tests/load_tests/. These stress-test coalescing, concurrency control, and timing under concurrent load:

# Run all load tests
python3 -m pytest tests/load_tests/ -v

# Run 10x to check for flakiness
for i in $(seq 1 10); do python3 -m pytest tests/load_tests/ -q; done

Test scenarios

File	Tests
test_coalescing.py	100/1000 concurrent same-key requests, mixed keys, leader cancellation, burst isolation
test_queueing.py	Concurrency limit enforcement, queue timeout rejection, all-requests-processed, pending count accuracy
test_integrated.py	Coalescing bypasses queue under load, fair multi-key queue sharing, throughput measurement, no future leaks
test_timing.py	Follower latency matches leader, fast recovery after leader failure

All load tests use InMemoryRequestCoalescer (no Redis needed). Full suite completes in ~3s.

Development in VS Code Dev Containers

The repository includes .devcontainer/devcontainer.json for a Python 3.12 container with Jupyter support:

1. Install VS Code and the Dev Containers extension
2. Open the repository and click "Reopen in Container"
3. Dependencies are installed automatically via postCreateCommand
4. The container reaches host Ollama via host.docker.internal:11434
5. Run tests: python3 -m pytest tests/
6. Run the API: uvicorn src.main:app --reload --host 0.0.0.0 --port 8000

Ports 8888 (Jupyter) and 8000 (API) are forwarded.

Observability

• Structured logs — JSON-formatted via JSONFormatter (src/utils/logging.py)
• Request IDs — X-Request-ID header or auto-generated UUID; included in logs and responses
• Prometheus metrics — available at /metrics with namespace semantic_cache
• Key metrics: cache_hits_total (by type), cache_misses_total, llm_requests_total, coalesced_total (by role), request_latency_ms, llm_latency_ms, cache_lookup_latency_ms, errors_total, lock_wait_duration_seconds, cache_entries, hit_rate
• Metrics can be disabled via METRICS__ENABLED=false

# Read metrics
curl http://localhost:8000/metrics | grep semantic_cache

# Follow logs
uvicorn src.main:app --reload 2>&1 | grep -E "(request_id|coalescing)"

Troubleshooting

Symptom	Solution
Connection refused to Ollama	Verify OLLAMA__BASE_URL. Inside Docker, use http://host.docker.internal:11434. With containerized Ollama, use http://ollama:11434.
Ollama model not found	Run ollama pull llama3.2:3b and ollama pull nomic-embed-text
Redis connection failure	Start Redis: docker compose up -d redis. The app falls back to in-memory on connection failure when fail_open=true.
Port 8000 in use	Change APP__PORT or use a different --port flag
Tests not discovered	Ensure you're in the repository root and using python3 -m pytest
API key rejected	Set SECURITY__ENABLED=false or provide a valid key via SECURITY__KEYS='["your-key"]'
Stale Redis cache	Invalidate with POST /v1/cache/invalidate -d '{"all_entries": true, "confirmation": "CONFIRM"}'

Repository structure

src/
├── api/                       # FastAPI routes, middleware, request/response models
│   ├── routes/routes.py       # RouterFactory — all endpoints
│   ├── middleware/             # RequestID, Metrics, APIKey middleware
│   ├── requests.py            # QueryRequest, InvalidateRequest, WarmRequest
│   └── responses.py           # QueryResponse, HealthResponse, CacheStatsResponse
├── bootstrap/                 # Composition root
│   ├── container.py           # ApplicationContainer.build()
│   ├── providers.py           # ProviderFactory (clock, metrics, LLM, embeddings, agent)
│   ├── caches.py              # CacheFactory (exact, semantic, coalescer)
│   └── services.py            # ServiceFactory (QueryService, CacheAdminService)
├── cache/
│   ├── protocol/              # Interfaces: ExactCacheStore, SemanticCacheStore, SimilarityStrategy, etc.
│   ├── stores/                # In-memory and Redis implementations
│   ├── coalescing/            # InMemoryRequestCoalescer, RedisRequestCoalescer
│   ├── model/                 # CacheEntry, CachePolicy
│   ├── cache_reader.py        # Exact + semantic lookup orchestration
│   ├── cache_writer.py        # Response validation + cache persistence
│   └── validation.py          # DefaultResponseValidator
├── llm/
│   ├── protocol/              # LLMProvider, EmbeddingProvider, AgentExecutor
│   └── providers/             # Ollama and deterministic implementations
├── services/
│   ├── query/                 # QueryService, QueryGenerator, QueryContext
│   └── admin/                 # CacheAdminService (stats, invalidate, warm)
├── clock/                     # SystemClock, DeterministicClock
├── infra/                     # MetricsRecorder, HealthCheckable, ResponseValidator protocols
├── metrics/recorder.py        # PrometheusMetricsRecorder, NoOpMetricsRecorder
└── utils/                     # LLMConcurrencyController, JSONFormatter, retry helpers
config/                        # local.yaml, production.yaml, test.yaml
tests/                         # 147 tests including load_tests/

Further documentation

• Architecture
• Contributing
• Extension guide

Project status and limitations

Production-ready:

• Exact and semantic caching with configurable providers and stores
• Request coalescing (local and distributed)
• Concurrency control for LLM requests
• Response validation
• Prometheus metrics and structured logging
• Cache warm and invalidate operations
• Docker Compose deployment with Redis

Experimental / limited:

• Lock renewal — not implemented. If a leader execution exceeds the Redis lock TTL, a second leader may start.
• Distributed result transport — not implemented. Remote followers receive RemoteLeaderCompleted and must re-read the cache rather than receive the result directly.
• Semantic index — in-memory search is O(n); Redis store uses sorted sets which scale better but have no priority for recent entries.This could be overcome in future when vector in-mem search is made available
• Provider ecosystem — only Ollama and deterministic providers are implemented.
• Circuit breakers — not implemented.
• Cache admission policies — not implemented.