Feature: Circuit breakers for the Redis cache and SPARQL endpoint (Ship 2)

[mdorf]

What this does

Adds circuit breakers around the two dependencies goo hits on the read path, the Redis cache and the SPARQL endpoint, so that a slow or dead dependency fails fast instead of making every worker wait out its timeout. Also tightens the timeouts a breaker depends on and replaces a fixed-interval retry with bounded backoff.

Off by default. Everything here is gated on OP_SPARQL_CIRCUIT_BREAKER; unset, protect_read and protect_best_effort are plain pass-throughs and behavior is identical to today.

Why

The cache is not an optimization, it is load-bearing: a single class-tree request is roughly 1 SPARQL query plus ~2000 Redis reads. That makes a Redis outage a first-class failure mode, and it makes the obvious fix wrong, since falling through to the store turns one request into ~2000 queries and takes the triple store down for every tenant. So an open breaker sheds load rather than passing it along. Details in docs/sparql-defork-review.md §2A and decisions D1/D2/D4.

This is an inherited risk, not a new one: the fork let Redis errors propagate out of #query too. Shipping it separately from the de-fork was deliberate (D13).

What's in it

• Goo::SPARQL::Resilience (new): Stoplight-backed breakers, per-process and in-memory on purpose, since a Redis-backed store would put breaker coordination inside a dependency the breaker exists to protect (D14). Only infra errors count, so a cache miss or a malformed query never trips anything.
• Two protection modes. Cache-dependent reads fail fast with CircuitOpenError; best-effort operations (cache writes, invalidation, log writes) degrade to a no-op and can never fail the query they accompany.
• Separate circuits for the cache Redis, the query-log Redis (which lives on its own instance per D6a), and each SPARQL endpoint, so one dependency's outage cannot shed load for another. There's a test for that isolation.
• Invalidation retry moves from a fixed sleep(5) x3, up to a 15 second stall on a request thread, to capped exponential backoff with full jitter, sub-second worst case (D4).
• Timeouts are now env-tunable (GOO_REDIS_TIMEOUT, default dropped from 300s to 5s; GOO_SPARQL_READ_TIMEOUT, unchanged at 10s). A breaker cannot trip faster than the call's own timeout.
• Failure-injection tests (T-6): 15 tests covering breaker states, both protection modes, circuit isolation, and bounded backoff.

Two defects found while reviewing this branch are fixed here (f26ab50):

• invalidate_with_backoff swallowed tracked Redis errors, so Stoplight saw every invalidation as a success and the Redis circuit could never open from that path. During a write-heavy outage with no concurrent reads to trip it, every write kept paying the full retry ladder against a Redis already known dead. It now re-raises after logging, and the caller still absorbs it, so the write is unaffected.
• An invalidation dropped by an already-open breaker was completely silent, neither logged nor reported to the metric hook. That is the case where invalidations are being dropped wholesale, and graphs written during the outage keep serving stale entries after Redis recovers until their next successful write. It now reports through the same hook, carrying a CircuitOpenError so a consumer can tell a skip from a failure.

Before enabling it in production

Merging is safe with the flag off, but two things are needed before turning it on, and they are not in this PR:

1. ontologies_api must map Goo::SPARQL::Resilience::CircuitOpenError to a 503. There is no handler today, so an open breaker would surface as a 500 rather than shedding load cleanly.
2. An on_state_change hook should be wired to alerting. A load-bearing dependency tripping is page-worthy; goo only writes a transition line to stderr.

Recorded as D15 in the review doc, which also notes that D13's ordering reversed: this depends on the query logger (Ship 3, #195), so it merges second.

Testing

• CI green on all four backends (4store, AllegroGraph, Virtuoso, GraphDB).
• Full local suite against 4store: 268 tests, 3345 assertions, 0 failures.
• The two fixes above are covered by new assertions in test/test_resilience.rb, verified to fail before the fix.

Ship 2 of the de-fork. Work by @alexskr; review, fixes, and shipping by @mdorf.