Add real Redis Sentinel, Cluster, authentication, and TLS tests

[hemju]

Implemented locally; review and merge pending

This issue was worth implementing before 1.0. The earlier Cluster/TLS startup and round-trip tests did not prove continuity after a leader failure. Local, uncommitted work tracked in #135 now adds real promotion, topology refresh, live slot migration, retained queued/in-flight work, and deliberate Sentinel TILT recovery. Keep this issue open until review and merge.

Acceptance criteria

• Real Redis Cluster store contract fixture exists.
• Authenticated TLS Cluster/Sentinel round-trip and invalid-credential tests exist.
• Exercise actual Sentinel primary failure, replica promotion, and client reconnection with queued and in-flight durable jobs.
• Exercise real Cluster primary failure, topology refresh, and slot migration/resharding where supported.
• Verify drain and concurrency holds after failover; document acknowledged-write/durability limitations.
• Run the shared store contract on the declared minimum Redis version and applicable topologies.
• Retain bounded automated failover coverage and document longer datastore soak qualification.

Fresh validation

The final full productionCheck -PdependencyScanRequired=true --max-workers=2 --no-build-cache passed in 6m 19s. All 405 Redis cases passed with zero failures/errors/skips. RedisFailoverTest exercises Redis 7.4 and 8.6 using independent processes: three Sentinels with a primary/replica, or three Cluster primaries plus three replicas. Each failover workload uses two existing ProcessingNodes and verifies all 203 jobs finish without exclusive overlap. The minimum-version shared contract also runs through Sentinel, and authentication/TLS/mTLS/invalid-certificate coverage remains enabled.

Primary failure exposed a runtime bug: Cluster-wide SCRIPT LOAD could keep failing on the dead former primary. Script hashes are now computed locally and NOSCRIPT recovery uses keyed EVAL against the current owner. A permanent script-cache regression and the promotion cases cover it.

The repeat gate also exposed an insufficient test deadline after a host timing interruption put Sentinel into TILT. Deliberate process-suspension tests now verify TILT recovery on both Redis versions. A bounded 90-second recovery allowance includes its 30 stable-second protection interval and election retries; actual promotion and drain remain mandatory. The fixture also uses a five-second down-after setting: its earlier one-second setting excluded the only replica after TILT delayed the initial down observation, according to Sentinel's replica-eligibility rule. TILT is not disabled. See Redis's TILT contract.

Reports/process logs live under threadmill-store-redis/build/redis-topology/; final and failed-run evidence is preserved under .local-reference/threadmill-1.0-audit-2026-09-09/validation/. Candidate source-manifest SHA-256: c156401965cf6ea1ac2f23f14dc08ce1d6c271e57ccd743f15ed628c50d32770.

Longer topology qualification plan (not executed)

After the separate standalone Redis baseline and 12h mixed + 12h retention phases in #135, provision real failure domains and run eight hours of retention churn per Sentinel/Cluster topology. Use one producer, three processing nodes, eight workers/node, initially 50 jobs/s, ten-minute node churn, live fail-fast invariants and independent output paths.

./gradlew :threadmill-soak:soakRedis \
  -Pscenario=retention-churn -Pduration=8h -PjobsPerSecond=50 \
  -Pproducers=1 -Pnodes=3 -PworkerCount=8 -PnodeChurn=10m \
  -PprogressInterval=30s -PfailFast=true \
  -PredisTopology=sentinel \
  '-PredisUrl=redis-sentinel://sentinel1:26379,sentinel2:26379,sentinel3:26379/0#soak-primary' \
  -PrunId=sentinel-candidate -PoutputDir=.local-reference/qualification/sentinel-candidate

Repeat separately for Cluster, replacing topology with -PredisTopology=cluster, using '-PredisUrl=redis://redis1:6379,redis://redis2:6379,redis://redis3:6379', and a unique run ID/output path. Advertised node endpoints must be reachable; Cluster seeds must use consistent credentials/TLS and database zero. Validate the deployment's actual certificates and ACLs. Harness startup resets the dedicated Threadmill namespace.

At hours 2 and 6, perform controlled primary handovers. At hour 4, migrate the namespace slot in Cluster; in Sentinel, pause the current primary for 20 seconds and observe discovery/recovery. At hour 7, perform a separate abrupt-primary-failure experiment and quantify acknowledged-write loss and recovery. Retain fault timelines, Sentinel events (including TILT), independent-client latency, raw invariants, state/index/counter reconciliation, memory/RSS/AOF growth, throughput and operation tails. Apply the baseline-relative thresholds and plateau criteria in #135 and local docs/soak-plan-1.0.md.

Limits: bounded fixtures co-locate processes in one container and use a fixture-only replication barrier for seeded work. They do not certify network partitions, independent failure domains or zero acknowledged-write loss under asynchronous replication. The common {threadmill} hash tag uses one Cluster slot, so it does not horizontally shard a namespace's load. Long-running deployment qualification remains a release prerequisite; this checkbox records that its plan is documented, not that it has run.