[production] gateway readiness flaps while scheduler full rebuild times out in multi-Pod deployment

[ranxi2001]

Summary

The v2.9.7 gateway Pod intermittently loses readiness and repeatedly times out during scheduler snapshot rebuilds when it shares PostgreSQL/Redis with the primary instance.

Deployment

• Repository: ranxi2001/sub2api
• Release: v2.9.7, commit bc83ff9c367883e5b7d0140e6bb42e2e7cc5239c
• Primary: systemd on freqtrade
• Gateway: k3s tosky-canary/tosky-nerd, runtime.role=gateway, one Pod
• Shared PostgreSQL and Redis are reached through the existing restricted SSH link.
• Gateway pool settings follow the repository multi Pod guidance: SQL max 16 / idle 4, Redis pool 32 / idle 4.

Observed behavior

On 2026-10-03 Asia/Shanghai:

• Gateway logs repeatedly reported:

  • [Scheduler] full rebuild failed: context deadline exceeded
  • [Scheduler] rebuild failed: bucket=0:antigravity:single ... context deadline exceeded
  • outbox lag warnings of 6–11 seconds
  • content_moderation.runtime_snapshot_refresh_failed ... context deadline exceeded

• The gateway /readyz endpoint alternated between {"status":"ready"} and {"status":"not_ready"} during repeated read-only probes. The Pod did not restart.

• The primary /readyz endpoint remained stable.

• The current /readyz implementation uses a one second shared budget for PostgreSQL and Redis PingContext checks, so it does not expose which dependency failed.

• A post-observation cache check still found all 39 sampled configured account cost multipliers present and matching in the database, sched:acc, and sched:meta. This confirms the issue is intermittent; it does not prove full rebuild completion.

Routing evidence

The host Nginx configuration routes requests to the primary by default and only selects the gateway when X-Sub2API-Node: nerd is explicitly supplied. Health probes confirmed both branches. No model requests reached the gateway during the observation window, so the readiness problem did not yet affect default production traffic.

Related upstream reports

• Wei-Shaw/sub2api#4772 reports a multi Pod scheduler state problem where one Pod returns no available accounts while another Pod works with the same shared PostgreSQL/Redis.
• Wei-Shaw/sub2api#1608 documents scheduler outbox watermark timeouts causing repeated event processing, delayed updates, and repeated full rebuilds.

Questions

1. Should full rebuild/outbox processing be isolated from the gateway readiness dependency check, or should the shared dependency timeout be configurable for remote PostgreSQL/Redis links?
2. Should readiness failures log a dependency-specific reason (without returning it to clients) so an intermittent 503 can be distinguished between PostgreSQL, Redis, and draining?
3. Is there a current recommended multi Pod configuration or patch for the outbox/full-rebuild timeout path that should be applied to the owner fork?

No production configuration or data was changed while collecting this evidence.

[ranxi2001]

Additional reference found after filing:

• Upstream PR #4643 is an open scheduler optimization PR with batch Redis load reads, metadata snapshot reuse, batched DB authority checks, incremental bucket maintenance, and scheduler observability. Its CI is green, but it targets upstream main and has not been backported or validated against owner production.
• The repository's current multi Pod deployment README already recommends one full instance plus gateway replicas, shared PostgreSQL/Redis, a ClusterIP gateway Service, and explicit ingress routing. It also warns that /readyz uses a shared one second PostgreSQL/Redis budget.

This makes Wei-Shaw#4643 a useful upstream candidate to review, not an applied fix. No source PR was opened from it because the patch is broad (26 files / 3,000+ lines) and targets a different base; a direct cherry-pick would be unsafe without a production-base backport and focused validation.

[ranxi2001]

已提交 #282，补齐 server.readiness_timeout_seconds / SERVER_READINESS_TIMEOUT_SECONDS：默认仍为 1 秒，可显式配置 1–60 秒，依赖错误继续返回 503。本地验证覆盖真实 provider 与 HTTP wrapper、sqlmock 慢 Ping、miniredis、取消/并发探针，三个相关包 race 通过，后端构建通过。它只提供探测预算配置，scheduler full rebuild/outbox 超时仍未解决，因此本 Issue 保持开放。

更正前面的上游类比：Wei-Shaw#1608 对应的 修复 #1702 已合并；当前 fork 的 pollOutbox 已使用独立 5 秒 context 写 watermark，不能将这次现象归因于旧的 context 复用问题。Wei-Shaw#4772 的进程内账号封锁问题也是相关背景，尚无证据证明与本次根因相同。

GitHub 另外找到两份 MIT 许可的部署代码，可作为结构参考：

• wchen99998/robust2api Helm chart（固定 6c02dac5）：生产 values 配置 2 个 gateway、HPA 2–10、PDB 和拓扑分布，并拆分 control/worker/billing。适合参考职责拆分和部署资源组织；这是另一套应用架构，不能直接用它的 chart 替换本 fork 的镜像运行契约。
• ben-wangz/k8s-at-home Sub2API chart（固定 daf292e4）：提供 Deployment、Service、Ingress、Secret/PVC，默认 1 副本、镜像 0.2.7；Deployment 多副本仍引用同一 data PVC，默认 RWO。可参考基础 Helm 封装，不是已验证的本 fork 多 Pod 方案。

目前最直接兼容的基线仍是本 fork 的 gateway StatefulSet 方案：一个 full、多个 gateway、每 Pod 独立 PVC、共享 PostgreSQL/Redis 和密钥。上述外部 chart 本次只做源码审阅，未实际部署或跑压测。

Agent 声明：Codex（GPT-6，具体模型 ID 和推理强度当前会话未提供）；参与诊断、代码、测试、检索和本文，未使用子 Agent。