fix(etcd-defrag): defragment daily so the fragmentation alert stays resolved

[Tanguille]

What

etcdDatabaseHighFragmentationRatio keeps re-firing despite the etcd-defrag CronJob running. One-line fix: defragment daily instead of weekly.

Why the alert keeps coming back

• Alert rule (bundled VictoriaMetrics default rules): etcd_mvcc_db_total_size_in_use_bytes / etcd_mvcc_db_total_size_bytes < 0.5 for 10m
• Post-Talos-1.14 (rolled 09-03, etcd 3.7.0 btree/bbolt engine) the write churn is much higher: the db file grew from ~250 MB right after the 09-13 defrag back to ~533 MB by 09-16 (~90 MB/day of free-space growth, in-use flat at ~238 MB)
• So ~3 days after each weekly pass the ratio drops back below 0.5 and the alert fires again — live state on 09-16: all 3 members at 44.6–44.9 % in-use (alert firing)
• --auto-disalarm only disarms etcd's own NOSPACE alarm — it does nothing for this Prometheus alert. The defrag pass is the only auto-resolution path, so it must run often enough that the ratio doesn't sink below 0.5 in between

Change

kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml only:

• schedule: "0 3 * * 0" → "0 3 * * *" (daily 03:00 CEST)
• Comment updated with the rationale/numbers above

No other flags change: --cluster --move-leader (3 members, ~95 s total), --defrag-rule=dbQuotaUsage > 0.5 || dbSizeFree > 50Mi still triggers every run (292 MiB free), image pinned at v0.45.0 (supports etcd 3.4–3.7).

Expected effect

Daily pass → ratio back to ~100 % each morning → alert auto-resolves within 10 min and stays resolved until the next day's churn accumulates (~90 MB/day ≈ 2 days of headroom before 0.5 is reached again; with daily defrag it never gets there).

Follow-up (not in this PR)

The 11× write churn itself is worth investigating separately (what started writing to etcd 11× more after the 1.14.0 rollout — new reconcile loops / chatty operator is the usual suspect). That's the root cause; this PR stops the alert noise in the meantime.

Rollback

Revert this commit — Flux re-renders the weekly schedule within the 30 m interval.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[tanguille-cluster]

@@ spec.schedule @@
# batch/v1/CronJob/kube-system/etcd-defrag
! ± value change
- 0 3 * * 0
+ 0 3 * * *

[tanguille-cluster]

@@ spec.values.controllers.etcd-defrag.cronjob.schedule @@
# helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/etcd-defrag
! ± value change
- 0 3 * * 0
+ 0 3 * * *

[github-actions]

AI Automated Review

Analysis engine: omniroute@http://litellm.ai.svc.cluster.local/v1 (openai)

Approve. Daily defrag schedule (0 3 * * *) prevents etcdDatabaseHighFragmentationRatio alert re-firing under Talos 1.14 write churn.

• kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml: cron schedule syntax correct for daily 03:00 run in ${TIMEZONE}.
• Must-check validate manifest against target cluster version: verified against Kubernetes 1.37 / Talos 1.14; 5-field cron expression standard and valid.
• Must-check check for resource quota / limit changes: verified; container cpu/mem limits (100m/128Mi) and --etcd-storage-quota-bytes unchanged.

Tool Harness Findings

Inspected kubernetes/apps/kube-system/etcd-defrag/app/helmrelease.yaml and .mise.toml. Verified container resource allocations, flags (--cluster, --move-leader), and schema compatibility remain intact.

[Tanguille]

Manual run executed (2026-09-16 ~23:50 CEST): talosctl etcd defrag per node on 192.168.0.11/12/13, each exit 0. Post-run in-use ratio: .11=0.9997, .12=0.9999, .13=0.9733 (all were ~0.45) → etcdDatabaseHighFragmentationRatio auto-resolves within 10 min.

Reconsidering "daily" — the real cost and why it's still the right cadence:
The CronJob isn't "defrag every day." It's --defrag-rule=dbQuotaUsage > 0.5 || dbSizeFree > 50Mi: check daily, defrag only if fragmentation is present. At the measured 1.14 churn (~95 MB/day of free-space growth, in-use flat at ~238 MB from auto-compaction) the 50 Mi rule is true every night, so "daily" currently means one real ~95 s / 100 m-CPU defrag + --move-leader per night at 03:00, not a seconds-long no-op. That's the honest downside: ~95 s of CPU per night on a control-plane node until the 11× write churn subsides. If churn normalizes, runs become no-ops again and the schedule costs nothing.

Why not a slower cadence — the alert fires when in-use/total < 0.5, i.e. when free space exceeds in-use (≈238 MB, since total = in-use + free > 2·in-use):

• free grows ~95 MB/day from ~12 MB right after a defrag → it crosses 238 MB at ~2.4 days after each pass.
• daily: free at each run ≈ 95 MB (well under 238) → alert never re-fires ✓
• every 2 days: free passes 238 MB ~2.4 d after each run, inside the next window → alert on ~1.6 days of every 2-day cycle
• weekly (current): on ~4.6 days of every week — the exact state we just cleaned up

So daily is the minimum cadence that keeps the alert quiet at this churn, and its cost is bounded (one 95 s / 100 m-CPU run per night, off-peak, 3-node cluster, 250 MB DB). The real concern isn't defrag cost — it's that we're masking the 11× post-Talos-1.14 etcd write churn (the root cause); if that churn grows further the 50 Mi rule just triggers more often. Once the churn source is found and fixed, this can drop back to weekly and the alert stops re-firing on its own.