diff --git a/.agents/common-operations.md b/.agents/common-operations.md index b05a0f8d3a..4622194759 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -13,6 +13,36 @@ Step-by-step procedures for frequent cluster tasks. - Shell-only changes: `mise exec -- shellcheck` on every touched `*.sh`. - Documentation-only changes: run `git diff --check` and verify every changed local reference exists. +### Pre-commit: `oxfmt not found` (agent box) + +Lefthook (`.lefthook.toml`) runs **bare** `oxfmt` on staged `*.yaml`/`*.yml` (excluding +`*.sops.yaml`) and `*.json*`. On the agent box `oxfmt` is a mise tool and **not on PATH**, so the +hook fails with "oxfmt not found" even though the code is fine. Verified facts: + +- `mise` itself is not on PATH in non-login shells; tool binaries live under + `/opt/data/home/.local/share/mise/installs/` (with `shims/` alongside). +- The repo pins `oxfmt = "0.67.0"` (`.mise.toml`) but the agent box has `0.66.0` installed — + the local binary lags the pin. + +Fixes, in order of preference: + +1. **Commit in a mise-active shell** so the pinned version resolves: `mise exec -- git commit …` + (after `mise trust` in the worktree — a fresh worktree's `.mise.toml` is untrusted, which also + breaks the shims: `mise ERROR Config files ... are not trusted`). +2. **Put an installed oxfmt on PATH** before committing (this is what actually unblocked the + kguardian PR): + + ```bash + # 0.66.0 is what's actually installed on the agent box (the repo's .mise.toml pins + # 0.67.0 — if the pinned version is installed, use that instead); verify with: + # ls /opt/data/home/.local/share/mise/installs/oxfmt/ + export PATH="/opt/data/home/.local/share/mise/installs/oxfmt/0.66.0/node_modules/.bin:$PATH" + git commit -m "..." + ``` + +If formatting output differs between the local 0.66.0 and the pinned 0.67.0 (CI uses the pin), +prefer option 1 so local and CI agree. + ## Ceph: `crash ls-new` hides archived crashes, not old ones `ceph crash ls-new` filters on exactly one thing, whether a crash is archived (`crash/module.py` @@ -71,14 +101,152 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ## Secrets management (SOPS) -1. Create unencrypted file first -2. Encrypt with: `sops --encrypt --in-place ` -3. Or create with: `sops .yaml` (edits encrypted) +### Where SOPS runs — local-first (2026-09-14: SSH no longer required) + +The agent box now holds an age key file at `~/.config/sops/age/keys.txt` +(mode 600, inside 700 directories, Ceph-backed — same regime as the SSH key) +containing **3 identities**: the agent box's own revocable key plus the two +master keys, copied over 2026-09-14 on the owner's explicit OK. It can therefore +**decrypt every file in the repo locally** (verified on real `kubernetes/` and +`talos/` files) — no ssh, no scp, no fish, no stale remote branch. No existing +file was re-encrypted and no recipients were changed. + +**The key is auto-discovered — no env var is required on this box** (verified: a +decrypt succeeds with `SOPS_AGE_KEY_FILE` unset; sops 3.13.3 reads +`$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` when +`XDG_CONFIG_HOME` is unset). Keep the export only if the key is ever moved off +that resolved path (then it is required, or you get "no identity matched"): + +```bash +export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" +``` + +Local `sops` binary (the mise shims are unreliable in non-login shells): +`/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + +**Full procedure → [cluster-sops skill](skills/cluster-sops/SKILL.md).** Canonical +local flows (new value / change value / `updatekeys`) and the three traps are there. + +#### Fallback: the remote management host (only if the local key is revoked) + +If the agent-box key is ever revoked, SOPS ops fall back to the management host, +where the original key still lives (`tanguille@192.168.0.181:~/cluster/age.key`). +The `k8s-management` ssh alias is **dead** in this environment: the user's passwd +home is `/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads +`/opt/data/.ssh/config` (key only, no config) and never sees the alias in +`/opt/data/home/.ssh/config`. Connect explicitly: + +```bash +ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +``` + +On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare +`sops --version` prints nothing. Prefix with `mise exec -- sops …` (or invoke the +shim under a trusted `cwd` with a trusted `.mise.toml`). The remote shell is +**fish** — pipe complex commands via `ssh … 'python3 -' < local.py`, never +multi-line heredocs. + +### Recipients differ per subtree (read `.sops.yaml`) + +Do not assume one key. `.sops.yaml` maps path → age recipient: + +| Path | Key (first 10 chars) | +|------|----------------------| +| `talos/**/*.sops.yaml` | `age12gul5m0…` | +| `(bootstrap\|kubernetes)/**/*.sops.yaml` | `age1pq1f69…` (post-quantum) | + +A CloudNativePG role secret under `kubernetes/` uses the `age1pq1…` key; a `talos/` file uses +`age12gul5m0…`. Encrypting with the wrong recipient (or hand-adding a `sops:` block) breaks +decrypt for the real owner. + +### The three SOPS traps (each cost real time — avoid them) + +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one into the other — + `encrypted_regex: ^(data|stringData)$` only encrypts whichever section already exists, and + this repo's `kubernetes/` and `talos/` secrets use **`stringData`** (there is no `data`), + so a blind `d["data"][key] = …` raises `KeyError` on them. Rebuild via a **dict** in + Python (load the *decrypted* YAML, write the value into whichever section the file already + uses, dump back, re-encrypt) rather than text-splicing. +2. **Stale `sops:` footer after a textual edit.** If you text-edit an encrypted file (or a + decrypt→edit), the old `sops:` metadata block remains and `sops --encrypt` fails on it. Always + **decrypt first**, edit the clean file, then encrypt — never edit the ciphertext in place. +3. **Missing `--input-type yaml` makes SOPS guess JSON and fail.** Always pass + `--input-type yaml --output-type yaml` on YAML files; do not rely on extension sniffing. + +### Canonical flows (remote = `mise exec -- sops …`, in a trusted cwd with `.sops.yaml`) + +1. **New secret value** — stage the file *at its final, rule-matching path* with + `data: {key: base64value}` or `stringData: {key: plain}`, then: + + ```bash + mise exec -- sops encrypt --in-place --input-type yaml --output-type yaml file.sops.yaml + # sanity: only data/stringData values are ENC[… ciphertext (match ENC[ — the + # ciphertext starts ENC[AES256_GCM, so a bare 32-char-hex-after-ENC grep matches nothing) + ``` + + `sops encrypt` is **not** a no-op on already-encrypted content — it fails + ("top-level entry called 'sops'", rc 203). Never re-run it over ciphertext. + +2. **Change an existing value** — `.sops.yaml` rules are path-based, so the + re-encrypt must target the rule-matching in-repo path (a `/tmp` temp matches no + creation rule — "no matching creation rules found", even with `--age`). Decrypt + to a unique mode-600 temp, edit via dict, copy back over the file, encrypt + in place: + + ```bash + T="$(mktemp /tmp/plain.XXXXXX)"; chmod 600 "$T" + trap 'rm -f "$T"' EXIT + mise exec -- sops decrypt --input-type yaml --output-type yaml file.sops.yaml > "$T" + # edit $T via a Python dict (write into whichever of data/stringData the file uses) + cp "$T" file.sops.yaml + mise exec -- sops encrypt --in-place --input-type yaml --output-type yaml file.sops.yaml + rm -f "$T" # never leave a decrypted secret on disk; trap also covers the failure path + ``` + +3. Verify the recipient line in the resulting `sops:` block matches the subtree above. + +### Standing rules + +- Never commit plaintext secrets or the age key. Use placeholders so the user adds values manually. +- **Ask before decrypting/editing SOPS** (AGENTS.md) and before `rm`-ing a decrypted temp. +- Post-quantum age (`age1pq1…`) is supported. Both halves of a key must be in `age.key`: a key + that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +- The remote `~/cluster` checkout may be on a **stale feature branch** (it was 18 commits behind + `origin/main` during the kguardian work). `git fetch` there before trusting its state; SOPS + only needs the `.sops.yaml` + `age.key`, not a fresh tree, but re-encrypting against a stale + tree can carry in stale content — prefer editing the specific file. + +## PR shepherding (re-shepherd pass) -Never commit plaintext secrets or the age key. Use placeholders so I can add the secrets manually. +Standing order: iterate an owner PR until CI + automated review are clean. One pass: -Post-quantum age (age1pq1) is supported. Both halves of the key must be present in `age.key`: a -key that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +1. **`git fetch origin` first** (hard rule — this worktree sits on a feature branch and + is always stale; never answer current-state questions from it). +2. Read the PR via ToolHive `github_pull_request_read` (`method: get`): state (`open`/ + `draft`), head SHA, base SHA, `mergeable`, `mergeable_state`, `merge_state_status`, + `commits` count, diffstat. +3. **Base moved?** If `base.sha != origin/main` head: compare overlap — + `git diff --name-only ...origin/main` vs the PR's file list. Overlap files + are rebase candidates; dry-run with `git merge-tree origin/main` + (or `git merge-tree --write-tree origin/main`) to confirm clean. +4. **Rebase = Gate A, server-side only**: `github_update_pull_request_branch` with + `expectedHeadSha` = the current head SHA (guards against concurrent pushes). **No + local `git push`** — the agent box has no GitHub token by default; branch updates go + through ToolHive `push_files` (full-file contents, no delete) or the server-side + rebase. +5. `mergeable_state: "unknown"` right after a base move usually just means GitHub is + recalculating — confirm with the rebase rather than looping on polls. +6. Re-poll `get_check_runs` until the new head is `success`. Normal draft-green shape: + ~12 success + 2 skipped (CodeRabbit/DeepSource skip on drafts). +7. Read comments for *new* feedback since the last pass; address or answer it. +8. **Budget: 3 fix cycles** per issue class, then escalate to the owner with evidence + (log excerpts, which checks failed, and the classification: flake vs diff vs + baseline) — don't silently keep retrying. +9. **Hard no-s without explicit per-instance owner approval:** merging the PR, + pushing to `main`, force-pushing any branch, `cluster-apply`, decrypting secrets + into a PR description/log/chat. +10. Status wording: report `mergeable_state`/`merge_state_status` verbatim; do not + imply "clean" while either is `unknown`/`behind`. ## Debugging @@ -90,4 +258,4 @@ Use [backup-restore](skills/backup-restore/SKILL.md) skill for kopiur Kopia oper ## Other skills -See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, and prometheus-cluster-health. +See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, cluster-sops, and prometheus-cluster-health. diff --git a/.agents/learned-preferences.md b/.agents/learned-preferences.md index 34bee84ef4..24c727700f 100644 --- a/.agents/learned-preferences.md +++ b/.agents/learned-preferences.md @@ -1,6 +1,6 @@ # Learned User Preferences -**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools. +**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools, PR shepherd/rebase, SOPS ask-first. Maintained from session feedback. Prefer git revert, don't undo user changes, only adjust resources where already set. @@ -17,6 +17,19 @@ If the available tools include `find_tool` and `call_tool` (ToolHive unified gat 3. Call `call_tool` to execute it. 4. Interpret the result and respond naturally—never return raw JSON or raw tool output to the user. +**Exact `call_tool` shape (two-level, the `github_` prefix matters):** + +```json +{"name": "mcp__toolhive__call_tool", + "arguments": {"tool_name": "github_pull_request_read", + "parameters": {"owner": "Tanguille", "repo": "cluster", + "pullNumber": 4998, "method": "get"}}} +``` + +- The inner tool name carries the **`github_` prefix**; the PR number is **`pullNumber`, not `number`** (both are easy to get wrong). +- **Large payloads** (multi-MB `push_files` bodies) can make the *local* SDK throw `SSE stream ended` / `TaskGroup` — while the **server-side operation still succeeded**. On such a failure: do NOT blindly retry (double-push / orphan-commit risk); first read the PR state back (`github_pull_request_read`) to see whether it landed. +- ~10 rapid consecutive calls can trip a transient "unreachable" for ~60s; a short pause + a single retry usually succeeds. + ### Other tools and behavior - For any other tools, call them directly when relevant, without unnecessary preamble. diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md new file mode 100644 index 0000000000..01a2c0c245 --- /dev/null +++ b/.agents/skills/cluster-sops/SKILL.md @@ -0,0 +1,207 @@ +--- +name: cluster-sops +description: >- + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent + box, where its own age key now lives (one local command, no SSH), with the remote + management host as fallback. + + user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients + user: "change the DB password" → decrypt → edit in dict form → re-encrypt + user: "re-encrypt after changing recipients" → sops updatekeys + + Use proactively whenever a *.sops.yaml value must be created or changed. Ask the + owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. +compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). +--- + +# Cluster SOPS (local-first secrets workflow) + +## Where it runs — and why (2026-09-14: SSH no longer required) + +- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` + (mode 600, inside 700 directories) with **3 identities**: the agent-box's own + revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) + copied from the remote 2026-09-14. It therefore decrypts **every** file in the + repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. + No existing file was re-encrypted and no recipients were changed. +- **The key is auto-discovered — no env var required on this box (verified: a + decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age + key from `sops/age/keys.txt` under the user config directory — + `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` + when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is + ever moved *off* that resolved path (then it is required, or you get + "no identity matched"): + + ```bash + export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" + ``` + +- **Local sops binary** (aqua install; the mise shims are unreliable in non-login + shells, and bare `sops` may fail before sops even runs — use the full path in + every canonical example below): + `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + `age-keygen` (for key ops) is at + `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the + identity file as a *positional* argument (no `-r` flag), and the `age` binary + in the same directory is the encrypt/decrypt CLI (no `-g` flag). + +### Fallback: the remote management host + +Only needed if the master keys are ever removed from `keys.txt` (to shrink the +agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS +decrypt falls back to the management host, where the original key lives +(`tanguille@192.168.0.181:~/cluster/age.key`): + +- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` + but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and + the alias never resolves. +- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use + `mise exec -- sops …`. +- Remote `~/cluster` may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch + before trusting file state there. +- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, + never multi-line heredocs. + +## Recipients (from `.sops.yaml` — read it, don't trust memory) + +| path_regex | recipient | +|---|---| +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | + +`sops` picks the rule by **path** — run it against the file's real in-repo path +(or with the file inside the matching subtree). A path outside the rules (e.g. +`/tmp/...`) matches **no** creation rule and fails with "no matching creation +rules found", even with an explicit `--age` — the `.sops.yaml` config wins over +CLI flags (verified on 3.13.3). + +**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds +**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two +master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing +file was re-encrypted or had its recipients changed ("don't touch existing +encryption" — honored). Consequences: + +- **Decrypt/edit any existing file: works locally**, no SSH (verified on real + `kubernetes/` and `talos/` files). +- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) + — the agent-box key is a *redundant* decryptor, not a recipient, unless you + later decide to add it to `.sops.yaml` + `updatekeys`. +- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If + the agent box is compromised, *rotate the master* (the revocable-agent-key + model is no longer the only blast-radius control). If you'd rather not keep + that exposure, the master lines can be removed from `keys.txt` (a backup of + the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would + then fall back to SSH. + +**PQ note (learned the hard way):** "post-quantum" is a key *format* +(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. +A brand-new PQ key cannot decrypt files encrypted to an older PQ key. + +## The three traps (each already cost time — avoid them) + +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field + into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever + section already exists — and this repo's `kubernetes/` and `talos/` secrets use + **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises + `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted + YAML, write the value into whichever section the file already uses, dump back — + instead of hand-editing text. +2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt + that left the metadata block) makes the next `sops encrypt` **fail** on the + existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it + is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata + stripped) before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess + JSON and fail on YAML secrets ("error: no matching creation rules found" or a + JSON-guess failure). **Fix:** pass both flags explicitly for `sops encrypt` + and `sops decrypt`. Use the supported flags documented for `sops updatekeys` + (it rejects `--output-type`). + +## Canonical flows (local-first, one command each) + +All examples use the full sops path rather than bare `sops` (which can fail +before sops runs in a non-login shell — the mise shims are unreliable outside a +login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and +— only if the key were ever moved off its auto-discovered path — the +`SOPS_AGE_KEY_FILE` override above. + +```bash +SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops +``` + +### New secret value + +```bash +F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule +# stage the file with data: {key: base64value} or stringData: {key: plain} +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… +# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep +# after ENC matches nothing on real files — plus the sops footer, and no plaintext +grep -n "ENC\[" "$F" | head +``` + +### Change an existing value (dict-based, no text surgery) + +The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp +matches no creation rule — see Recipients), so the edited plaintext goes back +over the file, then `encrypt --in-place`: + +```bash +set -euo pipefail # fail fast: a failed decrypt/edit must stop before `cp` overwrites the tracked file +F=kubernetes/apps///.sops.yaml +T="$(mktemp /tmp/secret.XXXXXX)"; chmod 600 "$T" # unique mode-600 temp (no pre-create/clobber) +trap 'rm -f "$T"' EXIT # cleaned up on success AND failure +"$SOPS" decrypt --input-type yaml --output-type yaml "$F" > "$T" +python3 - "$T" <<'PY' +import sys, yaml +p = sys.argv[1] +d = yaml.safe_load(open(p)) +# write the new value into whichever section the file ALREADY uses (data: is +# base64; stringData: is plain) — select the existing field, do not assume data: +if "data" in d: + d["data"]["key"] = "newbase64value" +else: + d["stringData"]["key"] = "newplainvalue" +yaml.safe_dump(d, open(p, "w"), sort_keys=False) +PY +cp "$T" "$F" # edited plaintext back over the rule-matching path +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f "$T" # trap also covers the failure path +``` + +### Re-encrypt after a recipient change + +```bash +# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document +git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ + "$SOPS" updatekeys --yes --input-type yaml +``` + +(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in +place — it has no `--in-place`/`--output-type` flags; passing them errors.) + +### Add a CNPG managed role / DB secret (CNPG subtree) + +Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its +`managed.roles[].passwordSecret` points at it), so it encrypts under the +`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the +password in the `.sops.yaml`. + +## Standing rules + +- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before + `rm`-ing a decrypted temp file. +- **Never echo decrypted values** into chat, PR descriptions, logs, or commit + messages. Use `[REDACTED]`. +- The **master age key now lives on the agent box too** (copied 2026-09-14 by + explicit owner decision; `keys.txt` = 3 identities). It is still not committed + and not echoed — only *public* keys ever go into `.sops.yaml`. If you later + want to shrink the blast radius, remove the master lines from `keys.txt` + (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. +- After encryption: `git diff` should show **only ciphertext changes** (no + plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit in the agent-box worktree and follow the normal commit/PR flow (PR + shepherding rules in `common-operations.md`). The remote is a fallback for + SOPS + the only push path if ToolHive is down. diff --git a/AGENTS.md b/AGENTS.md index 700405790c..23098b23a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ GitOps-based Kubernetes cluster on Talos Linux with FluxCD reconciliation. Make - [Learned preferences](.agents/learned-preferences.md): tool selection, ToolHive workflow, reversions, resources, and confidence - [Learned workspace](.agents/learned-workspace.md): cluster-specific Kubernetes, ToolHive, database, storage, Talos, and media facts - [Common operations](.agents/common-operations.md): validation, app operations, SOPS, debugging, and backup/restore -- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, debug-cluster, git-worktree-isolation, handoff, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory +- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, cluster-sops, debug-cluster, git-worktree-isolation, handoff, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory - [Useful commands](docs/useful_commands.md): flux/just/talos/sops command reference and app-specific runbooks - [Archived migrations](docs/archived-migrations.md): completed ZFS→Ceph, Radarr→Postgres and OPNsense BGP procedures, with git recovery pointers - [LLM hosting](docs/llm-hosting/): sglang/vLLM tuning constraints and benchmark history for `kubernetes/apps/ai/`