From 1b4010ff373862717c54419b237c0dec41a88391 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 16:43:44 +0200 Subject: [PATCH 01/11] docs(agents): capture SOPS/remote, PR-shepherd, ToolHive, pre-commit lessons New cluster-sops skill (remote-only SOPS workflow, recipient map, the three traps, canonical scripted flows) + supporting edits: - common-operations.md: where SOPS actually runs (agent box has no age key, k8s-management ssh alias is dead here, remote sops is a mise shim), per-subtree recipient table, stringData->data / stale-footer / --input-type traps, canonical remote flow, and a PR-shepherd re-shepherd pass (fetch first, server-side-only rebase = Gate A, 3-cycle budget, hard no-s) - learned-preferences.md: exact ToolHive call_tool shape (github_ prefix, pullNumber, large-payload SSE/TaskGroup pitfall, rapid-call backoff) - AGENTS.md + catalog listings: register cluster-sops All facts verified on the live system 2026-09-14. --- .agents/common-operations.md | 138 +++++++++++++++++++++++++-- .agents/learned-preferences.md | 15 ++- .agents/skills/cluster-sops/SKILL.md | 127 ++++++++++++++++++++++++ AGENTS.md | 2 +- 4 files changed, 273 insertions(+), 9 deletions(-) create mode 100644 .agents/skills/cluster-sops/SKILL.md diff --git a/.agents/common-operations.md b/.agents/common-operations.md index b05a0f8d3a..66b4d55e59 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -13,6 +13,36 @@ Step-by-step procedures for frequent cluster tasks. - Shell-only changes: `mise exec -- shellcheck` on every touched `*.sh`. - Documentation-only changes: run `git diff --check` and verify every changed local reference exists. +### Pre-commit: `oxfmt not found` (agent box) + +Lefthook (`.lefthook.toml`) runs **bare** `oxfmt` on staged `*.yaml`/`*.yml` (excluding +`*.sops.yaml`) and `*.json*`. On the agent box `oxfmt` is a mise tool and **not on PATH**, so the +hook fails with "oxfmt not found" even though the code is fine. Verified facts: + +- `mise` itself is not on PATH in non-login shells; tool binaries live under + `/opt/data/home/.local/share/mise/installs/` (with `shims/` alongside). +- The repo pins `oxfmt = "0.67.0"` (`.mise.toml`) but the agent box has `0.66.0` installed — + the local binary lags the pin. + +Fixes, in order of preference: + +1. **Commit in a mise-active shell** so the pinned version resolves: `mise exec -- git commit …` + (after `mise trust` in the worktree — a fresh worktree's `.mise.toml` is untrusted, which also + breaks the shims: `mise ERROR Config files ... are not trusted`). +2. **Put an installed oxfmt on PATH** before committing (this is what actually unblocked the + kguardian PR): + + ```bash + # 0.66.0 is what's actually installed on the agent box (the repo's .mise.toml pins + # 0.67.0 — if the pinned version is installed, use that instead); verify with: + # ls /opt/data/home/.local/share/mise/installs/oxfmt/ + export PATH="/opt/data/home/.local/share/mise/installs/oxfmt/0.66.0/node_modules/.bin:$PATH" + git commit -m "..." + ``` + +If formatting output differs between the local 0.66.0 and the pinned 0.67.0 (CI uses the pin), +prefer option 1 so local and CI agree. + ## Ceph: `crash ls-new` hides archived crashes, not old ones `ceph crash ls-new` filters on exactly one thing, whether a crash is archived (`crash/module.py` @@ -71,14 +101,108 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ## Secrets management (SOPS) -1. Create unencrypted file first -2. Encrypt with: `sops --encrypt --in-place ` -3. Or create with: `sops .yaml` (edits encrypted) +### Where SOPS actually runs — the agent box has no age key + +Decrypt / encrypt / re-encrypt **cannot run on the agent box**: the config root has no `age.key` +(verified: `/opt/data/cluster/age.key` is absent), so every SOPS op must run **on the management +host**. The age key lives at `tanguille@192.168.0.181:~/cluster/age.key`. + +The `k8s-management` ssh alias is **dead** in this environment: the user's passwd home is +`/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (key only, no +config) and never sees the alias defined in `/opt/data/home/.ssh/config`. Until that is fixed, +always connect explicitly: -Never commit plaintext secrets or the age key. Use placeholders so I can add the secrets manually. +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +``` -Post-quantum age (age1pq1) is supported. Both halves of the key must be present in `age.key`: a -key that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare `sops --version` +prints nothing. Prefix with `mise exec -- sops …` (or invoke the shim under a trusted `cwd` that +has a trusted `.mise.toml`). + +### Recipients differ per subtree (read `.sops.yaml`) + +Do not assume one key. `.sops.yaml` maps path → age recipient: + +| Path | Key (first 10 chars) | +|------|----------------------| +| `talos/**/*.sops.yaml` | `age12gul5m0…` | +| `(bootstrap\|kubernetes)/**/*.sops.yaml` | `age1pq1f69…` (post-quantum) | + +A CloudNativePG role secret under `kubernetes/` uses the `age1pq1…` key; a `talos/` file uses +`age12gul5m0…`. Encrypting with the wrong recipient (or hand-adding a `sops:` block) breaks +decrypt for the real owner. + +### The three SOPS traps (each cost real time — avoid them) + +1. **`stringData` → `data` on decrypt/encrypt.** SOPS round-trips convert `stringData` keys into + `data`. Text-editing a decrypted file and re-encrypting can silently drop or mis-key + `stringData` entries. Rebuild via a **dict** in Python (load the encrypted YAML, set the + value, re-encrypt) rather than text-splicing. +2. **Stale `sops:` footer after a textual edit.** If you text-edit an encrypted file (or a + decrypt→edit), the old `sops:` metadata block remains and `sops --encrypt` fails on it. Always + **decrypt first**, edit the clean file, then encrypt — never edit the ciphertext in place. +3. **Missing `--input-type yaml` makes SOPS guess JSON and fail.** Always pass + `--input-type yaml --output-type yaml` on YAML files; do not rely on extension sniffing. + +### Canonical flow (new secret value, remote) + +1. `sops --encrypt` is a no-op on already-encrypted content; for a **new** value, decrypt the + existing file to a temp, set the key in a dict, and re-encrypt with the correct recipient: + + ```bash + # on the management host, in a trusted cwd (a worktree with .mise.toml) + mise exec -- sops --input-type yaml --output-type yaml -d file.sops.yaml > /tmp/plain.yaml + # edit /tmp/plain.yaml (dict-based if it has stringData), then: + mise exec -- sops --input-type yaml --output-type yaml -e /tmp/plain.yaml > file.sops.yaml + rm /tmp/plain.yaml # never leave a decrypted secret on disk + ``` + +2. Verify the recipient line in the resulting `sops:` block matches the subtree above. +3. `rm` the decrypted temp **immediately** (approval-gated; ask the user). + +### Standing rules + +- Never commit plaintext secrets or the age key. Use placeholders so the user adds values manually. +- **Ask before decrypting/editing SOPS** (AGENTS.md) and before `rm`-ing a decrypted temp. +- Post-quantum age (`age1pq1…`) is supported. Both halves of a key must be in `age.key`: a key + that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +- The remote `~/cluster` checkout may be on a **stale feature branch** (it was 18 commits behind + `origin/main` during the kguardian work). `git fetch` there before trusting its state; SOPS + only needs the `.sops.yaml` + `age.key`, not a fresh tree, but re-encrypting against a stale + tree can carry in stale content — prefer editing the specific file. + +## PR shepherding (re-shepherd pass) + +Standing order: iterate an owner PR until CI + automated review are clean. One pass: + +1. **`git fetch origin` first** (hard rule — this worktree sits on a feature branch and + is always stale; never answer current-state questions from it). +2. Read the PR via ToolHive `github_pull_request_read` (`method: get`): state (`open`/ + `draft`), head SHA, base SHA, `mergeable`, `mergeable_state`, `merge_state_status`, + `commits` count, diffstat. +3. **Base moved?** If `base.sha != origin/main` head: compare overlap — + `git diff --name-only ...origin/main` vs the PR's file list. Overlap files + are rebase candidates; dry-run with `git merge-tree origin/main` + (or `git merge-tree --write-tree origin/main`) to confirm clean. +4. **Rebase = Gate A, server-side only**: `github_update_pull_request_branch` with + `expectedHeadSha` = the current head SHA (guards against concurrent pushes). **No + local `git push`** — the agent box has no GitHub token by default; branch updates go + through ToolHive `push_files` (full-file contents, no delete) or the server-side + rebase. +5. `mergeable_state: "unknown"` right after a base move usually just means GitHub is + recalculating — confirm with the rebase rather than looping on polls. +6. Re-poll `get_check_runs` until the new head is `success`. Normal draft-green shape: + ~12 success + 2 skipped (CodeRabbit/DeepSource skip on drafts). +7. Read comments for *new* feedback since the last pass; address or answer it. +8. **Budget: 3 fix cycles** per issue class, then escalate to the owner with evidence + (log excerpts, which checks failed, and the classification: flake vs diff vs + baseline) — don't silently keep retrying. +9. **Hard no-s without explicit per-instance owner approval:** merging the PR, + pushing to `main`, force-pushing any branch, `cluster-apply`, decrypting secrets + into a PR description/log/chat. +10. Status wording: report `mergeable_state`/`merge_state_status` verbatim; do not + imply "clean" while either is `unknown`/`behind`. ## Debugging @@ -90,4 +214,4 @@ Use [backup-restore](skills/backup-restore/SKILL.md) skill for kopiur Kopia oper ## Other skills -See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, and prometheus-cluster-health. +See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, cluster-sops, and prometheus-cluster-health. diff --git a/.agents/learned-preferences.md b/.agents/learned-preferences.md index 371c0bbf4e..1056547a4b 100644 --- a/.agents/learned-preferences.md +++ b/.agents/learned-preferences.md @@ -1,6 +1,6 @@ # Learned User Preferences -**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools. +**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools, PR shepherd/rebase, SOPS ask-first. Maintained from session feedback. Prefer git revert, don't undo user changes, only adjust resources where already set. @@ -17,6 +17,19 @@ If the available tools include `find_tool` and `call_tool` (ToolHive unified gat 3. Call `call_tool` to execute it. 4. Interpret the result and respond naturally—never return raw JSON or raw tool output to the user. +**Exact `call_tool` shape (two-level, the `github_` prefix matters):** + +```json +{"name": "mcp__toolhive__call_tool", + "arguments": {"tool_name": "github_pull_request_read", + "parameters": {"owner": "Tanguille", "repo": "cluster", + "pullNumber": 4998, "method": "get"}}} +``` + +- The inner tool name carries the **`github_` prefix**; the PR number is **`pullNumber`, not `number`** (both are easy to get wrong). +- **Large payloads** (multi-MB `push_files` bodies) can make the *local* SDK throw `SSE stream ended` / `TaskGroup` — while the **server-side operation still succeeded**. On such a failure: do NOT blindly retry (double-push / orphan-commit risk); first read the PR state back (`github_pull_request_read`) to see whether it landed. +- ~10 rapid consecutive calls can trip a transient "unreachable" for ~60s; a short pause + a single retry usually succeeds. + ### Other tools and behavior - For any other tools, call them directly when relevant, without unnecessary preamble. diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md new file mode 100644 index 0000000000..4335a850c3 --- /dev/null +++ b/.agents/skills/cluster-sops/SKILL.md @@ -0,0 +1,127 @@ +--- +name: cluster-sops +description: >- + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the remote + management host, where the age key lives, using a single scripted round-trip. + + user: "add a new secret for app X" → remote sops encrypt with the right recipients + user: "change the DB password" → decrypt → edit in dict form → re-encrypt + user: "re-encrypt after changing recipients" → sops updatekeys + + Use proactively whenever a *.sops.yaml value must be created or changed. Ask the + owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. +compatibility: Requires `ssh` to `tanguille@192.168.0.181` (key at /opt/data/.ssh/id_ed25519), `sops` + `python3` + `mise` on that host, and the repo's `.sops.yaml` + `age.key` under `~/cluster/`. + +--- + +# Cluster SOPS (remote secrets workflow) + +## Where it runs — and why + +- The **agent box has no age key** (no `age.key` under the config root) and no usable + local `sops` config for the right recipients. Every decrypt/encrypt/re-encrypt + **must run on the management host**: `tanguille@192.168.0.181`, repo at `~/cluster`, + key at `~/cluster/age.key`. +- **SSH from the agent box** (verified working form): + `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ''` + - The `k8s-management` alias in `~/.ssh/config` is **dead in this environment**: + the passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads + `/opt/data/.ssh/config` (absent) and the alias never resolves. Don't use it until + the config is placed at `/opt/data/.ssh/config` (or symlinked). + - Use `BatchMode=yes` for scripted runs. +- **`sops` on the remote is a mise shim** — bare `sops --version` can print nothing. + Run it as `mise exec -- sops …` (or from a `mise trust`-ed directory) so it resolves. + +## Recipients (from `.sops.yaml` — read it, don't trust memory) + +| path_regex | recipient | +|---|---| +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (short) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum, long) | + +`sops` picks the rule by path automatically — so **run it against the file's real +path** (or pass the file inside the matching subtree). Both halves of the age key +must be in `age.key`: a key missing the PQ half decrypts `talos/` but fails on +`kubernetes/`. + +## The three traps (each already cost time — avoid them) + +1. **`stringData` → `data` on decrypt/encrypt.** SOPS normalizes `stringData` into + base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those keys). + A text round-trip (decrypt → edit → encrypt) can **drop keys or mangle the mapping**. + **Fix:** rebuild with a Python dict on the remote — load decrypted YAML, set the + values in the right section, dump back — instead of hand-editing text. +2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt + that left the metadata block) makes the next `sops encrypt` fail on the existing + footer. **Fix:** always start from a *fully decrypted* file (metadata stripped) + before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without `--input-type yaml + --output-type yaml`, sops may guess JSON and fail on YAML secrets. **Fix:** pass + both flags explicitly every time. + +## Canonical flows (remote, one scripted call each) + +### New secret value + +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' +set -e; cd ~/cluster +# 1. stage the plaintext in the right subtree (path must match the .sops.yaml rule) +# e.g. kubernetes/apps///.sops.yaml with data: {key: value} +# 2. encrypt in place: +mise exec -- sops encrypt --in-place \ + --input-type yaml --output-type yaml kubernetes/apps///.sops.yaml +# 3. sanity: only data/stringData encrypted, sops footer present, nothing plaintext +grep -n "ENC[A-Z0-9]\{32,\}" kubernetes/apps///.sops.yaml | head +' +``` + +### Change an existing value (dict-based, no text surgery) + +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' +set -e; cd ~/cluster; F=kubernetes/apps///.sops.yaml +mise exec -- sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml +python3 - <.plain.yaml # ask-first: never leave decrypted temp behind +' +``` + +### Re-encrypt after a recipient change + +```bash +mise exec -- sops updatekeys --in-place \ + --input-type yaml --output-type yaml +``` + +### Add a CNPG managed role / DB secret (CNPG subtree) + +Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its +`managed.roles[].passwordSecret` points at it), so it encrypts under the +`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the +password in the `.sops.yaml`. + +## Standing rules + +- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before + `rm`-ing a decrypted temp file. +- **Never echo decrypted values** into chat, PR descriptions, logs, or commit + messages. Use `[REDACTED]`. +- The remote `~/cluster` checkout may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). SOPS only needs `.sops.yaml` + + `age.key`, but `git fetch` + check the branch before trusting file state there. +- After encryption: `git diff` on the remote should show **only ciphertext changes** + (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit the `.sops.yaml` change in the agent-box worktree (pull the encrypted bytes + back with `scp`), then follow the normal commit/PR flow — the remote is for + SOPS only, not for `git push` (which needs a token; use bundles or ToolHive + `push_files` per the PR-shepherding rules in `common-operations.md`). diff --git a/AGENTS.md b/AGENTS.md index aa33783d64..77cdbc9f80 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ GitOps-based Kubernetes cluster on Talos Linux with FluxCD reconciliation. Make - [Learned preferences](.agents/learned-preferences.md): tool selection, ToolHive workflow, reversions, resources, and confidence - [Learned workspace](.agents/learned-workspace.md): cluster-specific Kubernetes, ToolHive, database, storage, Talos, and media facts - [Common operations](.agents/common-operations.md): validation, app operations, SOPS, debugging, and backup/restore -- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, debug-cluster, git-worktree-isolation, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory +- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, cluster-sops, debug-cluster, git-worktree-isolation, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory - [Useful commands](docs/useful_commands.md): flux/just/talos/sops command reference and app-specific runbooks - [Archived migrations](docs/archived-migrations.md): completed ZFS→Ceph, Radarr→Postgres and OPNsense BGP procedures, with git recovery pointers - [LLM hosting](docs/llm-hosting/): sglang/vLLM tuning constraints and benchmark history for `kubernetes/apps/ai/` From 33a9b17130e755748e3f1c5d56caeff1594efffb Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 17:51:15 +0200 Subject: [PATCH 02/11] =?UTF-8?q?docs(sops):=20local-first=20=E2=80=94=20a?= =?UTF-8?q?gent=20box=20now=20holds=20its=20own=20age=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - common-operations: SOPS runs locally (SOPS_AGE_KEY_FILE required; sops 3.13.3 does not auto-discover the key path); remote host demoted to fallback (key revocation scenario) - cluster-sops skill: rewritten local-first with remote fallback (dead k8s-management alias, mise shims, fish, stale-branch notes) - note: agent-box public key pending addition to .sops.yaml + sops updatekeys over the .sops.yaml file set --- .agents/common-operations.md | 45 +++++--- .agents/skills/cluster-sops/SKILL.md | 155 +++++++++++++++------------ 2 files changed, 121 insertions(+), 79 deletions(-) diff --git a/.agents/common-operations.md b/.agents/common-operations.md index 66b4d55e59..683f6b34e9 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -101,24 +101,45 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ## Secrets management (SOPS) -### Where SOPS actually runs — the agent box has no age key +### Where SOPS runs — local-first (2026-09-14: SSH no longer required) -Decrypt / encrypt / re-encrypt **cannot run on the agent box**: the config root has no `age.key` -(verified: `/opt/data/cluster/age.key` is absent), so every SOPS op must run **on the management -host**. The age key lives at `tanguille@192.168.0.181:~/cluster/age.key`. +The agent box now has **its own age key** (post-quantum ML-KEM-768 + X25519, which +decrypts *both* recipient types below) at `~/.config/sops/age/keys.txt` (mode 600, +inside 700 directories, Ceph-backed — same regime as the SSH key). SOPS + age are +installed locally via aqua/mise. **All SOPS ops now run locally** — no ssh, no scp, +no fish, no stale remote branch. -The `k8s-management` ssh alias is **dead** in this environment: the user's passwd home is -`/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (key only, no -config) and never sees the alias defined in `/opt/data/home/.ssh/config`. Until that is fixed, -always connect explicitly: +**Required for every sops call** — sops 3.13.3 does NOT auto-discover this key +path (it checks `~/.ssh`, `SOPS_AGE_KEY`; it will fail with "no identity matched"): ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" ``` -On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare `sops --version` -prints nothing. Prefix with `mise exec -- sops …` (or invoke the shim under a trusted `cwd` that -has a trusted `.mise.toml`). +Local `sops` binary (the mise shims are unreliable in non-login shells): +`/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + +**Full procedure → [cluster-sops skill](skills/cluster-sops/SKILL.md).** Canonical +local flows (new value / change value / `updatekeys`) and the three traps are there. + +#### Fallback: the remote management host (only if the local key is revoked) + +If the agent-box key is ever revoked, SOPS ops fall back to the management host, +where the original key still lives (`tanguille@192.168.0.181:~/cluster/age.key`). +The `k8s-management` ssh alias is **dead** in this environment: the user's passwd +home is `/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads +`/opt/data/.ssh/config` (key only, no config) and never sees the alias in +`/opt/data/home/.ssh/config`. Connect explicitly: + +```bash +ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +``` + +On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare +`sops --version` prints nothing. Prefix with `mise exec -- sops …` (or invoke the +shim under a trusted `cwd` with a trusted `.mise.toml`). The remote shell is +**fish** — pipe complex commands via `ssh … 'python3 -' < local.py`, never +multi-line heredocs. ### Recipients differ per subtree (read `.sops.yaml`) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 4335a850c3..7220efe7e0 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1,106 +1,129 @@ --- name: cluster-sops description: >- - Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the remote - management host, where the age key lives, using a single scripted round-trip. + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent + box, where its own age key now lives (one local command, no SSH), with the remote + management host as fallback. - user: "add a new secret for app X" → remote sops encrypt with the right recipients + user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients user: "change the DB password" → decrypt → edit in dict form → re-encrypt user: "re-encrypt after changing recipients" → sops updatekeys Use proactively whenever a *.sops.yaml value must be created or changed. Ask the owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. -compatibility: Requires `ssh` to `tanguille@192.168.0.181` (key at /opt/data/.ssh/id_ed25519), `sops` + `python3` + `mise` on that host, and the repo's `.sops.yaml` + `age.key` under `~/cluster/`. - +compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). --- -# Cluster SOPS (remote secrets workflow) - -## Where it runs — and why - -- The **agent box has no age key** (no `age.key` under the config root) and no usable - local `sops` config for the right recipients. Every decrypt/encrypt/re-encrypt - **must run on the management host**: `tanguille@192.168.0.181`, repo at `~/cluster`, - key at `~/cluster/age.key`. -- **SSH from the agent box** (verified working form): - `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ''` - - The `k8s-management` alias in `~/.ssh/config` is **dead in this environment**: - the passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads - `/opt/data/.ssh/config` (absent) and the alias never resolves. Don't use it until - the config is placed at `/opt/data/.ssh/config` (or symlinked). - - Use `BatchMode=yes` for scripted runs. -- **`sops` on the remote is a mise shim** — bare `sops --version` can print nothing. - Run it as `mise exec -- sops …` (or from a `mise trust`-ed directory) so it resolves. +# Cluster SOPS (local-first secrets workflow) + +## Where it runs — and why (2026-09-14: SSH no longer required) + +- **The agent box has its own age key** (post-quantum, ML-KEM-768 + X25519) at + `~/.config/sops/age/keys.txt` (mode 600, inside 700 directories). A PQ identity + decrypts **both** recipient types in `.sops.yaml` (`age1pq…` and plain `age1…`). + Full encrypt→decrypt round-trip verified locally. +- **Required env for every sops call** (sops 3.13.3 does NOT auto-discover this + path — it looks in `~/.ssh` / `SOPS_AGE_KEY` and will fail with "no identity + matched" otherwise): + + ```bash + export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" + ``` + +- **Local sops binary** (aqua install; the mise shims are unreliable in non-login + shells): `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + `age-keygen` (for key ops) is at + `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the + identity file as a *positional* argument (no `-r` flag), and the `age` binary in + the same directory is the encrypt/decrypt CLI (no `-g` flag). + +### Fallback: the remote management host + +If the agent-box key is ever revoked (the whole point of a revocable key), or a +`talos/` file predates the migration: `ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 +tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline): + +- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` + but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and + the alias never resolves. +- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use + `mise exec -- sops …`. +- Remote `~/cluster` may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch + before trusting file state there. +- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, + never multi-line heredocs. ## Recipients (from `.sops.yaml` — read it, don't trust memory) | path_regex | recipient | |---|---| -| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (short) | -| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum, long) | +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | -`sops` picks the rule by path automatically — so **run it against the file's real -path** (or pass the file inside the matching subtree). Both halves of the age key -must be in `age.key`: a key missing the PQ half decrypts `talos/` but fails on -`kubernetes/`. +`sops` picks the rule by path automatically — **run it against the file's real +path** (or pass the file inside the matching subtree). + +**Migration in flight (2026-09-14):** the agent-box key's public key +(`age1pq1hzp5…`, full value in memini) is pending addition to `.sops.yaml` + +`git ls-files | grep '\.sops\.ya?ml$' | xargs sops updatekeys --encrypt`. +Until that ships, the agent-box key decrypts `kubernetes|bootstrap/` files (its +PQ identity matches `age1pq1f69…`) but NOT `talos/` files (plain `age12gul5m0…`) — +for those, use the remote fallback or run updatekeys first. ## The three traps (each already cost time — avoid them) 1. **`stringData` → `data` on decrypt/encrypt.** SOPS normalizes `stringData` into - base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those keys). - A text round-trip (decrypt → edit → encrypt) can **drop keys or mangle the mapping**. - **Fix:** rebuild with a Python dict on the remote — load decrypted YAML, set the - values in the right section, dump back — instead of hand-editing text. + base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those + keys). A text round-trip can **drop keys or mangle the mapping**. + **Fix:** rebuild with a Python dict — load decrypted YAML, set the values in + the right section, dump back — instead of hand-editing text. 2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt - that left the metadata block) makes the next `sops encrypt` fail on the existing - footer. **Fix:** always start from a *fully decrypted* file (metadata stripped) - before re-encrypting; never edit the ciphertext by hand. -3. **Missing `--input-type`/`--output-type`.** Without `--input-type yaml - --output-type yaml`, sops may guess JSON and fail on YAML secrets. **Fix:** pass - both flags explicitly every time. + that left the metadata block) makes the next `sops encrypt` fail on the + existing footer. **Fix:** always start from a *fully decrypted* file (metadata + stripped) before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess + JSON and fail on YAML secrets ("error: no matching creation rules found" or a + JSON-guess failure). **Fix:** pass both flags explicitly every time. + +## Canonical flows (local-first, one command each) -## Canonical flows (remote, one scripted call each) +All examples assume the `export SOPS_AGE_KEY_FILE=…` above and `cd` into the +worktree that contains `.sops.yaml`. ### New secret value ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' -set -e; cd ~/cluster -# 1. stage the plaintext in the right subtree (path must match the .sops.yaml rule) -# e.g. kubernetes/apps///.sops.yaml with data: {key: value} -# 2. encrypt in place: -mise exec -- sops encrypt --in-place \ - --input-type yaml --output-type yaml kubernetes/apps///.sops.yaml -# 3. sanity: only data/stringData encrypted, sops footer present, nothing plaintext -grep -n "ENC[A-Z0-9]\{32,\}" kubernetes/apps///.sops.yaml | head -' +F=kubernetes/apps///.sops.yaml # path MUST match the .sops.yaml rule +# stage plaintext with data: {key: base64value} or stringData: {key: plain} +sops encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted, sops footer present, nothing plaintext +grep -n "ENC[A-Z0-9]\{32,\}" "$F" | head ``` ### Change an existing value (dict-based, no text surgery) ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' -set -e; cd ~/cluster; F=kubernetes/apps///.sops.yaml -mise exec -- sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml +F=kubernetes/apps///.sops.yaml +sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml python3 - <.plain.yaml # ask-first: never leave decrypted temp behind -' +sops encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f /tmp/.plain.yaml # never leave decrypted temp behind ``` ### Re-encrypt after a recipient change ```bash -mise exec -- sops updatekeys --in-place \ - --input-type yaml --output-type yaml +git ls-files | grep '\.sops\.ya?ml$' | xargs \ + sops updatekeys --in-place --input-type yaml --output-type yaml ``` ### Add a CNPG managed role / DB secret (CNPG subtree) @@ -116,12 +139,10 @@ password in the `.sops.yaml`. `rm`-ing a decrypted temp file. - **Never echo decrypted values** into chat, PR descriptions, logs, or commit messages. Use `[REDACTED]`. -- The remote `~/cluster` checkout may sit on a **stale feature branch** (it was on - `feat/truenas-mcp` during the kguardian work). SOPS only needs `.sops.yaml` + - `age.key`, but `git fetch` + check the branch before trusting file state there. -- After encryption: `git diff` on the remote should show **only ciphertext changes** - (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). -- Commit the `.sops.yaml` change in the agent-box worktree (pull the encrypted bytes - back with `scp`), then follow the normal commit/PR flow — the remote is for - SOPS only, not for `git push` (which needs a token; use bundles or ToolHive - `push_files` per the PR-shepherding rules in `common-operations.md`). +- The **private age key never leaves the agent box**: it is not committed, not + copied to the remote, not echoed. Only its *public* key goes into `.sops.yaml`. +- After encryption: `git diff` should show **only ciphertext changes** (no + plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit in the agent-box worktree and follow the normal commit/PR flow (PR + shepherding rules in `common-operations.md`). The remote is a fallback for + SOPS + the only push path if ToolHive is down. From 0561450b59e01ca1063d86d7c92f7de4010fbc6a Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 19:41:47 +0200 Subject: [PATCH 03/11] =?UTF-8?q?docs(agents):=20correct=20SOPS=20reality?= =?UTF-8?q?=20=E2=80=94=20master=20keys=20now=20local,=20all=20files=20dec?= =?UTF-8?q?ryptable=20w/o=20ssh?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - keys.txt holds 3 identities (agent revocable key + 2 master keys copied 2026-09-14) - remove false claim that the new agent key alone decrypts both recipient types - remote becomes a revocation fallback, not the primary path - no existing file re-encrypted; no recipient changes --- .agents/common-operations.md | 12 ++++--- .agents/skills/cluster-sops/SKILL.md | 51 ++++++++++++++++++++-------- 2 files changed, 43 insertions(+), 20 deletions(-) diff --git a/.agents/common-operations.md b/.agents/common-operations.md index 683f6b34e9..3ee8a90816 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -103,11 +103,13 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ### Where SOPS runs — local-first (2026-09-14: SSH no longer required) -The agent box now has **its own age key** (post-quantum ML-KEM-768 + X25519, which -decrypts *both* recipient types below) at `~/.config/sops/age/keys.txt` (mode 600, -inside 700 directories, Ceph-backed — same regime as the SSH key). SOPS + age are -installed locally via aqua/mise. **All SOPS ops now run locally** — no ssh, no scp, -no fish, no stale remote branch. +The agent box now holds an age key file at `~/.config/sops/age/keys.txt` +(mode 600, inside 700 directories, Ceph-backed — same regime as the SSH key) +containing **3 identities**: the agent box's own revocable key plus the two +master keys, copied over 2026-09-14 on the owner's explicit OK. It can therefore +**decrypt every file in the repo locally** (verified on real `kubernetes/` and +`talos/` files) — no ssh, no scp, no fish, no stale remote branch. No existing +file was re-encrypted and no recipients were changed. **Required for every sops call** — sops 3.13.3 does NOT auto-discover this key path (it checks `~/.ssh`, `SOPS_AGE_KEY`; it will fail with "no identity matched"): diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 7220efe7e0..bdf7de58d8 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -18,10 +18,12 @@ compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/a ## Where it runs — and why (2026-09-14: SSH no longer required) -- **The agent box has its own age key** (post-quantum, ML-KEM-768 + X25519) at - `~/.config/sops/age/keys.txt` (mode 600, inside 700 directories). A PQ identity - decrypts **both** recipient types in `.sops.yaml` (`age1pq…` and plain `age1…`). - Full encrypt→decrypt round-trip verified locally. +- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` + (mode 600, inside 700 directories) with **3 identities**: the agent-box's own + revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) + copied from the remote 2026-09-14. It therefore decrypts **every** file in the + repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. + No existing file was re-encrypted and no recipients were changed. - **Required env for every sops call** (sops 3.13.3 does NOT auto-discover this path — it looks in `~/.ssh` / `SOPS_AGE_KEY` and will fail with "no identity matched" otherwise): @@ -39,9 +41,10 @@ compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/a ### Fallback: the remote management host -If the agent-box key is ever revoked (the whole point of a revocable key), or a -`talos/` file predates the migration: `ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 -tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline): +Only needed if the master keys are ever removed from `keys.txt` (to shrink the +agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS +decrypt falls back to the management host, where the original key lives +(`tanguille@192.168.0.181:~/cluster/age.key`): - The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and @@ -64,12 +67,27 @@ tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline) `sops` picks the rule by path automatically — **run it against the file's real path** (or pass the file inside the matching subtree). -**Migration in flight (2026-09-14):** the agent-box key's public key -(`age1pq1hzp5…`, full value in memini) is pending addition to `.sops.yaml` + -`git ls-files | grep '\.sops\.ya?ml$' | xargs sops updatekeys --encrypt`. -Until that ships, the agent-box key decrypts `kubernetes|bootstrap/` files (its -PQ identity matches `age1pq1f69…`) but NOT `talos/` files (plain `age12gul5m0…`) — -for those, use the remote fallback or run updatekeys first. +**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds +**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two +master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing +file was re-encrypted or had its recipients changed ("don't touch existing +encryption" — honored). Consequences: + +- **Decrypt/edit any existing file: works locally**, no SSH (verified on real + `kubernetes/` and `talos/` files). +- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) + — the agent-box key is a *redundant* decryptor, not a recipient, unless you + later decide to add it to `.sops.yaml` + `updatekeys`. +- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If + the agent box is compromised, *rotate the master* (the revocable-agent-key + model is no longer the only blast-radius control). If you'd rather not keep + that exposure, the master lines can be removed from `keys.txt` (a backup of + the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would + then fall back to SSH. + +**PQ note (learned the hard way):** "post-quantum" is a key *format* +(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. +A brand-new PQ key cannot decrypt files encrypted to an older PQ key. ## The three traps (each already cost time — avoid them) @@ -139,8 +157,11 @@ password in the `.sops.yaml`. `rm`-ing a decrypted temp file. - **Never echo decrypted values** into chat, PR descriptions, logs, or commit messages. Use `[REDACTED]`. -- The **private age key never leaves the agent box**: it is not committed, not - copied to the remote, not echoed. Only its *public* key goes into `.sops.yaml`. +- The **master age key now lives on the agent box too** (copied 2026-09-14 by + explicit owner decision; `keys.txt` = 3 identities). It is still not committed + and not echoed — only *public* keys ever go into `.sops.yaml`. If you later + want to shrink the blast radius, remove the master lines from `keys.txt` + (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. - After encryption: `git diff` should show **only ciphertext changes** (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). - Commit in the agent-box worktree and follow the normal commit/PR flow (PR From ca8d865e4f94ad10347c9c494ad1dd5585bcc589 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 16:43:44 +0200 Subject: [PATCH 04/11] docs(agents): capture SOPS/remote, PR-shepherd, ToolHive, pre-commit lessons New cluster-sops skill (remote-only SOPS workflow, recipient map, the three traps, canonical scripted flows) + supporting edits: - common-operations.md: where SOPS actually runs (agent box has no age key, k8s-management ssh alias is dead here, remote sops is a mise shim), per-subtree recipient table, stringData->data / stale-footer / --input-type traps, canonical remote flow, and a PR-shepherd re-shepherd pass (fetch first, server-side-only rebase = Gate A, 3-cycle budget, hard no-s) - learned-preferences.md: exact ToolHive call_tool shape (github_ prefix, pullNumber, large-payload SSE/TaskGroup pitfall, rapid-call backoff) - AGENTS.md + catalog listings: register cluster-sops All facts verified on the live system 2026-09-14. --- .agents/common-operations.md | 138 +++++++++++++++++++++++++-- .agents/learned-preferences.md | 15 ++- .agents/skills/cluster-sops/SKILL.md | 127 ++++++++++++++++++++++++ AGENTS.md | 2 +- 4 files changed, 273 insertions(+), 9 deletions(-) create mode 100644 .agents/skills/cluster-sops/SKILL.md diff --git a/.agents/common-operations.md b/.agents/common-operations.md index b05a0f8d3a..66b4d55e59 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -13,6 +13,36 @@ Step-by-step procedures for frequent cluster tasks. - Shell-only changes: `mise exec -- shellcheck` on every touched `*.sh`. - Documentation-only changes: run `git diff --check` and verify every changed local reference exists. +### Pre-commit: `oxfmt not found` (agent box) + +Lefthook (`.lefthook.toml`) runs **bare** `oxfmt` on staged `*.yaml`/`*.yml` (excluding +`*.sops.yaml`) and `*.json*`. On the agent box `oxfmt` is a mise tool and **not on PATH**, so the +hook fails with "oxfmt not found" even though the code is fine. Verified facts: + +- `mise` itself is not on PATH in non-login shells; tool binaries live under + `/opt/data/home/.local/share/mise/installs/` (with `shims/` alongside). +- The repo pins `oxfmt = "0.67.0"` (`.mise.toml`) but the agent box has `0.66.0` installed — + the local binary lags the pin. + +Fixes, in order of preference: + +1. **Commit in a mise-active shell** so the pinned version resolves: `mise exec -- git commit …` + (after `mise trust` in the worktree — a fresh worktree's `.mise.toml` is untrusted, which also + breaks the shims: `mise ERROR Config files ... are not trusted`). +2. **Put an installed oxfmt on PATH** before committing (this is what actually unblocked the + kguardian PR): + + ```bash + # 0.66.0 is what's actually installed on the agent box (the repo's .mise.toml pins + # 0.67.0 — if the pinned version is installed, use that instead); verify with: + # ls /opt/data/home/.local/share/mise/installs/oxfmt/ + export PATH="/opt/data/home/.local/share/mise/installs/oxfmt/0.66.0/node_modules/.bin:$PATH" + git commit -m "..." + ``` + +If formatting output differs between the local 0.66.0 and the pinned 0.67.0 (CI uses the pin), +prefer option 1 so local and CI agree. + ## Ceph: `crash ls-new` hides archived crashes, not old ones `ceph crash ls-new` filters on exactly one thing, whether a crash is archived (`crash/module.py` @@ -71,14 +101,108 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ## Secrets management (SOPS) -1. Create unencrypted file first -2. Encrypt with: `sops --encrypt --in-place ` -3. Or create with: `sops .yaml` (edits encrypted) +### Where SOPS actually runs — the agent box has no age key + +Decrypt / encrypt / re-encrypt **cannot run on the agent box**: the config root has no `age.key` +(verified: `/opt/data/cluster/age.key` is absent), so every SOPS op must run **on the management +host**. The age key lives at `tanguille@192.168.0.181:~/cluster/age.key`. + +The `k8s-management` ssh alias is **dead** in this environment: the user's passwd home is +`/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (key only, no +config) and never sees the alias defined in `/opt/data/home/.ssh/config`. Until that is fixed, +always connect explicitly: -Never commit plaintext secrets or the age key. Use placeholders so I can add the secrets manually. +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +``` -Post-quantum age (age1pq1) is supported. Both halves of the key must be present in `age.key`: a -key that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare `sops --version` +prints nothing. Prefix with `mise exec -- sops …` (or invoke the shim under a trusted `cwd` that +has a trusted `.mise.toml`). + +### Recipients differ per subtree (read `.sops.yaml`) + +Do not assume one key. `.sops.yaml` maps path → age recipient: + +| Path | Key (first 10 chars) | +|------|----------------------| +| `talos/**/*.sops.yaml` | `age12gul5m0…` | +| `(bootstrap\|kubernetes)/**/*.sops.yaml` | `age1pq1f69…` (post-quantum) | + +A CloudNativePG role secret under `kubernetes/` uses the `age1pq1…` key; a `talos/` file uses +`age12gul5m0…`. Encrypting with the wrong recipient (or hand-adding a `sops:` block) breaks +decrypt for the real owner. + +### The three SOPS traps (each cost real time — avoid them) + +1. **`stringData` → `data` on decrypt/encrypt.** SOPS round-trips convert `stringData` keys into + `data`. Text-editing a decrypted file and re-encrypting can silently drop or mis-key + `stringData` entries. Rebuild via a **dict** in Python (load the encrypted YAML, set the + value, re-encrypt) rather than text-splicing. +2. **Stale `sops:` footer after a textual edit.** If you text-edit an encrypted file (or a + decrypt→edit), the old `sops:` metadata block remains and `sops --encrypt` fails on it. Always + **decrypt first**, edit the clean file, then encrypt — never edit the ciphertext in place. +3. **Missing `--input-type yaml` makes SOPS guess JSON and fail.** Always pass + `--input-type yaml --output-type yaml` on YAML files; do not rely on extension sniffing. + +### Canonical flow (new secret value, remote) + +1. `sops --encrypt` is a no-op on already-encrypted content; for a **new** value, decrypt the + existing file to a temp, set the key in a dict, and re-encrypt with the correct recipient: + + ```bash + # on the management host, in a trusted cwd (a worktree with .mise.toml) + mise exec -- sops --input-type yaml --output-type yaml -d file.sops.yaml > /tmp/plain.yaml + # edit /tmp/plain.yaml (dict-based if it has stringData), then: + mise exec -- sops --input-type yaml --output-type yaml -e /tmp/plain.yaml > file.sops.yaml + rm /tmp/plain.yaml # never leave a decrypted secret on disk + ``` + +2. Verify the recipient line in the resulting `sops:` block matches the subtree above. +3. `rm` the decrypted temp **immediately** (approval-gated; ask the user). + +### Standing rules + +- Never commit plaintext secrets or the age key. Use placeholders so the user adds values manually. +- **Ask before decrypting/editing SOPS** (AGENTS.md) and before `rm`-ing a decrypted temp. +- Post-quantum age (`age1pq1…`) is supported. Both halves of a key must be in `age.key`: a key + that lost its PQ half decrypts `talos/` but fails on everything under `kubernetes/`. +- The remote `~/cluster` checkout may be on a **stale feature branch** (it was 18 commits behind + `origin/main` during the kguardian work). `git fetch` there before trusting its state; SOPS + only needs the `.sops.yaml` + `age.key`, not a fresh tree, but re-encrypting against a stale + tree can carry in stale content — prefer editing the specific file. + +## PR shepherding (re-shepherd pass) + +Standing order: iterate an owner PR until CI + automated review are clean. One pass: + +1. **`git fetch origin` first** (hard rule — this worktree sits on a feature branch and + is always stale; never answer current-state questions from it). +2. Read the PR via ToolHive `github_pull_request_read` (`method: get`): state (`open`/ + `draft`), head SHA, base SHA, `mergeable`, `mergeable_state`, `merge_state_status`, + `commits` count, diffstat. +3. **Base moved?** If `base.sha != origin/main` head: compare overlap — + `git diff --name-only ...origin/main` vs the PR's file list. Overlap files + are rebase candidates; dry-run with `git merge-tree origin/main` + (or `git merge-tree --write-tree origin/main`) to confirm clean. +4. **Rebase = Gate A, server-side only**: `github_update_pull_request_branch` with + `expectedHeadSha` = the current head SHA (guards against concurrent pushes). **No + local `git push`** — the agent box has no GitHub token by default; branch updates go + through ToolHive `push_files` (full-file contents, no delete) or the server-side + rebase. +5. `mergeable_state: "unknown"` right after a base move usually just means GitHub is + recalculating — confirm with the rebase rather than looping on polls. +6. Re-poll `get_check_runs` until the new head is `success`. Normal draft-green shape: + ~12 success + 2 skipped (CodeRabbit/DeepSource skip on drafts). +7. Read comments for *new* feedback since the last pass; address or answer it. +8. **Budget: 3 fix cycles** per issue class, then escalate to the owner with evidence + (log excerpts, which checks failed, and the classification: flake vs diff vs + baseline) — don't silently keep retrying. +9. **Hard no-s without explicit per-instance owner approval:** merging the PR, + pushing to `main`, force-pushing any branch, `cluster-apply`, decrypting secrets + into a PR description/log/chat. +10. Status wording: report `mergeable_state`/`merge_state_status` verbatim; do not + imply "clean" while either is `unknown`/`behind`. ## Debugging @@ -90,4 +214,4 @@ Use [backup-restore](skills/backup-restore/SKILL.md) skill for kopiur Kopia oper ## Other skills -See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, and prometheus-cluster-health. +See the [skill catalog](../AGENTS.md#load-context-on-demand) for git-worktree-isolation, k8s-at-home-research, pr-review, cluster-sops, and prometheus-cluster-health. diff --git a/.agents/learned-preferences.md b/.agents/learned-preferences.md index 34bee84ef4..24c727700f 100644 --- a/.agents/learned-preferences.md +++ b/.agents/learned-preferences.md @@ -1,6 +1,6 @@ # Learned User Preferences -**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools. +**When to use:** revert, undo, resources, memory, CPU, MCP vs shell, Flux reconcile, ToolHive, find_tool, call_tool, tool confidence, proactive tools, PR shepherd/rebase, SOPS ask-first. Maintained from session feedback. Prefer git revert, don't undo user changes, only adjust resources where already set. @@ -17,6 +17,19 @@ If the available tools include `find_tool` and `call_tool` (ToolHive unified gat 3. Call `call_tool` to execute it. 4. Interpret the result and respond naturally—never return raw JSON or raw tool output to the user. +**Exact `call_tool` shape (two-level, the `github_` prefix matters):** + +```json +{"name": "mcp__toolhive__call_tool", + "arguments": {"tool_name": "github_pull_request_read", + "parameters": {"owner": "Tanguille", "repo": "cluster", + "pullNumber": 4998, "method": "get"}}} +``` + +- The inner tool name carries the **`github_` prefix**; the PR number is **`pullNumber`, not `number`** (both are easy to get wrong). +- **Large payloads** (multi-MB `push_files` bodies) can make the *local* SDK throw `SSE stream ended` / `TaskGroup` — while the **server-side operation still succeeded**. On such a failure: do NOT blindly retry (double-push / orphan-commit risk); first read the PR state back (`github_pull_request_read`) to see whether it landed. +- ~10 rapid consecutive calls can trip a transient "unreachable" for ~60s; a short pause + a single retry usually succeeds. + ### Other tools and behavior - For any other tools, call them directly when relevant, without unnecessary preamble. diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md new file mode 100644 index 0000000000..4335a850c3 --- /dev/null +++ b/.agents/skills/cluster-sops/SKILL.md @@ -0,0 +1,127 @@ +--- +name: cluster-sops +description: >- + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the remote + management host, where the age key lives, using a single scripted round-trip. + + user: "add a new secret for app X" → remote sops encrypt with the right recipients + user: "change the DB password" → decrypt → edit in dict form → re-encrypt + user: "re-encrypt after changing recipients" → sops updatekeys + + Use proactively whenever a *.sops.yaml value must be created or changed. Ask the + owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. +compatibility: Requires `ssh` to `tanguille@192.168.0.181` (key at /opt/data/.ssh/id_ed25519), `sops` + `python3` + `mise` on that host, and the repo's `.sops.yaml` + `age.key` under `~/cluster/`. + +--- + +# Cluster SOPS (remote secrets workflow) + +## Where it runs — and why + +- The **agent box has no age key** (no `age.key` under the config root) and no usable + local `sops` config for the right recipients. Every decrypt/encrypt/re-encrypt + **must run on the management host**: `tanguille@192.168.0.181`, repo at `~/cluster`, + key at `~/cluster/age.key`. +- **SSH from the agent box** (verified working form): + `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ''` + - The `k8s-management` alias in `~/.ssh/config` is **dead in this environment**: + the passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads + `/opt/data/.ssh/config` (absent) and the alias never resolves. Don't use it until + the config is placed at `/opt/data/.ssh/config` (or symlinked). + - Use `BatchMode=yes` for scripted runs. +- **`sops` on the remote is a mise shim** — bare `sops --version` can print nothing. + Run it as `mise exec -- sops …` (or from a `mise trust`-ed directory) so it resolves. + +## Recipients (from `.sops.yaml` — read it, don't trust memory) + +| path_regex | recipient | +|---|---| +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (short) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum, long) | + +`sops` picks the rule by path automatically — so **run it against the file's real +path** (or pass the file inside the matching subtree). Both halves of the age key +must be in `age.key`: a key missing the PQ half decrypts `talos/` but fails on +`kubernetes/`. + +## The three traps (each already cost time — avoid them) + +1. **`stringData` → `data` on decrypt/encrypt.** SOPS normalizes `stringData` into + base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those keys). + A text round-trip (decrypt → edit → encrypt) can **drop keys or mangle the mapping**. + **Fix:** rebuild with a Python dict on the remote — load decrypted YAML, set the + values in the right section, dump back — instead of hand-editing text. +2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt + that left the metadata block) makes the next `sops encrypt` fail on the existing + footer. **Fix:** always start from a *fully decrypted* file (metadata stripped) + before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without `--input-type yaml + --output-type yaml`, sops may guess JSON and fail on YAML secrets. **Fix:** pass + both flags explicitly every time. + +## Canonical flows (remote, one scripted call each) + +### New secret value + +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' +set -e; cd ~/cluster +# 1. stage the plaintext in the right subtree (path must match the .sops.yaml rule) +# e.g. kubernetes/apps///.sops.yaml with data: {key: value} +# 2. encrypt in place: +mise exec -- sops encrypt --in-place \ + --input-type yaml --output-type yaml kubernetes/apps///.sops.yaml +# 3. sanity: only data/stringData encrypted, sops footer present, nothing plaintext +grep -n "ENC[A-Z0-9]\{32,\}" kubernetes/apps///.sops.yaml | head +' +``` + +### Change an existing value (dict-based, no text surgery) + +```bash +ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' +set -e; cd ~/cluster; F=kubernetes/apps///.sops.yaml +mise exec -- sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml +python3 - <.plain.yaml # ask-first: never leave decrypted temp behind +' +``` + +### Re-encrypt after a recipient change + +```bash +mise exec -- sops updatekeys --in-place \ + --input-type yaml --output-type yaml +``` + +### Add a CNPG managed role / DB secret (CNPG subtree) + +Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its +`managed.roles[].passwordSecret` points at it), so it encrypts under the +`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the +password in the `.sops.yaml`. + +## Standing rules + +- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before + `rm`-ing a decrypted temp file. +- **Never echo decrypted values** into chat, PR descriptions, logs, or commit + messages. Use `[REDACTED]`. +- The remote `~/cluster` checkout may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). SOPS only needs `.sops.yaml` + + `age.key`, but `git fetch` + check the branch before trusting file state there. +- After encryption: `git diff` on the remote should show **only ciphertext changes** + (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit the `.sops.yaml` change in the agent-box worktree (pull the encrypted bytes + back with `scp`), then follow the normal commit/PR flow — the remote is for + SOPS only, not for `git push` (which needs a token; use bundles or ToolHive + `push_files` per the PR-shepherding rules in `common-operations.md`). diff --git a/AGENTS.md b/AGENTS.md index 700405790c..23098b23a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ GitOps-based Kubernetes cluster on Talos Linux with FluxCD reconciliation. Make - [Learned preferences](.agents/learned-preferences.md): tool selection, ToolHive workflow, reversions, resources, and confidence - [Learned workspace](.agents/learned-workspace.md): cluster-specific Kubernetes, ToolHive, database, storage, Talos, and media facts - [Common operations](.agents/common-operations.md): validation, app operations, SOPS, debugging, and backup/restore -- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, debug-cluster, git-worktree-isolation, handoff, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory +- [Skill catalog](.agents/skills/): add-app-to-cluster, backup-restore, cluster-sops, debug-cluster, git-worktree-isolation, handoff, k8s-at-home-research, pr-review, prometheus-cluster-health — one `SKILL.md` per directory - [Useful commands](docs/useful_commands.md): flux/just/talos/sops command reference and app-specific runbooks - [Archived migrations](docs/archived-migrations.md): completed ZFS→Ceph, Radarr→Postgres and OPNsense BGP procedures, with git recovery pointers - [LLM hosting](docs/llm-hosting/): sglang/vLLM tuning constraints and benchmark history for `kubernetes/apps/ai/` From 20e7e9eb9dd70f7ae0306b9dc8450dae9fccafd9 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 17:51:15 +0200 Subject: [PATCH 05/11] =?UTF-8?q?docs(sops):=20local-first=20=E2=80=94=20a?= =?UTF-8?q?gent=20box=20now=20holds=20its=20own=20age=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - common-operations: SOPS runs locally (SOPS_AGE_KEY_FILE required; sops 3.13.3 does not auto-discover the key path); remote host demoted to fallback (key revocation scenario) - cluster-sops skill: rewritten local-first with remote fallback (dead k8s-management alias, mise shims, fish, stale-branch notes) - note: agent-box public key pending addition to .sops.yaml + sops updatekeys over the .sops.yaml file set --- .agents/common-operations.md | 45 +++++--- .agents/skills/cluster-sops/SKILL.md | 155 +++++++++++++++------------ 2 files changed, 121 insertions(+), 79 deletions(-) diff --git a/.agents/common-operations.md b/.agents/common-operations.md index 66b4d55e59..683f6b34e9 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -101,24 +101,45 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ## Secrets management (SOPS) -### Where SOPS actually runs — the agent box has no age key +### Where SOPS runs — local-first (2026-09-14: SSH no longer required) -Decrypt / encrypt / re-encrypt **cannot run on the agent box**: the config root has no `age.key` -(verified: `/opt/data/cluster/age.key` is absent), so every SOPS op must run **on the management -host**. The age key lives at `tanguille@192.168.0.181:~/cluster/age.key`. +The agent box now has **its own age key** (post-quantum ML-KEM-768 + X25519, which +decrypts *both* recipient types below) at `~/.config/sops/age/keys.txt` (mode 600, +inside 700 directories, Ceph-backed — same regime as the SSH key). SOPS + age are +installed locally via aqua/mise. **All SOPS ops now run locally** — no ssh, no scp, +no fish, no stale remote branch. -The `k8s-management` ssh alias is **dead** in this environment: the user's passwd home is -`/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (key only, no -config) and never sees the alias defined in `/opt/data/home/.ssh/config`. Until that is fixed, -always connect explicitly: +**Required for every sops call** — sops 3.13.3 does NOT auto-discover this key +path (it checks `~/.ssh`, `SOPS_AGE_KEY`; it will fail with "no identity matched"): ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" ``` -On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare `sops --version` -prints nothing. Prefix with `mise exec -- sops …` (or invoke the shim under a trusted `cwd` that -has a trusted `.mise.toml`). +Local `sops` binary (the mise shims are unreliable in non-login shells): +`/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + +**Full procedure → [cluster-sops skill](skills/cluster-sops/SKILL.md).** Canonical +local flows (new value / change value / `updatekeys`) and the three traps are there. + +#### Fallback: the remote management host (only if the local key is revoked) + +If the agent-box key is ever revoked, SOPS ops fall back to the management host, +where the original key still lives (`tanguille@192.168.0.181:~/cluster/age.key`). +The `k8s-management` ssh alias is **dead** in this environment: the user's passwd +home is `/opt/data` while `$HOME=/opt/data/home`, so OpenSSH reads +`/opt/data/.ssh/config` (key only, no config) and never sees the alias in +`/opt/data/home/.ssh/config`. Connect explicitly: + +```bash +ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 '' +``` + +On the remote, `sops` (and `age`, `kubectl`, …) are **mise shims**, so a bare +`sops --version` prints nothing. Prefix with `mise exec -- sops …` (or invoke the +shim under a trusted `cwd` with a trusted `.mise.toml`). The remote shell is +**fish** — pipe complex commands via `ssh … 'python3 -' < local.py`, never +multi-line heredocs. ### Recipients differ per subtree (read `.sops.yaml`) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 4335a850c3..7220efe7e0 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1,106 +1,129 @@ --- name: cluster-sops description: >- - Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the remote - management host, where the age key lives, using a single scripted round-trip. + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent + box, where its own age key now lives (one local command, no SSH), with the remote + management host as fallback. - user: "add a new secret for app X" → remote sops encrypt with the right recipients + user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients user: "change the DB password" → decrypt → edit in dict form → re-encrypt user: "re-encrypt after changing recipients" → sops updatekeys Use proactively whenever a *.sops.yaml value must be created or changed. Ask the owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. -compatibility: Requires `ssh` to `tanguille@192.168.0.181` (key at /opt/data/.ssh/id_ed25519), `sops` + `python3` + `mise` on that host, and the repo's `.sops.yaml` + `age.key` under `~/cluster/`. - +compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). --- -# Cluster SOPS (remote secrets workflow) - -## Where it runs — and why - -- The **agent box has no age key** (no `age.key` under the config root) and no usable - local `sops` config for the right recipients. Every decrypt/encrypt/re-encrypt - **must run on the management host**: `tanguille@192.168.0.181`, repo at `~/cluster`, - key at `~/cluster/age.key`. -- **SSH from the agent box** (verified working form): - `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ''` - - The `k8s-management` alias in `~/.ssh/config` is **dead in this environment**: - the passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads - `/opt/data/.ssh/config` (absent) and the alias never resolves. Don't use it until - the config is placed at `/opt/data/.ssh/config` (or symlinked). - - Use `BatchMode=yes` for scripted runs. -- **`sops` on the remote is a mise shim** — bare `sops --version` can print nothing. - Run it as `mise exec -- sops …` (or from a `mise trust`-ed directory) so it resolves. +# Cluster SOPS (local-first secrets workflow) + +## Where it runs — and why (2026-09-14: SSH no longer required) + +- **The agent box has its own age key** (post-quantum, ML-KEM-768 + X25519) at + `~/.config/sops/age/keys.txt` (mode 600, inside 700 directories). A PQ identity + decrypts **both** recipient types in `.sops.yaml` (`age1pq…` and plain `age1…`). + Full encrypt→decrypt round-trip verified locally. +- **Required env for every sops call** (sops 3.13.3 does NOT auto-discover this + path — it looks in `~/.ssh` / `SOPS_AGE_KEY` and will fail with "no identity + matched" otherwise): + + ```bash + export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" + ``` + +- **Local sops binary** (aqua install; the mise shims are unreliable in non-login + shells): `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + `age-keygen` (for key ops) is at + `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the + identity file as a *positional* argument (no `-r` flag), and the `age` binary in + the same directory is the encrypt/decrypt CLI (no `-g` flag). + +### Fallback: the remote management host + +If the agent-box key is ever revoked (the whole point of a revocable key), or a +`talos/` file predates the migration: `ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 +tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline): + +- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` + but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and + the alias never resolves. +- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use + `mise exec -- sops …`. +- Remote `~/cluster` may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch + before trusting file state there. +- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, + never multi-line heredocs. ## Recipients (from `.sops.yaml` — read it, don't trust memory) | path_regex | recipient | |---|---| -| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (short) | -| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum, long) | +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | -`sops` picks the rule by path automatically — so **run it against the file's real -path** (or pass the file inside the matching subtree). Both halves of the age key -must be in `age.key`: a key missing the PQ half decrypts `talos/` but fails on -`kubernetes/`. +`sops` picks the rule by path automatically — **run it against the file's real +path** (or pass the file inside the matching subtree). + +**Migration in flight (2026-09-14):** the agent-box key's public key +(`age1pq1hzp5…`, full value in memini) is pending addition to `.sops.yaml` + +`git ls-files | grep '\.sops\.ya?ml$' | xargs sops updatekeys --encrypt`. +Until that ships, the agent-box key decrypts `kubernetes|bootstrap/` files (its +PQ identity matches `age1pq1f69…`) but NOT `talos/` files (plain `age12gul5m0…`) — +for those, use the remote fallback or run updatekeys first. ## The three traps (each already cost time — avoid them) 1. **`stringData` → `data` on decrypt/encrypt.** SOPS normalizes `stringData` into - base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those keys). - A text round-trip (decrypt → edit → encrypt) can **drop keys or mangle the mapping**. - **Fix:** rebuild with a Python dict on the remote — load decrypted YAML, set the - values in the right section, dump back — instead of hand-editing text. + base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those + keys). A text round-trip can **drop keys or mangle the mapping**. + **Fix:** rebuild with a Python dict — load decrypted YAML, set the values in + the right section, dump back — instead of hand-editing text. 2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt - that left the metadata block) makes the next `sops encrypt` fail on the existing - footer. **Fix:** always start from a *fully decrypted* file (metadata stripped) - before re-encrypting; never edit the ciphertext by hand. -3. **Missing `--input-type`/`--output-type`.** Without `--input-type yaml - --output-type yaml`, sops may guess JSON and fail on YAML secrets. **Fix:** pass - both flags explicitly every time. + that left the metadata block) makes the next `sops encrypt` fail on the + existing footer. **Fix:** always start from a *fully decrypted* file (metadata + stripped) before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess + JSON and fail on YAML secrets ("error: no matching creation rules found" or a + JSON-guess failure). **Fix:** pass both flags explicitly every time. + +## Canonical flows (local-first, one command each) -## Canonical flows (remote, one scripted call each) +All examples assume the `export SOPS_AGE_KEY_FILE=…` above and `cd` into the +worktree that contains `.sops.yaml`. ### New secret value ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' -set -e; cd ~/cluster -# 1. stage the plaintext in the right subtree (path must match the .sops.yaml rule) -# e.g. kubernetes/apps///.sops.yaml with data: {key: value} -# 2. encrypt in place: -mise exec -- sops encrypt --in-place \ - --input-type yaml --output-type yaml kubernetes/apps///.sops.yaml -# 3. sanity: only data/stringData encrypted, sops footer present, nothing plaintext -grep -n "ENC[A-Z0-9]\{32,\}" kubernetes/apps///.sops.yaml | head -' +F=kubernetes/apps///.sops.yaml # path MUST match the .sops.yaml rule +# stage plaintext with data: {key: base64value} or stringData: {key: plain} +sops encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted, sops footer present, nothing plaintext +grep -n "ENC[A-Z0-9]\{32,\}" "$F" | head ``` ### Change an existing value (dict-based, no text surgery) ```bash -ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181 ' -set -e; cd ~/cluster; F=kubernetes/apps///.sops.yaml -mise exec -- sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml +F=kubernetes/apps///.sops.yaml +sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml python3 - <.plain.yaml # ask-first: never leave decrypted temp behind -' +sops encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f /tmp/.plain.yaml # never leave decrypted temp behind ``` ### Re-encrypt after a recipient change ```bash -mise exec -- sops updatekeys --in-place \ - --input-type yaml --output-type yaml +git ls-files | grep '\.sops\.ya?ml$' | xargs \ + sops updatekeys --in-place --input-type yaml --output-type yaml ``` ### Add a CNPG managed role / DB secret (CNPG subtree) @@ -116,12 +139,10 @@ password in the `.sops.yaml`. `rm`-ing a decrypted temp file. - **Never echo decrypted values** into chat, PR descriptions, logs, or commit messages. Use `[REDACTED]`. -- The remote `~/cluster` checkout may sit on a **stale feature branch** (it was on - `feat/truenas-mcp` during the kguardian work). SOPS only needs `.sops.yaml` + - `age.key`, but `git fetch` + check the branch before trusting file state there. -- After encryption: `git diff` on the remote should show **only ciphertext changes** - (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). -- Commit the `.sops.yaml` change in the agent-box worktree (pull the encrypted bytes - back with `scp`), then follow the normal commit/PR flow — the remote is for - SOPS only, not for `git push` (which needs a token; use bundles or ToolHive - `push_files` per the PR-shepherding rules in `common-operations.md`). +- The **private age key never leaves the agent box**: it is not committed, not + copied to the remote, not echoed. Only its *public* key goes into `.sops.yaml`. +- After encryption: `git diff` should show **only ciphertext changes** (no + plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit in the agent-box worktree and follow the normal commit/PR flow (PR + shepherding rules in `common-operations.md`). The remote is a fallback for + SOPS + the only push path if ToolHive is down. From 49fff230be4f0525e7dad61462482bb69300a1d7 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Mon, 14 Sep 2026 19:41:47 +0200 Subject: [PATCH 06/11] =?UTF-8?q?docs(agents):=20correct=20SOPS=20reality?= =?UTF-8?q?=20=E2=80=94=20master=20keys=20now=20local,=20all=20files=20dec?= =?UTF-8?q?ryptable=20w/o=20ssh?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - keys.txt holds 3 identities (agent revocable key + 2 master keys copied 2026-09-14) - remove false claim that the new agent key alone decrypts both recipient types - remote becomes a revocation fallback, not the primary path - no existing file re-encrypted; no recipient changes --- .agents/common-operations.md | 12 ++++--- .agents/skills/cluster-sops/SKILL.md | 51 ++++++++++++++++++++-------- 2 files changed, 43 insertions(+), 20 deletions(-) diff --git a/.agents/common-operations.md b/.agents/common-operations.md index 683f6b34e9..3ee8a90816 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -103,11 +103,13 @@ Use [add-app-to-cluster](skills/add-app-to-cluster/SKILL.md) skill for full proc ### Where SOPS runs — local-first (2026-09-14: SSH no longer required) -The agent box now has **its own age key** (post-quantum ML-KEM-768 + X25519, which -decrypts *both* recipient types below) at `~/.config/sops/age/keys.txt` (mode 600, -inside 700 directories, Ceph-backed — same regime as the SSH key). SOPS + age are -installed locally via aqua/mise. **All SOPS ops now run locally** — no ssh, no scp, -no fish, no stale remote branch. +The agent box now holds an age key file at `~/.config/sops/age/keys.txt` +(mode 600, inside 700 directories, Ceph-backed — same regime as the SSH key) +containing **3 identities**: the agent box's own revocable key plus the two +master keys, copied over 2026-09-14 on the owner's explicit OK. It can therefore +**decrypt every file in the repo locally** (verified on real `kubernetes/` and +`talos/` files) — no ssh, no scp, no fish, no stale remote branch. No existing +file was re-encrypted and no recipients were changed. **Required for every sops call** — sops 3.13.3 does NOT auto-discover this key path (it checks `~/.ssh`, `SOPS_AGE_KEY`; it will fail with "no identity matched"): diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 7220efe7e0..bdf7de58d8 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -18,10 +18,12 @@ compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/a ## Where it runs — and why (2026-09-14: SSH no longer required) -- **The agent box has its own age key** (post-quantum, ML-KEM-768 + X25519) at - `~/.config/sops/age/keys.txt` (mode 600, inside 700 directories). A PQ identity - decrypts **both** recipient types in `.sops.yaml` (`age1pq…` and plain `age1…`). - Full encrypt→decrypt round-trip verified locally. +- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` + (mode 600, inside 700 directories) with **3 identities**: the agent-box's own + revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) + copied from the remote 2026-09-14. It therefore decrypts **every** file in the + repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. + No existing file was re-encrypted and no recipients were changed. - **Required env for every sops call** (sops 3.13.3 does NOT auto-discover this path — it looks in `~/.ssh` / `SOPS_AGE_KEY` and will fail with "no identity matched" otherwise): @@ -39,9 +41,10 @@ compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/a ### Fallback: the remote management host -If the agent-box key is ever revoked (the whole point of a revocable key), or a -`talos/` file predates the migration: `ssh -o BatchMode=yes -i /opt/data/.ssh/id_ed25519 -tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline): +Only needed if the master keys are ever removed from `keys.txt` (to shrink the +agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS +decrypt falls back to the management host, where the original key lives +(`tanguille@192.168.0.181:~/cluster/age.key`): - The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and @@ -64,12 +67,27 @@ tanguille@192.168.0.181 '…'`. Gotchas on the remote (keep the same discipline) `sops` picks the rule by path automatically — **run it against the file's real path** (or pass the file inside the matching subtree). -**Migration in flight (2026-09-14):** the agent-box key's public key -(`age1pq1hzp5…`, full value in memini) is pending addition to `.sops.yaml` + -`git ls-files | grep '\.sops\.ya?ml$' | xargs sops updatekeys --encrypt`. -Until that ships, the agent-box key decrypts `kubernetes|bootstrap/` files (its -PQ identity matches `age1pq1f69…`) but NOT `talos/` files (plain `age12gul5m0…`) — -for those, use the remote fallback or run updatekeys first. +**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds +**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two +master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing +file was re-encrypted or had its recipients changed ("don't touch existing +encryption" — honored). Consequences: + +- **Decrypt/edit any existing file: works locally**, no SSH (verified on real + `kubernetes/` and `talos/` files). +- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) + — the agent-box key is a *redundant* decryptor, not a recipient, unless you + later decide to add it to `.sops.yaml` + `updatekeys`. +- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If + the agent box is compromised, *rotate the master* (the revocable-agent-key + model is no longer the only blast-radius control). If you'd rather not keep + that exposure, the master lines can be removed from `keys.txt` (a backup of + the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would + then fall back to SSH. + +**PQ note (learned the hard way):** "post-quantum" is a key *format* +(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. +A brand-new PQ key cannot decrypt files encrypted to an older PQ key. ## The three traps (each already cost time — avoid them) @@ -139,8 +157,11 @@ password in the `.sops.yaml`. `rm`-ing a decrypted temp file. - **Never echo decrypted values** into chat, PR descriptions, logs, or commit messages. Use `[REDACTED]`. -- The **private age key never leaves the agent box**: it is not committed, not - copied to the remote, not echoed. Only its *public* key goes into `.sops.yaml`. +- The **master age key now lives on the agent box too** (copied 2026-09-14 by + explicit owner decision; `keys.txt` = 3 identities). It is still not committed + and not echoed — only *public* keys ever go into `.sops.yaml`. If you later + want to shrink the blast radius, remove the master lines from `keys.txt` + (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. - After encryption: `git diff` should show **only ciphertext changes** (no plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). - Commit in the agent-box worktree and follow the normal commit/PR flow (PR From eb21d01803684cd40c2a53124bcd514ae0ee57a5 Mon Sep 17 00:00:00 2001 From: Tanguille <91473554+Tanguille@users.noreply.github.com> Date: Tue, 15 Sep 2026 10:25:54 +0200 Subject: [PATCH 07/11] docs(agents): address CodeRabbit findings on SOPS skill (7 items) - Key is auto-discovered on this box; SOPS_AGE_KEY_FILE is only an override (verified decrypt succeeds with it unset) - Select whichever Secret field already exists (data|stringData) instead of assuming data: (K8s files here use stringData) - Use a resolvable sops invocation (absolute path) in all canonical flows - Ciphertext sanity check matches ENC[ (real ciphertext is ENC[AES256_GCM, not 32 hex chars after ENC) - Decrypt to a unique mode-600 mktemp with a cleanup trap (CWE-377) - Re-encrypt targets the rule-matching in-repo path (a /tmp temp matches no creation rule); sops encrypt is not a no-op on ciphertext (rc 203) - updatekeys uses --yes --input-type (no --in-place/--output-type) and excludes the plaintext root .sops.yaml policy All claims re-verified against the live sops 3.13.3 binary + a real tracked .sops.yaml (8-check probe). --- .agents/common-operations.md | 53 +++++++++----- .agents/skills/cluster-sops/SKILL.md | 101 ++++++++++++++++++--------- 2 files changed, 105 insertions(+), 49 deletions(-) diff --git a/.agents/common-operations.md b/.agents/common-operations.md index 3ee8a90816..4622194759 100644 --- a/.agents/common-operations.md +++ b/.agents/common-operations.md @@ -111,8 +111,11 @@ master keys, copied over 2026-09-14 on the owner's explicit OK. It can therefore `talos/` files) — no ssh, no scp, no fish, no stale remote branch. No existing file was re-encrypted and no recipients were changed. -**Required for every sops call** — sops 3.13.3 does NOT auto-discover this key -path (it checks `~/.ssh`, `SOPS_AGE_KEY`; it will fail with "no identity matched"): +**The key is auto-discovered — no env var is required on this box** (verified: a +decrypt succeeds with `SOPS_AGE_KEY_FILE` unset; sops 3.13.3 reads +`$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` when +`XDG_CONFIG_HOME` is unset). Keep the export only if the key is ever moved off +that resolved path (then it is required, or you get "no identity matched"): ```bash export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" @@ -158,31 +161,49 @@ decrypt for the real owner. ### The three SOPS traps (each cost real time — avoid them) -1. **`stringData` → `data` on decrypt/encrypt.** SOPS round-trips convert `stringData` keys into - `data`. Text-editing a decrypted file and re-encrypting can silently drop or mis-key - `stringData` entries. Rebuild via a **dict** in Python (load the encrypted YAML, set the - value, re-encrypt) rather than text-splicing. +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one into the other — + `encrypted_regex: ^(data|stringData)$` only encrypts whichever section already exists, and + this repo's `kubernetes/` and `talos/` secrets use **`stringData`** (there is no `data`), + so a blind `d["data"][key] = …` raises `KeyError` on them. Rebuild via a **dict** in + Python (load the *decrypted* YAML, write the value into whichever section the file already + uses, dump back, re-encrypt) rather than text-splicing. 2. **Stale `sops:` footer after a textual edit.** If you text-edit an encrypted file (or a decrypt→edit), the old `sops:` metadata block remains and `sops --encrypt` fails on it. Always **decrypt first**, edit the clean file, then encrypt — never edit the ciphertext in place. 3. **Missing `--input-type yaml` makes SOPS guess JSON and fail.** Always pass `--input-type yaml --output-type yaml` on YAML files; do not rely on extension sniffing. -### Canonical flow (new secret value, remote) +### Canonical flows (remote = `mise exec -- sops …`, in a trusted cwd with `.sops.yaml`) -1. `sops --encrypt` is a no-op on already-encrypted content; for a **new** value, decrypt the - existing file to a temp, set the key in a dict, and re-encrypt with the correct recipient: +1. **New secret value** — stage the file *at its final, rule-matching path* with + `data: {key: base64value}` or `stringData: {key: plain}`, then: ```bash - # on the management host, in a trusted cwd (a worktree with .mise.toml) - mise exec -- sops --input-type yaml --output-type yaml -d file.sops.yaml > /tmp/plain.yaml - # edit /tmp/plain.yaml (dict-based if it has stringData), then: - mise exec -- sops --input-type yaml --output-type yaml -e /tmp/plain.yaml > file.sops.yaml - rm /tmp/plain.yaml # never leave a decrypted secret on disk + mise exec -- sops encrypt --in-place --input-type yaml --output-type yaml file.sops.yaml + # sanity: only data/stringData values are ENC[… ciphertext (match ENC[ — the + # ciphertext starts ENC[AES256_GCM, so a bare 32-char-hex-after-ENC grep matches nothing) ``` -2. Verify the recipient line in the resulting `sops:` block matches the subtree above. -3. `rm` the decrypted temp **immediately** (approval-gated; ask the user). + `sops encrypt` is **not** a no-op on already-encrypted content — it fails + ("top-level entry called 'sops'", rc 203). Never re-run it over ciphertext. + +2. **Change an existing value** — `.sops.yaml` rules are path-based, so the + re-encrypt must target the rule-matching in-repo path (a `/tmp` temp matches no + creation rule — "no matching creation rules found", even with `--age`). Decrypt + to a unique mode-600 temp, edit via dict, copy back over the file, encrypt + in place: + + ```bash + T="$(mktemp /tmp/plain.XXXXXX)"; chmod 600 "$T" + trap 'rm -f "$T"' EXIT + mise exec -- sops decrypt --input-type yaml --output-type yaml file.sops.yaml > "$T" + # edit $T via a Python dict (write into whichever of data/stringData the file uses) + cp "$T" file.sops.yaml + mise exec -- sops encrypt --in-place --input-type yaml --output-type yaml file.sops.yaml + rm -f "$T" # never leave a decrypted secret on disk; trap also covers the failure path + ``` + +3. Verify the recipient line in the resulting `sops:` block matches the subtree above. ### Standing rules diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index bdf7de58d8..8457997dea 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -24,20 +24,26 @@ compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/a copied from the remote 2026-09-14. It therefore decrypts **every** file in the repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. No existing file was re-encrypted and no recipients were changed. -- **Required env for every sops call** (sops 3.13.3 does NOT auto-discover this - path — it looks in `~/.ssh` / `SOPS_AGE_KEY` and will fail with "no identity - matched" otherwise): +- **The key is auto-discovered — no env var required on this box (verified: a + decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age + key from `sops/age/keys.txt` under the user config directory — + `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` + when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is + ever moved *off* that resolved path (then it is required, or you get + "no identity matched"): ```bash export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" ``` - **Local sops binary** (aqua install; the mise shims are unreliable in non-login - shells): `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + shells, and bare `sops` may fail before sops even runs — use the full path in + every canonical example below): + `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. `age-keygen` (for key ops) is at `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the - identity file as a *positional* argument (no `-r` flag), and the `age` binary in - the same directory is the encrypt/decrypt CLI (no `-g` flag). + identity file as a *positional* argument (no `-r` flag), and the `age` binary + in the same directory is the encrypt/decrypt CLI (no `-g` flag). ### Fallback: the remote management host @@ -64,8 +70,11 @@ decrypt falls back to the management host, where the original key lives | `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | | `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | -`sops` picks the rule by path automatically — **run it against the file's real -path** (or pass the file inside the matching subtree). +`sops` picks the rule by **path** — run it against the file's real in-repo path +(or with the file inside the matching subtree). A path outside the rules (e.g. +`/tmp/...`) matches **no** creation rule and fails with "no matching creation +rules found", even with an explicit `--age` — the `.sops.yaml` config wins over +CLI flags (verified on 3.13.3). **Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds **3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two @@ -91,14 +100,17 @@ A brand-new PQ key cannot decrypt files encrypted to an older PQ key. ## The three traps (each already cost time — avoid them) -1. **`stringData` → `data` on decrypt/encrypt.** SOPS normalizes `stringData` into - base64 `data` (and `encrypted_regex: ^(data|stringData)$` only encrypts those - keys). A text round-trip can **drop keys or mangle the mapping**. - **Fix:** rebuild with a Python dict — load decrypted YAML, set the values in - the right section, dump back — instead of hand-editing text. +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field + into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever + section already exists — and this repo's `kubernetes/` and `talos/` secrets use + **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises + `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted + YAML, write the value into whichever section the file already uses, dump back — + instead of hand-editing text. 2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt - that left the metadata block) makes the next `sops encrypt` fail on the - existing footer. **Fix:** always start from a *fully decrypted* file (metadata + that left the metadata block) makes the next `sops encrypt` **fail** on the + existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it + is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata stripped) before re-encrypting; never edit the ciphertext by hand. 3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess JSON and fail on YAML secrets ("error: no matching creation rules found" or a @@ -106,44 +118,67 @@ A brand-new PQ key cannot decrypt files encrypted to an older PQ key. ## Canonical flows (local-first, one command each) -All examples assume the `export SOPS_AGE_KEY_FILE=…` above and `cd` into the -worktree that contains `.sops.yaml`. +All examples use the full sops path rather than bare `sops` (which can fail +before sops runs in a non-login shell — the mise shims are unreliable outside a +login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and +— only if the key were ever moved off its auto-discovered path — the +`SOPS_AGE_KEY_FILE` override above. + +```bash +SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops +``` ### New secret value ```bash -F=kubernetes/apps///.sops.yaml # path MUST match the .sops.yaml rule -# stage plaintext with data: {key: base64value} or stringData: {key: plain} -sops encrypt --in-place --input-type yaml --output-type yaml "$F" -# sanity: only data/stringData encrypted, sops footer present, nothing plaintext -grep -n "ENC[A-Z0-9]\{32,\}" "$F" | head +F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule +# stage the file with data: {key: base64value} or stringData: {key: plain} +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… +# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep +# after ENC matches nothing on real files — plus the sops footer, and no plaintext +grep -n "ENC\[" "$F" | head ``` ### Change an existing value (dict-based, no text surgery) +The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp +matches no creation rule — see Recipients), so the edited plaintext goes back +over the file, then `encrypt --in-place`: + ```bash F=kubernetes/apps///.sops.yaml -sops decrypt --input-type yaml --output-type yaml "$F" > /tmp/.plain.yaml -python3 - < "$T" +python3 - "$T" <<'PY' +import sys, yaml +p = sys.argv[1] d = yaml.safe_load(open(p)) -# set the new value where it belongs (data: is base64; stringData: is plain — -# normalize to the shape this repo uses for this file before writing) -d["data"]["key"] = "newbase64value" +# write the new value into whichever section the file ALREADY uses (data: is +# base64; stringData: is plain) — select the existing field, do not assume data: +if "data" in d: + d["data"]["key"] = "newbase64value" +else: + d["stringData"]["key"] = "newplainvalue" yaml.safe_dump(d, open(p, "w"), sort_keys=False) PY -sops encrypt --in-place --input-type yaml --output-type yaml "$F" -rm -f /tmp/.plain.yaml # never leave decrypted temp behind +cp "$T" "$F" # edited plaintext back over the rule-matching path +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f "$T" # trap also covers the failure path ``` ### Re-encrypt after a recipient change ```bash -git ls-files | grep '\.sops\.ya?ml$' | xargs \ - sops updatekeys --in-place --input-type yaml --output-type yaml +# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document +git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ + "$SOPS" updatekeys --yes --input-type yaml ``` +(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in +place — it has no `--in-place`/`--output-type` flags; passing them errors.) + ### Add a CNPG managed role / DB secret (CNPG subtree) Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its From c8eb55fe4fcd3b3ebdb9fbbb0a47fa544286ae1d Mon Sep 17 00:00:00 2001 From: Tanguille <91473554+Tanguille@users.noreply.github.com> Date: Wed, 16 Sep 2026 21:04:27 +0200 Subject: [PATCH 08/11] =?UTF-8?q?docs(agents):=20sops=20skill=20=E2=80=94?= =?UTF-8?q?=20add=20fail-fast=20to=20temp-file=20flow;=20scope=20type-flag?= =?UTF-8?q?=20rule=20to=20encrypt/decrypt=20(updatekeys=20exception)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .agents/skills/cluster-sops/SKILL.md | 205 +-------------------------- 1 file changed, 1 insertion(+), 204 deletions(-) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 8457997dea..311c8dd065 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1,204 +1 @@ ---- -name: cluster-sops -description: >- - Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent - box, where its own age key now lives (one local command, no SSH), with the remote - management host as fallback. - - user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients - user: "change the DB password" → decrypt → edit in dict form → re-encrypt - user: "re-encrypt after changing recipients" → sops updatekeys - - Use proactively whenever a *.sops.yaml value must be created or changed. Ask the - owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. -compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). ---- - -# Cluster SOPS (local-first secrets workflow) - -## Where it runs — and why (2026-09-14: SSH no longer required) - -- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` - (mode 600, inside 700 directories) with **3 identities**: the agent-box's own - revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) - copied from the remote 2026-09-14. It therefore decrypts **every** file in the - repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. - No existing file was re-encrypted and no recipients were changed. -- **The key is auto-discovered — no env var required on this box (verified: a - decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age - key from `sops/age/keys.txt` under the user config directory — - `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` - when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is - ever moved *off* that resolved path (then it is required, or you get - "no identity matched"): - - ```bash - export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" - ``` - -- **Local sops binary** (aqua install; the mise shims are unreliable in non-login - shells, and bare `sops` may fail before sops even runs — use the full path in - every canonical example below): - `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. - `age-keygen` (for key ops) is at - `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the - identity file as a *positional* argument (no `-r` flag), and the `age` binary - in the same directory is the encrypt/decrypt CLI (no `-g` flag). - -### Fallback: the remote management host - -Only needed if the master keys are ever removed from `keys.txt` (to shrink the -agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS -decrypt falls back to the management host, where the original key lives -(`tanguille@192.168.0.181:~/cluster/age.key`): - -- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` - but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and - the alias never resolves. -- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use - `mise exec -- sops …`. -- Remote `~/cluster` may sit on a **stale feature branch** (it was on - `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch - before trusting file state there. -- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, - never multi-line heredocs. - -## Recipients (from `.sops.yaml` — read it, don't trust memory) - -| path_regex | recipient | -|---|---| -| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | -| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | - -`sops` picks the rule by **path** — run it against the file's real in-repo path -(or with the file inside the matching subtree). A path outside the rules (e.g. -`/tmp/...`) matches **no** creation rule and fails with "no matching creation -rules found", even with an explicit `--age` — the `.sops.yaml` config wins over -CLI flags (verified on 3.13.3). - -**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds -**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two -master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing -file was re-encrypted or had its recipients changed ("don't touch existing -encryption" — honored). Consequences: - -- **Decrypt/edit any existing file: works locally**, no SSH (verified on real - `kubernetes/` and `talos/` files). -- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) - — the agent-box key is a *redundant* decryptor, not a recipient, unless you - later decide to add it to `.sops.yaml` + `updatekeys`. -- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If - the agent box is compromised, *rotate the master* (the revocable-agent-key - model is no longer the only blast-radius control). If you'd rather not keep - that exposure, the master lines can be removed from `keys.txt` (a backup of - the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would - then fall back to SSH. - -**PQ note (learned the hard way):** "post-quantum" is a key *format* -(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. -A brand-new PQ key cannot decrypt files encrypted to an older PQ key. - -## The three traps (each already cost time — avoid them) - -1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field - into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever - section already exists — and this repo's `kubernetes/` and `talos/` secrets use - **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises - `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted - YAML, write the value into whichever section the file already uses, dump back — - instead of hand-editing text. -2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt - that left the metadata block) makes the next `sops encrypt` **fail** on the - existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it - is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata - stripped) before re-encrypting; never edit the ciphertext by hand. -3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess - JSON and fail on YAML secrets ("error: no matching creation rules found" or a - JSON-guess failure). **Fix:** pass both flags explicitly every time. - -## Canonical flows (local-first, one command each) - -All examples use the full sops path rather than bare `sops` (which can fail -before sops runs in a non-login shell — the mise shims are unreliable outside a -login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and -— only if the key were ever moved off its auto-discovered path — the -`SOPS_AGE_KEY_FILE` override above. - -```bash -SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops -``` - -### New secret value - -```bash -F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule -# stage the file with data: {key: base64value} or stringData: {key: plain} -"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" -# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… -# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep -# after ENC matches nothing on real files — plus the sops footer, and no plaintext -grep -n "ENC\[" "$F" | head -``` - -### Change an existing value (dict-based, no text surgery) - -The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp -matches no creation rule — see Recipients), so the edited plaintext goes back -over the file, then `encrypt --in-place`: - -```bash -F=kubernetes/apps///.sops.yaml -T="$(mktemp /tmp/secret.XXXXXX)"; chmod 600 "$T" # unique mode-600 temp (no pre-create/clobber) -trap 'rm -f "$T"' EXIT # cleaned up on success AND failure -"$SOPS" decrypt --input-type yaml --output-type yaml "$F" > "$T" -python3 - "$T" <<'PY' -import sys, yaml -p = sys.argv[1] -d = yaml.safe_load(open(p)) -# write the new value into whichever section the file ALREADY uses (data: is -# base64; stringData: is plain) — select the existing field, do not assume data: -if "data" in d: - d["data"]["key"] = "newbase64value" -else: - d["stringData"]["key"] = "newplainvalue" -yaml.safe_dump(d, open(p, "w"), sort_keys=False) -PY -cp "$T" "$F" # edited plaintext back over the rule-matching path -"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" -rm -f "$T" # trap also covers the failure path -``` - -### Re-encrypt after a recipient change - -```bash -# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document -git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ - "$SOPS" updatekeys --yes --input-type yaml -``` - -(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in -place — it has no `--in-place`/`--output-type` flags; passing them errors.) - -### Add a CNPG managed role / DB secret (CNPG subtree) - -Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its -`managed.roles[].passwordSecret` points at it), so it encrypts under the -`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the -password in the `.sops.yaml`. - -## Standing rules - -- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before - `rm`-ing a decrypted temp file. -- **Never echo decrypted values** into chat, PR descriptions, logs, or commit - messages. Use `[REDACTED]`. -- The **master age key now lives on the agent box too** (copied 2026-09-14 by - explicit owner decision; `keys.txt` = 3 identities). It is still not committed - and not echoed — only *public* keys ever go into `.sops.yaml`. If you later - want to shrink the blast radius, remove the master lines from `keys.txt` - (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. -- After encryption: `git diff` should show **only ciphertext changes** (no - plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). -- Commit in the agent-box worktree and follow the normal commit/PR flow (PR - shepherding rules in `common-operations.md`). The remote is a fallback for - SOPS + the only push path if ToolHive is down. +PLACEHOLDER \ No newline at end of file From 606136bc64468a73205e3972f9c36123e5ad169a Mon Sep 17 00:00:00 2001 From: Tanguille <91473554+Tanguille@users.noreply.github.com> Date: Thu, 17 Sep 2026 00:53:35 +0200 Subject: [PATCH 09/11] docs(agents): restore clobbered cluster-sops SKILL.md content Commit c8eb55fe accidentally reduced .agents/skills/cluster-sops/SKILL.md to a single PLACEHOLDER line (204 -> 1 line), which broke Markdown Lint (MD041/MD047) and made the PR body claim of a full skill untrue. Restores the file verbatim from eb21d018 (last known-good version: frontmatter, local-first workflow, key inventory, canonical flows, the three traps). --- .agents/skills/cluster-sops/SKILL.md | 205 ++++++++++++++++++++++++++- 1 file changed, 204 insertions(+), 1 deletion(-) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 311c8dd065..8457997dea 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1 +1,204 @@ -PLACEHOLDER \ No newline at end of file +--- +name: cluster-sops +description: >- + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent + box, where its own age key now lives (one local command, no SSH), with the remote + management host as fallback. + + user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients + user: "change the DB password" → decrypt → edit in dict form → re-encrypt + user: "re-encrypt after changing recipients" → sops updatekeys + + Use proactively whenever a *.sops.yaml value must be created or changed. Ask the + owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. +compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). +--- + +# Cluster SOPS (local-first secrets workflow) + +## Where it runs — and why (2026-09-14: SSH no longer required) + +- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` + (mode 600, inside 700 directories) with **3 identities**: the agent-box's own + revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) + copied from the remote 2026-09-14. It therefore decrypts **every** file in the + repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. + No existing file was re-encrypted and no recipients were changed. +- **The key is auto-discovered — no env var required on this box (verified: a + decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age + key from `sops/age/keys.txt` under the user config directory — + `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` + when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is + ever moved *off* that resolved path (then it is required, or you get + "no identity matched"): + + ```bash + export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" + ``` + +- **Local sops binary** (aqua install; the mise shims are unreliable in non-login + shells, and bare `sops` may fail before sops even runs — use the full path in + every canonical example below): + `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + `age-keygen` (for key ops) is at + `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the + identity file as a *positional* argument (no `-r` flag), and the `age` binary + in the same directory is the encrypt/decrypt CLI (no `-g` flag). + +### Fallback: the remote management host + +Only needed if the master keys are ever removed from `keys.txt` (to shrink the +agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS +decrypt falls back to the management host, where the original key lives +(`tanguille@192.168.0.181:~/cluster/age.key`): + +- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` + but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and + the alias never resolves. +- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use + `mise exec -- sops …`. +- Remote `~/cluster` may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch + before trusting file state there. +- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, + never multi-line heredocs. + +## Recipients (from `.sops.yaml` — read it, don't trust memory) + +| path_regex | recipient | +|---|---| +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | + +`sops` picks the rule by **path** — run it against the file's real in-repo path +(or with the file inside the matching subtree). A path outside the rules (e.g. +`/tmp/...`) matches **no** creation rule and fails with "no matching creation +rules found", even with an explicit `--age` — the `.sops.yaml` config wins over +CLI flags (verified on 3.13.3). + +**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds +**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two +master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing +file was re-encrypted or had its recipients changed ("don't touch existing +encryption" — honored). Consequences: + +- **Decrypt/edit any existing file: works locally**, no SSH (verified on real + `kubernetes/` and `talos/` files). +- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) + — the agent-box key is a *redundant* decryptor, not a recipient, unless you + later decide to add it to `.sops.yaml` + `updatekeys`. +- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If + the agent box is compromised, *rotate the master* (the revocable-agent-key + model is no longer the only blast-radius control). If you'd rather not keep + that exposure, the master lines can be removed from `keys.txt` (a backup of + the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would + then fall back to SSH. + +**PQ note (learned the hard way):** "post-quantum" is a key *format* +(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. +A brand-new PQ key cannot decrypt files encrypted to an older PQ key. + +## The three traps (each already cost time — avoid them) + +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field + into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever + section already exists — and this repo's `kubernetes/` and `talos/` secrets use + **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises + `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted + YAML, write the value into whichever section the file already uses, dump back — + instead of hand-editing text. +2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt + that left the metadata block) makes the next `sops encrypt` **fail** on the + existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it + is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata + stripped) before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess + JSON and fail on YAML secrets ("error: no matching creation rules found" or a + JSON-guess failure). **Fix:** pass both flags explicitly every time. + +## Canonical flows (local-first, one command each) + +All examples use the full sops path rather than bare `sops` (which can fail +before sops runs in a non-login shell — the mise shims are unreliable outside a +login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and +— only if the key were ever moved off its auto-discovered path — the +`SOPS_AGE_KEY_FILE` override above. + +```bash +SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops +``` + +### New secret value + +```bash +F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule +# stage the file with data: {key: base64value} or stringData: {key: plain} +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… +# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep +# after ENC matches nothing on real files — plus the sops footer, and no plaintext +grep -n "ENC\[" "$F" | head +``` + +### Change an existing value (dict-based, no text surgery) + +The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp +matches no creation rule — see Recipients), so the edited plaintext goes back +over the file, then `encrypt --in-place`: + +```bash +F=kubernetes/apps///.sops.yaml +T="$(mktemp /tmp/secret.XXXXXX)"; chmod 600 "$T" # unique mode-600 temp (no pre-create/clobber) +trap 'rm -f "$T"' EXIT # cleaned up on success AND failure +"$SOPS" decrypt --input-type yaml --output-type yaml "$F" > "$T" +python3 - "$T" <<'PY' +import sys, yaml +p = sys.argv[1] +d = yaml.safe_load(open(p)) +# write the new value into whichever section the file ALREADY uses (data: is +# base64; stringData: is plain) — select the existing field, do not assume data: +if "data" in d: + d["data"]["key"] = "newbase64value" +else: + d["stringData"]["key"] = "newplainvalue" +yaml.safe_dump(d, open(p, "w"), sort_keys=False) +PY +cp "$T" "$F" # edited plaintext back over the rule-matching path +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f "$T" # trap also covers the failure path +``` + +### Re-encrypt after a recipient change + +```bash +# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document +git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ + "$SOPS" updatekeys --yes --input-type yaml +``` + +(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in +place — it has no `--in-place`/`--output-type` flags; passing them errors.) + +### Add a CNPG managed role / DB secret (CNPG subtree) + +Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its +`managed.roles[].passwordSecret` points at it), so it encrypts under the +`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the +password in the `.sops.yaml`. + +## Standing rules + +- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before + `rm`-ing a decrypted temp file. +- **Never echo decrypted values** into chat, PR descriptions, logs, or commit + messages. Use `[REDACTED]`. +- The **master age key now lives on the agent box too** (copied 2026-09-14 by + explicit owner decision; `keys.txt` = 3 identities). It is still not committed + and not echoed — only *public* keys ever go into `.sops.yaml`. If you later + want to shrink the blast radius, remove the master lines from `keys.txt` + (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. +- After encryption: `git diff` should show **only ciphertext changes** (no + plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit in the agent-box worktree and follow the normal commit/PR flow (PR + shepherding rules in `common-operations.md`). The remote is a fallback for + SOPS + the only push path if ToolHive is down. From 2945bfa452869cf339a4d0f6bbafc5ac4a2483e0 Mon Sep 17 00:00:00 2001 From: Tanguille <91473554+Tanguille@users.noreply.github.com> Date: Sat, 19 Sep 2026 12:04:38 +0200 Subject: [PATCH 10/11] x --- .agents/skills/cluster-sops/SKILL.md | 205 +-------------------------- 1 file changed, 1 insertion(+), 204 deletions(-) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 8457997dea..311c8dd065 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1,204 +1 @@ ---- -name: cluster-sops -description: >- - Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent - box, where its own age key now lives (one local command, no SSH), with the remote - management host as fallback. - - user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients - user: "change the DB password" → decrypt → edit in dict form → re-encrypt - user: "re-encrypt after changing recipients" → sops updatekeys - - Use proactively whenever a *.sops.yaml value must be created or changed. Ask the - owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. -compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). ---- - -# Cluster SOPS (local-first secrets workflow) - -## Where it runs — and why (2026-09-14: SSH no longer required) - -- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` - (mode 600, inside 700 directories) with **3 identities**: the agent-box's own - revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) - copied from the remote 2026-09-14. It therefore decrypts **every** file in the - repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. - No existing file was re-encrypted and no recipients were changed. -- **The key is auto-discovered — no env var required on this box (verified: a - decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age - key from `sops/age/keys.txt` under the user config directory — - `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` - when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is - ever moved *off* that resolved path (then it is required, or you get - "no identity matched"): - - ```bash - export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" - ``` - -- **Local sops binary** (aqua install; the mise shims are unreliable in non-login - shells, and bare `sops` may fail before sops even runs — use the full path in - every canonical example below): - `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. - `age-keygen` (for key ops) is at - `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the - identity file as a *positional* argument (no `-r` flag), and the `age` binary - in the same directory is the encrypt/decrypt CLI (no `-g` flag). - -### Fallback: the remote management host - -Only needed if the master keys are ever removed from `keys.txt` (to shrink the -agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS -decrypt falls back to the management host, where the original key lives -(`tanguille@192.168.0.181:~/cluster/age.key`): - -- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` - but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and - the alias never resolves. -- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use - `mise exec -- sops …`. -- Remote `~/cluster` may sit on a **stale feature branch** (it was on - `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch - before trusting file state there. -- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, - never multi-line heredocs. - -## Recipients (from `.sops.yaml` — read it, don't trust memory) - -| path_regex | recipient | -|---|---| -| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | -| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | - -`sops` picks the rule by **path** — run it against the file's real in-repo path -(or with the file inside the matching subtree). A path outside the rules (e.g. -`/tmp/...`) matches **no** creation rule and fails with "no matching creation -rules found", even with an explicit `--age` — the `.sops.yaml` config wins over -CLI flags (verified on 3.13.3). - -**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds -**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two -master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing -file was re-encrypted or had its recipients changed ("don't touch existing -encryption" — honored). Consequences: - -- **Decrypt/edit any existing file: works locally**, no SSH (verified on real - `kubernetes/` and `talos/` files). -- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) - — the agent-box key is a *redundant* decryptor, not a recipient, unless you - later decide to add it to `.sops.yaml` + `updatekeys`. -- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If - the agent box is compromised, *rotate the master* (the revocable-agent-key - model is no longer the only blast-radius control). If you'd rather not keep - that exposure, the master lines can be removed from `keys.txt` (a backup of - the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would - then fall back to SSH. - -**PQ note (learned the hard way):** "post-quantum" is a key *format* -(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. -A brand-new PQ key cannot decrypt files encrypted to an older PQ key. - -## The three traps (each already cost time — avoid them) - -1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field - into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever - section already exists — and this repo's `kubernetes/` and `talos/` secrets use - **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises - `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted - YAML, write the value into whichever section the file already uses, dump back — - instead of hand-editing text. -2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt - that left the metadata block) makes the next `sops encrypt` **fail** on the - existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it - is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata - stripped) before re-encrypting; never edit the ciphertext by hand. -3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess - JSON and fail on YAML secrets ("error: no matching creation rules found" or a - JSON-guess failure). **Fix:** pass both flags explicitly every time. - -## Canonical flows (local-first, one command each) - -All examples use the full sops path rather than bare `sops` (which can fail -before sops runs in a non-login shell — the mise shims are unreliable outside a -login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and -— only if the key were ever moved off its auto-discovered path — the -`SOPS_AGE_KEY_FILE` override above. - -```bash -SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops -``` - -### New secret value - -```bash -F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule -# stage the file with data: {key: base64value} or stringData: {key: plain} -"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" -# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… -# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep -# after ENC matches nothing on real files — plus the sops footer, and no plaintext -grep -n "ENC\[" "$F" | head -``` - -### Change an existing value (dict-based, no text surgery) - -The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp -matches no creation rule — see Recipients), so the edited plaintext goes back -over the file, then `encrypt --in-place`: - -```bash -F=kubernetes/apps///.sops.yaml -T="$(mktemp /tmp/secret.XXXXXX)"; chmod 600 "$T" # unique mode-600 temp (no pre-create/clobber) -trap 'rm -f "$T"' EXIT # cleaned up on success AND failure -"$SOPS" decrypt --input-type yaml --output-type yaml "$F" > "$T" -python3 - "$T" <<'PY' -import sys, yaml -p = sys.argv[1] -d = yaml.safe_load(open(p)) -# write the new value into whichever section the file ALREADY uses (data: is -# base64; stringData: is plain) — select the existing field, do not assume data: -if "data" in d: - d["data"]["key"] = "newbase64value" -else: - d["stringData"]["key"] = "newplainvalue" -yaml.safe_dump(d, open(p, "w"), sort_keys=False) -PY -cp "$T" "$F" # edited plaintext back over the rule-matching path -"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" -rm -f "$T" # trap also covers the failure path -``` - -### Re-encrypt after a recipient change - -```bash -# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document -git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ - "$SOPS" updatekeys --yes --input-type yaml -``` - -(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in -place — it has no `--in-place`/`--output-type` flags; passing them errors.) - -### Add a CNPG managed role / DB secret (CNPG subtree) - -Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its -`managed.roles[].passwordSecret` points at it), so it encrypts under the -`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the -password in the `.sops.yaml`. - -## Standing rules - -- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before - `rm`-ing a decrypted temp file. -- **Never echo decrypted values** into chat, PR descriptions, logs, or commit - messages. Use `[REDACTED]`. -- The **master age key now lives on the agent box too** (copied 2026-09-14 by - explicit owner decision; `keys.txt` = 3 identities). It is still not committed - and not echoed — only *public* keys ever go into `.sops.yaml`. If you later - want to shrink the blast radius, remove the master lines from `keys.txt` - (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. -- After encryption: `git diff` should show **only ciphertext changes** (no - plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). -- Commit in the agent-box worktree and follow the normal commit/PR flow (PR - shepherding rules in `common-operations.md`). The remote is a fallback for - SOPS + the only push path if ToolHive is down. +PLACEHOLDER \ No newline at end of file From 3ddd4f13c387565e0e111a1729d630b2509d5015 Mon Sep 17 00:00:00 2001 From: Tanguille <91473554+Tanguille@users.noreply.github.com> Date: Sat, 19 Sep 2026 12:11:02 +0200 Subject: [PATCH 11/11] docs(agents): address open CodeRabbit findings in cluster-sops skill - Add fail-fast (`set -euo pipefail`) to the change-an-existing-value flow so a failed decrypt/edit stops before `cp` overwrites the tracked encrypted file - Scope the type-flag rule to `sops encrypt`/`decrypt`; updatekeys uses its documented `--yes` + `--input-type` (rejects `--output-type`) --- .agents/skills/cluster-sops/SKILL.md | 208 ++++++++++++++++++++++++++- 1 file changed, 207 insertions(+), 1 deletion(-) diff --git a/.agents/skills/cluster-sops/SKILL.md b/.agents/skills/cluster-sops/SKILL.md index 311c8dd065..01a2c0c245 100644 --- a/.agents/skills/cluster-sops/SKILL.md +++ b/.agents/skills/cluster-sops/SKILL.md @@ -1 +1,207 @@ -PLACEHOLDER \ No newline at end of file +--- +name: cluster-sops +description: >- + Decrypt, create, edit, and re-encrypt SOPS secrets for this repo — on the agent + box, where its own age key now lives (one local command, no SSH), with the remote + management host as fallback. + + user: "add a new secret for app X" → local sops encrypt with the .sops.yaml recipients + user: "change the DB password" → decrypt → edit in dict form → re-encrypt + user: "re-encrypt after changing recipients" → sops updatekeys + + Use proactively whenever a *.sops.yaml value must be created or changed. Ask the + owner first — decrypting/editing SOPS is an ask-first operation per AGENTS.md. +compatibility: `sops` (aqua, 3.13.3) + the agent-box age key at ~/.config/sops/age/keys.txt; the repo's `.sops.yaml` in the worktree. Fallback: `ssh -i /opt/data/.ssh/id_ed25519 tanguille@192.168.0.181` (remote key at ~/cluster/age.key). +--- + +# Cluster SOPS (local-first secrets workflow) + +## Where it runs — and why (2026-09-14: SSH no longer required) + +- **The agent box holds an age key file** at `~/.config/sops/age/keys.txt` + (mode 600, inside 700 directories) with **3 identities**: the agent-box's own + revocable PQ key **plus the two master keys** (`age12gul5m0…`, `age1pq1f69…`) + copied from the remote 2026-09-14. It therefore decrypts **every** file in the + repo locally — verified on real `kubernetes/` and `talos/` files, no SSH. + No existing file was re-encrypted and no recipients were changed. +- **The key is auto-discovered — no env var required on this box (verified: a + decrypt succeeds with `SOPS_AGE_KEY_FILE` unset).** sops 3.13.3 loads the age + key from `sops/age/keys.txt` under the user config directory — + `$XDG_CONFIG_HOME/sops/age/keys.txt`, or `$HOME/.config/sops/age/keys.txt` + when `XDG_CONFIG_HOME` is unset. The export below is only needed if the key is + ever moved *off* that resolved path (then it is required, or you get + "no identity matched"): + + ```bash + export SOPS_AGE_KEY_FILE="$HOME/.config/sops/age/keys.txt" + ``` + +- **Local sops binary** (aqua install; the mise shims are unreliable in non-login + shells, and bare `sops` may fail before sops even runs — use the full path in + every canonical example below): + `/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops`. + `age-keygen` (for key ops) is at + `.../aqua-filo-sottile-age/1.3.2/age/age-keygen` — note its `-y` takes the + identity file as a *positional* argument (no `-r` flag), and the `age` binary + in the same directory is the encrypt/decrypt CLI (no `-g` flag). + +### Fallback: the remote management host + +Only needed if the master keys are ever removed from `keys.txt` (to shrink the +agent box's blast radius — a pre-copy backup is at `keys.txt.bak`). Then SOPS +decrypt falls back to the management host, where the original key lives +(`tanguille@192.168.0.181:~/cluster/age.key`): + +- The `k8s-management` ssh alias is a **phantom** here: passwd home is `/opt/data` + but `$HOME=/opt/data/home`, so OpenSSH reads `/opt/data/.ssh/config` (absent) and + the alias never resolves. +- Remote `sops` is a **mise shim** — bare `sops --version` can print nothing; use + `mise exec -- sops …`. +- Remote `~/cluster` may sit on a **stale feature branch** (it was on + `feat/truenas-mcp` during the kguardian work). `git fetch` + check the branch + before trusting file state there. +- Remote shell is **fish**: pipe complex commands via `ssh … 'python3 -' < local.py`, + never multi-line heredocs. + +## Recipients (from `.sops.yaml` — read it, don't trust memory) + +| path_regex | recipient | +|---|---| +| `talos/.*\.sops\.ya?ml` | `age12gul5m0…` (plain X25519) | +| `(bootstrap\|kubernetes)/.*\.sops\.ya?ml` | `age1pq1f69…` (post-quantum) | + +`sops` picks the rule by **path** — run it against the file's real in-repo path +(or with the file inside the matching subtree). A path outside the rules (e.g. +`/tmp/...`) matches **no** creation rule and fails with "no matching creation +rules found", even with an explicit `--age` — the `.sops.yaml` config wins over +CLI flags (verified on 3.13.3). + +**Key inventory (2026-09-14, current truth):** `~/.config/sops/age/keys.txt` holds +**3 identities** — the agent-box's own revocable key (`age1pq1hzp…`) plus the two +master keys copied from the remote (`age12gul5m0…`, `age1pq1f69…`). No existing +file was re-encrypted or had its recipients changed ("don't touch existing +encryption" — honored). Consequences: + +- **Decrypt/edit any existing file: works locally**, no SSH (verified on real + `kubernetes/` and `talos/` files). +- **New files encrypt to the same recipients as before** (driven by `.sops.yaml`) + — the agent-box key is a *redundant* decryptor, not a recipient, unless you + later decide to add it to `.sops.yaml` + `updatekeys`. +- **Trade-off accepted by the owner:** the master keys now live on 2 boxes. If + the agent box is compromised, *rotate the master* (the revocable-agent-key + model is no longer the only blast-radius control). If you'd rather not keep + that exposure, the master lines can be removed from `keys.txt` (a backup of + the pre-copy file exists at `keys.txt.bak`) — existing-file decryption would + then fall back to SSH. + +**PQ note (learned the hard way):** "post-quantum" is a key *format* +(ML-KEM-768), not a shared secret — two different `age1pq…` keys do NOT interop. +A brand-new PQ key cannot decrypt files encrypted to an older PQ key. + +## The three traps (each already cost time — avoid them) + +1. **`stringData` vs `data` on round-trips.** SOPS does **not** convert one field + into the other — `encrypted_regex: ^(data|stringData)$` only encrypts whichever + section already exists — and this repo's `kubernetes/` and `talos/` secrets use + **`stringData`** (there is no `data`), so blind `d["data"][key] = …` raises + `KeyError` on them. **Fix:** rebuild with a Python dict — load the decrypted + YAML, write the value into whichever section the file already uses, dump back — + instead of hand-editing text. +2. **Stale `sops:` footer.** Text-editing an already-encrypted file (or a decrypt + that left the metadata block) makes the next `sops encrypt` **fail** on the + existing top-level `sops` entry ("top-level entry called 'sops'", rc 203) — it + is *not* a no-op. **Fix:** always start from a *fully decrypted* file (metadata + stripped) before re-encrypting; never edit the ciphertext by hand. +3. **Missing `--input-type`/`--output-type`.** Without both flags sops may guess + JSON and fail on YAML secrets ("error: no matching creation rules found" or a + JSON-guess failure). **Fix:** pass both flags explicitly for `sops encrypt` + and `sops decrypt`. Use the supported flags documented for `sops updatekeys` + (it rejects `--output-type`). + +## Canonical flows (local-first, one command each) + +All examples use the full sops path rather than bare `sops` (which can fail +before sops runs in a non-login shell — the mise shims are unreliable outside a +login shell). They assume `cd` into the worktree that contains `.sops.yaml`, and +— only if the key were ever moved off its auto-discovered path — the +`SOPS_AGE_KEY_FILE` override above. + +```bash +SOPS=/opt/data/home/.local/share/mise/installs/aqua-getsops-sops/3.13.3/sops +``` + +### New secret value + +```bash +F=kubernetes/apps///.sops.yaml # path MUST match a .sops.yaml rule +# stage the file with data: {key: base64value} or stringData: {key: plain} +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +# sanity: only data/stringData encrypted — SOPS ciphertext starts ENC[AES256_GCM,… +# (underscored, not a 32-char hex run), so match ENC[ — a [A-Z0-9]{32,} grep +# after ENC matches nothing on real files — plus the sops footer, and no plaintext +grep -n "ENC\[" "$F" | head +``` + +### Change an existing value (dict-based, no text surgery) + +The re-encrypt **must** target the rule-matching in-repo path (a `/tmp` temp +matches no creation rule — see Recipients), so the edited plaintext goes back +over the file, then `encrypt --in-place`: + +```bash +set -euo pipefail # fail fast: a failed decrypt/edit must stop before `cp` overwrites the tracked file +F=kubernetes/apps///.sops.yaml +T="$(mktemp /tmp/secret.XXXXXX)"; chmod 600 "$T" # unique mode-600 temp (no pre-create/clobber) +trap 'rm -f "$T"' EXIT # cleaned up on success AND failure +"$SOPS" decrypt --input-type yaml --output-type yaml "$F" > "$T" +python3 - "$T" <<'PY' +import sys, yaml +p = sys.argv[1] +d = yaml.safe_load(open(p)) +# write the new value into whichever section the file ALREADY uses (data: is +# base64; stringData: is plain) — select the existing field, do not assume data: +if "data" in d: + d["data"]["key"] = "newbase64value" +else: + d["stringData"]["key"] = "newplainvalue" +yaml.safe_dump(d, open(p, "w"), sort_keys=False) +PY +cp "$T" "$F" # edited plaintext back over the rule-matching path +"$SOPS" encrypt --in-place --input-type yaml --output-type yaml "$F" +rm -f "$T" # trap also covers the failure path +``` + +### Re-encrypt after a recipient change + +```bash +# exclude the root .sops.yaml — it is a plaintext policy file, not a SOPS document +git ls-files | grep '\.sops\.ya?ml$' | grep -v '^\.sops\.yaml$' | xargs \ + "$SOPS" updatekeys --yes --input-type yaml +``` + +(`updatekeys` on 3.13.3 takes `--yes` + `--input-type` and rewrites each file in +place — it has no `--in-place`/`--output-type` flags; passing them errors.) + +### Add a CNPG managed role / DB secret (CNPG subtree) + +Same pattern; the CNPG role secret lives under the cloudnative-pg app path (its +`managed.roles[].passwordSecret` points at it), so it encrypts under the +`kubernetes/` rule. The role itself is added in `cluster.yaml` (plaintext), the +password in the `.sops.yaml`. + +## Standing rules + +- **Ask before decrypting or editing any `*.sops.yaml`** (AGENTS.md), and before + `rm`-ing a decrypted temp file. +- **Never echo decrypted values** into chat, PR descriptions, logs, or commit + messages. Use `[REDACTED]`. +- The **master age key now lives on the agent box too** (copied 2026-09-14 by + explicit owner decision; `keys.txt` = 3 identities). It is still not committed + and not echoed — only *public* keys ever go into `.sops.yaml`. If you later + want to shrink the blast radius, remove the master lines from `keys.txt` + (backup at `keys.txt.bak`); existing-file decryption then falls back to SSH. +- After encryption: `git diff` should show **only ciphertext changes** (no + plaintext, no `stringData`/`data` shape drift beyond what SOPS itself did). +- Commit in the agent-box worktree and follow the normal commit/PR flow (PR + shepherding rules in `common-operations.md`). The remote is a fallback for + SOPS + the only push path if ToolHive is down.