feat(secrets): add optional SOPS age workflow#24
Conversation
- Deno 2.x project structure with deno.json and task definitions - JSR dependencies: @cliffy/command, @std/assert, @std/testing, @std/yaml, @std/dotenv, @std/fs, @std/path - Full CLI command tree with stubs for all 15 issues - Shared interfaces (ProcessRunner, config types, ExitCode) for parallel work - FakeProcessRunner with recording, pre-programmed responses, and dry-run support - CI pipeline: fmt, lint, typecheck, test, coverage, and cross-platform build - .gitignore for generated and environment-specific files
- Default config values with sensible defaults - Deep merge for 5-layer config resolution (defaults -> base -> profile -> local -> local-profile) - Filesystem discovery (.stackctl, .stackctl.<profile>, .stackctl.local, .stackctl.local.<profile>) - Post-merge validation returning all errors at once - Template generation with inline comments, --detect, --preset, --profile, --force, --dry-run - STACKCTL_PROFILE env var support - 43 config tests + existing 15 = 58 passing - CLI init command wired to real implementation
Port of tools/generate_stacks.py from AniTrend/local-stack to idiomatic Deno TypeScript: - File discovery: walks repo root, finds docker-compose.yml/yaml files with x-stack metadata - Fragment loading: optional swarm.fragment.yml deep-merge per service - Compose deep merge (dict recursive, array replacement, scalar override) - Service transforms: strip compose-only keys (container_name, restart, build), inject logging defaults, rewrite env_file and bind-mount paths to repo-root relative - Named volume collection (external: true), default traefik-public overlay network - YAML output with header comment, --dry-run support - CLI generate command wired to real implementation - 60 compose tests + 58 existing = 118 passing
- composeOverrideMerge: scalars replace, maps merge, sequences append (distinct from fragment merge which replaces arrays) - loadOverrideFile: load YAML override from relative/absolute path - applyOverrides: load and apply chain of override files to base compose - Override integration in generateStacks via GenerateOptions.overrides - 26 tests covering all merge rules, file loading, edge cases - CLI generate command accepts --override flag
- Variable interpolation: ${VAR}, ${VAR-default}, ${VAR:-default}, $VAR, $$
- Variable scope resolution: shell env -> env_file(s) -> service.environment
- Deep interpolation through all string values in compose structures
- Path absolutization for env_file and bind-mount paths
- Strict mode (fail on unresolved) and non-strict mode (leave as-is with warnings)
- CLI pipeline: resolveConfig -> generateStacks -> renderStack -> output
- 49 comprehensive tests covering all interpolation forms and edge cases
Covers config migration, command mapping, profiles, overrides, rollback, troubleshooting, and behavior differences.
- Add composite action at .github/actions/setup-stackctl/action.yml - Support linux-x64, linux-arm64, macos-x64, macos-arm64 - Download from GitHub Releases, verify SHA256, cache in tool cache - Resolve latest version via GitHub API, accept explicit versions - Add PATH integration for subsequent workflow steps - Document CI usage in docs/migration.md Closes #11
- Add RealProcessRunner using Deno.Command with dry-run and signal forwarding - Add Docker CLI integration module (deploy, rm, services, ps, logs, info, swarm) - Add full sync pipeline: config -> discover -> generate -> render -> deploy - Wire CLI commands: up, down, status, logs, doctor, sync - Replace all issue #6 stubs with real implementations - Add 31 new tests (22 docker + 9 sync) all using FakeProcessRunner
Implements config-first, change-aware stack reload: - reloadStacks() in src/compose/reload.ts with SHA-256 checksum comparison - CLI wiring in src/cli/mod.ts with --skip-generate, --follow-logs, --dry-run - 19 unit tests covering dry-run, unchanged detection, deployment, error handling - Only deploys stacks whose rendered output has changed Ref: #9
- Add EnvExample, EnvDiff, CreateResult, BatchCreateResult types - Implement discoverEnvExamples with profile-driven discovery - Implement createEnvFromExample with dry-run and force support - Implement diffEnvFiles for key comparison - Add batchCreateEnvs helper for bulk operations - Wire env list, create, diff subcommands to CLI - Add 30 unit tests: discovery, creation, diff, batch ops Issue: #14
Implement encrypt, decrypt, deploy, clean, and check subcommands for managing SOPS-encrypted dotenv files with age keys. All operations go through the ProcessRunner interface enabling dry-run and test faking. - Add ToolingStatus, EncryptResult, DecryptResult, DeployResult, CleanResult types - Implement checkTooling() for sops/age availability detection - Implement resolveAgeKey() with config file, env var, and CLI flag resolution - Implement discoverEncryptedFiles() / discoverDecryptedFiles() for file discovery - Implement encryptFile() / decryptFile() with --dry-run support - Implement deploySecrets() for decrypting and creating Docker secrets - Implement cleanTempFiles() for removing .tmp and stray decrypted files - Add ageKeyFile and secretsDir to SecretsConfig - Wire all secrets subcommands in CLI with RealProcessRunner - Add 42 comprehensive tests using FakeProcessRunner Ref: #7
wax911
left a comment
wax911
left a comment
There was a problem hiding this comment.
Blocking review notes against #7:
-
This implements a different secrets model from the plan. The required local-stack workflow is dotenv file encryption/decryption: decrypt encrypted env files to active plaintext env files, render/deploy affected stacks, then clean plaintext env files. This PR instead creates Docker secrets (
docker secret create) from decrypted files. That is not what #7 specified and will not preserve local-stack behavior. -
The SOPS command uses
--input-type=yaml --output-type=yaml. The required format is dotenv:--input-type dotenv --output-type dotenv, matching the existing local-stack.env.encworkflow. -
Decrypt should not require resolving/passing an age key as an explicit CLI/config value. SOPS already resolves age private keys through its configured key file/environment. The existing workflow depends on SOPS config and age key files, not passing recipient/public keys into every operation.
-
The required
secrets deploypipeline is: decrypt target env files -> determine affected stacks -> generate/render/deploy affected stacks -> remove plaintext env files created by this run. This PR does not implement that pipeline. -
Cleanup should use the required
shred -uwithrm -ffallback for active plaintext env files. Temp directory cleanup alone is insufficient because the planned deploy workflow intentionally writes service.envfiles temporarily. -
Optional dependency behavior should fail before mutation if
sopsorageis missing. Verify this is enforced at the command entrypoint, not only exposed as a helper.
This PR needs to be reworked around the existing local-stack SOPS dotenv workflow, not Docker Swarm secrets.
Replace Docker Swarm secret creation with SOPS+age dotenv file encryption/decryption pipeline. Key changes: - encryptEnvFile/decryptEnvFile use --input-type dotenv --output-type dotenv - No explicit age key/recipient passed; SOPS resolves from .sops.yaml - New deployPipeline: decrypt .env.enc -> generate -> render -> deploy -> cleanup - cleanDecryptedEnvFiles uses shred -u with rm -f fallback - ensureTooling throws clear error before any file mutation - findEncryptedEnvFiles/findEnvExampleFiles discover env files in repo - Removed all Docker secret create code (local-stack pattern) - Updated CLI handlers for secrets encrypt/decrypt/deploy/clean/check Ref: #24
wax911
left a comment
wax911
left a comment
There was a problem hiding this comment.
Follow-up review after the push:
Good fixes:
- The implementation now uses the local-stack-compatible dotenv model instead of Docker Swarm secrets.
- SOPS commands now use
--input-type dotenv --output-type dotenv. - Explicit age key passing was removed from encrypt/decrypt operations.
- Cleanup now uses
shred -uwithrm -ffallback. - The deploy pipeline is now framed as decrypt -> generate -> render -> deploy -> cleanup.
Remaining issues:
-
The PR is still a draft and targets
mainwith a cumulative diff. Retarget it onto the correct predecessor branch when the stack order is settled. -
deployPipeline()should call the tooling check before any mutation. The helper exists, but from the visible implementation I do not seeensureTooling()enforced at the start of the deploy path. This was a hard requirement: ifsopsorageis missing, no decrypt/render/deploy/cleanup flow should begin. -
Affected stack detection is probably wrong. It derives affected stacks from the parent directory names of
.env.encfiles. In local-stack, service directory names are not necessarily stack names. The correct mapping should use discovered compose metadata /x-stackto map service paths to stack names. -
Decrypt writes directly to the final
.envpath via SOPS--output. The issue required temp file + atomic move for decrypted outputs. Add that to avoid partial.envfiles on interrupted/failed decrypts. -
Cleanup must only remove plaintext env files created by this run, not arbitrary
.envfiles discovered later. Keep an explicit created-files list from decrypt results.
This is a major improvement, but it still needs hardening before it can close #7.
1 participant
Closes #7