homelab

Docker Compose setup for my Synology NAS. main is what runs: the NAS pulls it every 5 minutes.

How a push reaches the NAS

scripts/deploy.sh (root, every 5 minutes) fast-forwards the NAS checkout to origin/main, then, for what changed since the last deployed commit (.last-deployed):

• first the *.before.sh one-time operations not run yet
• stacks/<stack>/… → scripts/compose.sh <stack> up -d --remove-orphans, skipped for a stack whose services all sit behind a profile (backup: nothing to run until asked)
• config/<name>/… → docker restart <name>
• then the other one-time operations not run yet

A failure leaves the commit unmarked, so the next run retries it and DSM emails the output. A deleted stack is never torn down automatically: the run prints the docker compose -p <stack> down to run. Every run also checks that the nightly database backup still succeeds (see Backups).

Layout

stacks/<stack>/compose.yml     one Compose project per folder
stacks/<stack>/.env.example    the stack's variables (absent when it only uses common ones)
stacks/common.env.example      variables shared by every stack
config/<name>/                 files mounted into the container named <name>
operations/                    one-time scripts, see below
scripts/                       deploy and maintenance scripts, tests in scripts/tests/

The real stacks/common.env and stacks/<stack>/.env exist only on the NAS (gitignored, mode 600).

Rules the scripts rely on

• Run Compose through scripts/compose.sh <stack> …: plain docker compose doesn't load the env files.
• Every service has a container_name, and config/<name>/ is named after it.
• App data lives in host folders (${DOCKERCONFDIR}, ${DOCKERSTORAGEDIR}), never in the repo or a Docker volume.
• Every ${VAR} used in a compose file is listed in an .env.example (CI checks it; the deploy refuses a stack whose .env lacks a listed key).
• Images are pinned to exact versions.
• Only portainer and docker-socket-proxy may mount the Docker socket, and nothing runs privileged (CI checks it; allowlist in scripts/check_stacks.sh).
• Every service with a writable volume has a homelab.backup label saying how its data is backed up (CI checks it; see Backups).

Settings and secrets

Edit them on the NAS, never in git:

sudo scripts/edit_env.sh <stack>    # or: common

It validates the file, keeps the old one as .previous, and redeploys what it affects. Single-quote values containing $ (password hashes).

Updates

Renovate opens a pull request per image or GitHub Action update and merges it once CI passes; the NAS deploys it within 5 minutes. Both Opusline images update together. The Dependency Dashboard issue shows what's pending.

• Never proposed, on purpose: Postgres major versions and Immich's database/cache images (see renovate.json).
• CI checks the configuration, not the app: a broken release still deploys.
• To roll back, git revert, and in the same commit block the version in renovate.json ({"matchPackageNames": ["<image>"], "allowedVersions": "<x.y.z"}), or Renovate merges it again.

One-time operations

For commands that must run once on the NAS (fix a database, move a folder), like Laravel's one-time operations:

scripts/new_operation.sh "reset immich password"             # runs after the stacks are updated
scripts/new_operation.sh --before "create the foo folder"    # runs right after the pull, before any container changes

Write the commands (container commands through scripts/compose.sh <stack> exec -T …), push. The next deploy runs it once, as root from the repo root, in file name order: *.before.sh operations before the env files are checked and before any container changes (to prepare a folder or a missing .env key), the others after the stacks are updated. It is then recorded in .operations-done and never runs again, even if edited. A failing one blocks the deploy and is retried. sudo scripts/run_operations.sh --list shows what ran.

Sonarr and Radarr settings

Quality profiles and custom formats live in config/recyclarr/: profiles and scores in configs/instances.yml, each custom format as a JSON file in custom-formats/<service>/. scripts/sync_arr_settings.sh keeps them in sync both ways, every 15 minutes on the NAS:

• changed in Sonarr/Radarr → pull request arr-settings-sync with the new files, merged once CI passes;
• changed in the repo → applied to the apps by Recyclarr once deployed;
• changed on both sides since the last sync → nothing is touched and the job fails (DSM emails you) until you pick a side: sudo scripts/sync_arr_settings.sh --take-apps or --take-repo.

Not synced: naming (Recyclarr only knows the TRaSH Guides' presets), and deleting a quality profile from the repo (delete it in the app). Edit the files the way the export writes them (a score of 0 is no entry): after applying, the sync reads the apps back and fails if they don't match the repo exactly. Before pushing a change, scripts/preview_recyclarr.sh (your computer) shows what it would do to the apps. scripts/export_arr_settings.sh (your computer) copies the apps into the repo by hand.

Backups

scripts/backup_nas.sh runs every night as root and takes the NAS's own data off the NAS, to the Hetzner Storage Box:

1. scripts/backup_databases.sh dumps every database into ${BACKUPDIR}/<date>/ (14 days kept), by label, see below.
2. restic, from stacks/backup, backs up the photos, the app data, those dumps, the home folders and this checkout (its .env files included), minus stacks/backup/excludes.txt: live database folders, caches and what the apps regenerate. Encrypted with RESTIC_PASSWORD.
3. On Sundays, restic forgets old snapshots (7 daily, 4 weekly, 12 monthly), prunes, and reads back a 5% sample.

Movies, TV and torrents are left out: they are re-downloadable and don't fit the Storage Box.

Database labels

Every service with a writable volume has a homelab.backup label. CI fails until a new one has it.

Label	What backup_databases.sh does	Used by
postgres	pg_dumpall into <container>.sql.gz, kept only if the dump is complete	Immich, Opusline, Prowlarr databases
sqlite	copies every SQLite file in the container's ${DOCKERCONFDIR} folders with SQLite's online backup, as the file's owner, under the same relative path, and checks each copy with PRAGMA quick_check; skips an app's own dated copies (name-YYYY-MM-DD)	Vaultwarden, Sonarr, Radarr, Tautulli, Seerr, Maintainerr
sqlite-unchecked	like sqlite, without the check: for databases only the app's own SQLite build can open fully; the log marks each copy (not checked)	Plex
bolt	stops the container, archives its ${DOCKERCONFDIR} folders into <container>.tar.gz, starts it again (seconds of downtime)	Portainer, Filebrowser
none	nothing: plain files restic copies as they are, or data not worth keeping; a comment beside the label says which	everything else with a writable volume

The job fails if a sqlite container holds no SQLite file.

Noticing a broken backup

Every deploy run checks that both backups succeeded in the last 26 hours: ${BACKUPDIR}/last-success for the dumps, ${BACKUPDIR}/offsite-last-success for restic. When one didn't, or never did, the deploy run fails once, so DSM emails you, then stays quiet until backups succeed again. It never holds a deploy back.

Dashboard

Glance's Home page shows the backups under the NAS stats:

• Backups: for PC → NAS and PC → Storage Box (the PC backs itself up, from its dotfiles) and NAS → Storage Box, the number of snapshots, the age of the last one and the space taken, plus the NAS's snapshots of the PC's share. scripts/backup_status.sh reads them every hour from the repositories' files, without their passwords (restic writes each snapshot as one file when the backup ends), into ${DOCKERCONFDIR}/backup-status/backups.json, which the backup-status container serves to Glance. A repository it can't read shows its error in red, and so does "Updated" when the hourly task stops.
• Storage Box: space used out of the quota, split into data and snapshots, and the Storage Box snapshots, live from the Hetzner API.

Setup

Once, on the NAS, as root:

1. The folders the backup stack mounts (Synology's Docker refuses to start a container whose bind-mounted folder doesn't exist, so create BACKUPDIR too), then the SSH key for the Storage Box, readable by root only:

   sudo mkdir -p /volume1/docker/appdata/restic/ssh /volume1/docker/appdata/restic/cache /volume1/backup/databases
   sudo chmod 700 /volume1/docker/appdata/restic/ssh /volume1/backup/databases
   sudo ssh-keygen -t ed25519 -N '' -C jeancloud-restic -f /volume1/docker/appdata/restic/ssh/id_ed25519

2. Add the public key to the Storage Box's .ssh/authorized_keys (with SFTP, from a machine that can already log in), as the usual one-line OpenSSH key, and in Hetzner Console keep "SSH Support" and "External Reachability" on.
3. /volume1/docker/appdata/restic/ssh/config (mode 600; IdentityFile is the path inside the container). Port 23: the Storage Box only accepts one-line OpenSSH keys there, port 22 wants them in RFC4716 format:

   Host storagebox
       HostName u000000.your-storagebox.de
       Port 23
       User u000000
       IdentityFile /root/.ssh/id_ed25519
   

   Then connect once, which records the host key (DSM has no ssh-keyscan) and checks the key login:

   echo 'ls -la' | sudo sftp -b - \
     -F /volume1/docker/appdata/restic/ssh/config \
     -i /volume1/docker/appdata/restic/ssh/id_ed25519 \
     -o StrictHostKeyChecking=accept-new \
     -o UserKnownHostsFile=/volume1/docker/appdata/restic/ssh/known_hosts \
     storagebox

4. sudo scripts/edit_env.sh common (BACKUPDIR), then sudo scripts/edit_env.sh backup (RESTIC_REPOSITORY, RESTIC_PASSWORD, HOMESDIR). Keep RESTIC_PASSWORD outside the NAS: Vaultwarden runs on it.
5. Create the repository: sudo scripts/compose.sh backup run --rm -T restic init.
6. Task Scheduler: bash /volume1/docker/homelab/scripts/backup_nas.sh as root daily at 02:30, email on failure. Run it once by hand: the first upload takes hours, and a run still going the next night is skipped.
7. Dashboard: in Hetzner Console, create a Read-only API token in the project holding the Storage Box, then sudo scripts/edit_env.sh infrastructure (HETZNER_API_TOKEN, HETZNER_STORAGE_BOX_ID) and sudo scripts/edit_env.sh backup (the PC_RESTIC_* paths). Task Scheduler: bash /volume1/docker/homelab/scripts/backup_status.sh as root every hour, email on failure.

Restore

Every restic command runs through the stack, e.g. sudo scripts/compose.sh backup run --rm -T restic snapshots.

• Files: mount a folder to restore into, e.g. sudo scripts/compose.sh backup run --rm -T -v /volume1/restore:/restore restic restore latest --target /restore --include /source/appdata/sonarr. Snapshot paths start with /source/photos, /source/appdata, /source/databases, /source/homes and /source/homelab.
• Postgres: restore the dump from /source/databases/<date>/, then gunzip -c <file>.sql.gz | sudo docker exec -i <container> psql -U <user> -d postgres.
• SQLite: stop the container, replace the database file with its copy from /source/databases/<date>/, delete its -wal and -shm files, start it. The copies under /source/appdata were taken live and may be inconsistent.
• Bolt: stop the container, sudo tar -xzf <container>.tar.gz -C ${DOCKERCONFDIR} (it overwrites the folder's files), start it.
• Whole NAS lost: on any machine with Docker, run restic/restic with the same .ssh folder and RESTIC_PASSWORD (from outside the NAS) and restore /source first; this checkout and its .env files come back with it.

Scripts

Script	Purpose
deploy.sh	the 5-minute deploy (DSM Task Scheduler, root)
compose.sh <stack> …	docker compose with the stack's env files
edit_env.sh <stack>|common	edit settings and secrets on the NAS
new_operation.sh, run_operations.sh	one-time operations (--before/--after, --list, --mark-all-done)
premigration_check.sh <stack>	compare running containers with the compose file before replacing them
cleanup_deluge.sh	remove orphaned torrents (scheduled; reads its API keys from stacks/media/.env, DRY_RUN=1 to simulate)
backup_nas.sh	nightly: database dumps, then the restic backup to the Storage Box (see Backups)
backup_databases.sh	the database dumps, by homelab.backup label (run by backup_nas.sh)
backup_status.sh	hourly: the backup numbers the dashboard shows (see Backups, Dashboard)
check_backups.sh	fails once when a backup is over 26 hours old (run by deploy.sh)
sync_arr_settings.sh	two-way sync of the Sonarr/Radarr settings (every 15 minutes, root; --take-apps, --take-repo)
export_arr_settings.sh, preview_recyclarr.sh	copy Sonarr/Radarr settings into the repo, preview a sync (your computer)
check_stacks.sh, check_glance_config.sh	CI checks, runnable locally
migration_helpers.sh	helpers used once, for the migration from Portainer

Tests: for test_file in scripts/tests/*_test.sh; do bash "$test_file"; done (needs bash, git, jq, flock, Docker Compose). CI runs them, the two checks and gitleaks on every push and pull request.

New server

1. Install Git (Package Center), clone the repo as root (LinuxServer images only run root-owned init scripts).
2. Generate the dashboard login:

   image=$(awk '$1 == "image:" && $2 ~ /^glanceapp\/glance:/ { print $2 }' stacks/infrastructure/compose.yml)
   docker run --rm --entrypoint /app/glance "$image" secret:make                     # GLANCE_SECRET_KEY
   docker run --rm --entrypoint /app/glance "$image" password:hash '<your password>'  # GLANCE_PASSWORD_HASH

3. sudo scripts/edit_env.sh common, then sudo scripts/edit_env.sh <stack> for each stack with an .env.example.
4. sudo scripts/deploy.sh once. It runs every one-time operation: on a rebuilt server whose data already went through them, run sudo scripts/run_operations.sh --mark-all-done first.
5. Task Scheduler: bash /volume1/docker/homelab/scripts/deploy.sh as root every 5 minutes, email on failure.
6. Backups: follow Backups, Setup. Until a backup succeeds, the deploy reports it.
7. Settings sync: create a fine-grained GitHub token for this repository only, with Contents and Pull requests set to read and write, and save it root-only on the NAS: sudo sh -c 'umask 077; cat > /volume1/docker/homelab/.github-token' (paste, Enter, Ctrl-D). Turn on "Allow auto-merge" in the repository settings. Then Task Scheduler: bash /volume1/docker/homelab/scripts/sync_arr_settings.sh as root every 15 minutes, email on failure. When the token expires, the job fails until you save a new one.

Security

Pushing to main (or publishing an image Renovate picks up) deploys as root on the NAS. Keep 2FA on, and protect main:

gh api -X PUT repos/Androlax2/homelab/branches/main/protection --input - <<'JSON'
{
  "required_status_checks": {"strict": true, "contexts": ["stacks", "glance-config", "scripts", "secrets"]},
  "enforce_admins": true,
  "required_pull_request_reviews": {"required_approving_review_count": 0},
  "restrictions": null,
  "allow_force_pushes": false,
  "allow_deletions": false
}
JSON

The repo is public: secrets only in the NAS .env files, and security reviews are not committed. The NAS holds a GitHub token for the settings sync: its pull requests may only change config/recyclarr/ (CI checks it).

Troubleshooting

• Deploy says .env is missing / lacks keys: sudo scripts/edit_env.sh <stack> (or common).
• Variables empty / "variable is not set": Compose was run directly; use scripts/compose.sh.
• git merge --ff-only fails: local edits or a force push. git status, then git reset --hard origin/main (.env files are gitignored, so they survive).
• Glance won't start: sudo scripts/compose.sh infrastructure logs glance names the missing value.
• Deploy says Backups: …: the nightly backup_nas.sh task didn't succeed. Open its last output in Task Scheduler, or run sudo bash scripts/backup_nas.sh to see which step fails.