fix: retry flaky SSH connections in confidential-artifacts.sh - #34
Conversation
PR CI failed in 'Prepare confidential VM artifacts' with the known fresh-droplet flake (kex_exchange_identification: Connection reset by peer) while scp-ing sevctl from CRN 0. crn-up.sh already mitigates this with ControlMaster multiplexing plus a retry-on-exit-255 wrapper; port the same mitigation to confidential-artifacts.sh (tee_ssh, sevctl_ssh, and the three scp call sites). The remote commands are idempotent, so retrying connection-level failures is safe.
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Clean, correct port of the proven crn-up.sh SSH retry mitigation to confidential-artifacts.sh. The retry_255 wrapper is safe under set -euo pipefail, all ssh/scp call sites are covered, and the ControlPath namespace (ca vs crn) prevents socket collisions between the two scripts. The change is behavior-only for transient connection failures; persistent failures still exit after 5 attempts. No correctness, security, or quality issues found.
scripts/confidential-artifacts.sh (line 92): Minor nit (identical to crn-up.sh): on the final iteration (attempt 5/5), the function still prints "retrying in 5s ..." and sleeps, but no retry follows — the loop falls through to return 255. The message is slightly misleading on the last attempt. Not worth changing unless crn-up.sh is also updated.
scripts/confidential-artifacts.sh (line 156): Pre-existing, not introduced by this PR: mktemp -d is not idempotent — if the SSH connection drops after the remote mktemp succeeds but before the output is received, the retry creates a second workdir, orphaning the first under /opt/aleph-ci-cache/build-. The prune at line 179 only cleans rootfs-.img files, not build dirs. Low impact in CI (ephemeral servers), but could be addressed by cleaning stale build-* dirs in the prune step.
Problem
PR #32's CI run failed at "Prepare confidential VM artifacts":
This is the known fresh-droplet SSH flake that
crn-up.shalready documents and mitigates (ControlMaster multiplexing + retry on ssh/scp exit code 255).confidential-artifacts.shruns plain ssh/scp with no protection, so a single reset kills the whole job 25 minutes in.Fix
Port the same mitigation to
confidential-artifacts.sh:SSH_OPTSgainsControlMaster=auto/ControlPersist=600(own control path,/tmp/ssh-cm-ca-*), so all calls to a given host reuse one TCP connection and only the first is exposed to the flake.retry_255wrapper (5 attempts, 5s apart) aroundtee_ssh,sevctl_ssh, and the threescpcall sites. All remote commands are idempotent, so re-running on a connection-level failure is safe; non-255 exit codes still fail immediately.Behavior-only change for transient failures; a persistent connection problem still fails after 5 attempts.