Skip to content

fix: retry flaky SSH connections in confidential-artifacts.sh - #34

Merged
odesenfans merged 1 commit into
mainfrom
od/fix-confidential-artifacts-ssh-retry
Aug 10, 2026
Merged

fix: retry flaky SSH connections in confidential-artifacts.sh#34
odesenfans merged 1 commit into
mainfrom
od/fix-confidential-artifacts-ssh-retry

Conversation

@odesenfans

Copy link
Copy Markdown
Contributor

Problem

PR #32's CI run failed at "Prepare confidential VM artifacts":

==> Fetching sevctl from 209.38.37.171...
kex_exchange_identification: read: Connection reset by peer
Connection reset by 209.38.37.171 port 22
scp: Connection closed

This is the known fresh-droplet SSH flake that crn-up.sh already documents and mitigates (ControlMaster multiplexing + retry on ssh/scp exit code 255). confidential-artifacts.sh runs plain ssh/scp with no protection, so a single reset kills the whole job 25 minutes in.

Fix

Port the same mitigation to confidential-artifacts.sh:

  • SSH_OPTS gains ControlMaster=auto / ControlPersist=600 (own control path, /tmp/ssh-cm-ca-*), so all calls to a given host reuse one TCP connection and only the first is exposed to the flake.
  • New retry_255 wrapper (5 attempts, 5s apart) around tee_ssh, sevctl_ssh, and the three scp call sites. All remote commands are idempotent, so re-running on a connection-level failure is safe; non-255 exit codes still fail immediately.

Behavior-only change for transient failures; a persistent connection problem still fails after 5 attempts.

PR CI failed in 'Prepare confidential VM artifacts' with the known
fresh-droplet flake (kex_exchange_identification: Connection reset by
peer) while scp-ing sevctl from CRN 0.

crn-up.sh already mitigates this with ControlMaster multiplexing plus
a retry-on-exit-255 wrapper; port the same mitigation to
confidential-artifacts.sh (tee_ssh, sevctl_ssh, and the three scp
call sites). The remote commands are idempotent, so retrying
connection-level failures is safe.

@foxpatch-aleph foxpatch-aleph left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, correct port of the proven crn-up.sh SSH retry mitigation to confidential-artifacts.sh. The retry_255 wrapper is safe under set -euo pipefail, all ssh/scp call sites are covered, and the ControlPath namespace (ca vs crn) prevents socket collisions between the two scripts. The change is behavior-only for transient connection failures; persistent failures still exit after 5 attempts. No correctness, security, or quality issues found.

scripts/confidential-artifacts.sh (line 92): Minor nit (identical to crn-up.sh): on the final iteration (attempt 5/5), the function still prints "retrying in 5s ..." and sleeps, but no retry follows — the loop falls through to return 255. The message is slightly misleading on the last attempt. Not worth changing unless crn-up.sh is also updated.

scripts/confidential-artifacts.sh (line 156): Pre-existing, not introduced by this PR: mktemp -d is not idempotent — if the SSH connection drops after the remote mktemp succeeds but before the output is received, the retry creates a second workdir, orphaning the first under /opt/aleph-ci-cache/build-. The prune at line 179 only cleans rootfs-.img files, not build dirs. Low impact in CI (ephemeral servers), but could be addressed by cleaning stale build-* dirs in the prune step.

@odesenfans
odesenfans merged commit a5eff72 into main Aug 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants