fix(validator): promote snapshots transactionally and fail closed

[POPPIN-FUMI]

Outcome

Make validator snapshot refresh resumable and transactional after the 2026-09-01 mainnet migration exposed a non-empty /mnt/snapshot/remote nested-directory promotion failure.

Changes

• stable network-scoped same-filesystem staging protected by flock
• source-isolated resumable downloads (wget -c); mainnet direct-FRA → FRA HTTPS → AMS HTTPS, testnet explicit URL only
• full/incremental archive and slot validation before service stop
• exact live→rollback and staging→live renames; no remote/remote nesting
• EXIT recovery restores prior snapshot/service after any post-stop failure
• live snapshot and ledger remain untouched on download/validation/stop failure
• 3-hour download allowance; no masked download failure
• inventory validator_type pass-through instead of hardcoded solv
• synchronized active template and checked-in OSS distribution

Evidence

• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh: PASS, including success, stop failure, ledger failure recovery, lock exclusion, blank-URL testnet, failed-download preservation, and second-invocation resume
• Ansible syntax checks: PASS for copy + wget playbooks
• template/dist byte-parity assertions: PASS
• git diff --check: PASS

Safety

No live server operation, deployment, secret, snapshot download, ledger deletion, or validator restart was performed by this PR. Runtime direct-FRA URL remains an optional inventory input and is not committed. Rollback is reverting this commit before regeneration/release.

[POPPIN-FUMI]

Infra Review v2 claim acquired for PR #411.
Target is fixed to the exact current head for round 1; review-only inspection and local checks are in progress.

auron gate claim / round 1 / 対象 983115f / auron-pr411-r1-cbb31a87

[POPPIN-FUMI]

cyan gate claim / round 1 / 対象 983115f / cyan-pr411-r1-25379702

[POPPIN-FUMI]

cyan gate — round 1 — target 983115f7e8074b32523f7980a818fa7f6a096fba

Verdict: FAIL. The target substantially improves pre-stop staging, source isolation, locking, exact-directory promotion, and the happy path, but the post-stop recovery and checked-in distribution are not fail-closed yet.

Blocking findings

1. High — rollback can start the validator with no live snapshot and then suppress the EXIT retry. In template/2026.6.6.2026/jinja/cmn/restart.sh.j2:111-123 (and the byte-identical dist copy), mv "${rollback}" "${live_snapshot}" || true discards a failed restore, then a successful systemctl start clears rollback_required. I rendered the target template and injected a ledger failure plus an mv failure only when .rollback.* was the source. The script exited nonzero but issued start solv, left remote absent, and stranded .rollback.*. Required correction: make snapshot restoration a checked precondition for restart/flag clearing, keep a usable live candidate if the old snapshot cannot be restored, stop any partially started service before moving archives, and add this negative-path test.

2. High — the new mandatory network selector is not closed over all common-template callers. The template rejects an empty selector at lines 46-48, but a complete import census found no restarter_snapshot_network binding in template/2026.6.6.2026/ansible/mainnet-rpc/init.yml, mainnet-rpc/init-old.yml, or devnet-rpc/init.yml. Those flows still install this common restarter, so the head turns their previously usable script into an unconditional refusal. Required correction: bind the correct network for every supported caller, or stop installing this restarter for unsupported networks, and enforce the importer census in the contract test.

3. High — the newly added OSS Allnodes playbook is not executable as distributed. ansible-playbook --syntax-check -i 'localhost,' dist/oss-skills/slv-validator/ansible/mainnet-validator/init-allnodes-jito.yml fails because install_allnodes_jito.yml is absent. The same new file also imports a missing local wget_snapshot.yml; the available downloader is ../cmn/wget_snapshot.yml. Required correction: include all required dependencies/use the correct import and syntax-check the self-contained dist entrypoint, not only its copied leaf files.

4. Medium — the restarter download phase has no total wall-clock bound. Lines 94-95 invoke two wget -c commands without a phase deadline. The 10,800-second Ansible async limit applies to the separate initialization downloader, not to /home/solv/restarter.sh; only catch-up is explicitly bounded. Required correction: add a total download/candidate deadline that preserves resumability and a hanging-downloader negative test.

Independent evidence

• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh: PASS (happy path, stop failure, ledger failure, lock exclusion, blank testnet URL, failed download, resume).
• git diff --check d68e9342e2cae223bdc9878dbfa3bbd752d36b15..983115f7e8074b32523f7980a818fa7f6a096fba: PASS.
• Byte-parity checks for the eight changed template/dist counterparts: PASS. Dependency closure still fails as finding 3 demonstrates.
• Independent rollback-rename failure injection: reproduced the unsafe start with no live snapshot.
• Common-import census: exactly the three unbound callers named above.
• Targeted Ansible syntax checks for both common leaf playbooks: PASS; the new dist Allnodes entrypoint: FAIL as above.
• GitHub Run Tests: SUCCESS on this target; PR state OPEN and mergeability MERGEABLE immediately before this review.
• Added-line scan found no credential/private-key markers and the 18-file diff changes no workflow/deploy path. No live host, secret, deployment, or service was accessed.

Requirements, non-regression, and limits

Requirement satisfaction is partial: stable per-source staging, flock, mainnet/testnet candidate separation, pre-stop filename/slot/readability checks, exact non-nesting promotion, and bounded catch-up are present and exercised. Transactional recovery, total download bounding, and template/dist execution closure are not satisfied. Separately, non-regression fails because the now-required selector is absent from three existing callers. I did not run this destructive restarter on a validator or test real snapshot endpoints; those remain deliberately unverified and cannot erase the deterministic failures above.

Counterfactual: if a rollback rename failure prevented restart/flag clearing, every common caller rendered a supported network, the new dist entrypoint passed syntax resolution, and download had a total deadline, this target would not have these blockers. Predicate check: recovery_exact && caller_network_total && dist_entrypoint_resolves && download_bounded is false on the target.

Gate: cyan
Round: 1
Target: 983115f
Verdict: FAIL
Claim: cyan-pr411-r1-25379702

[POPPIN-FUMI]

FAIL — Auron infra Review v2, round 1

Target: 983115f7e8074b32523f7980a818fa7f6a096fba; baseline: current main d68e9342e2cae223bdc9878dbfa3bbd752d36b15. I reviewed the snapshot restarter state machine, Ansible rendering/failure propagation, caller blast radius, validator type propagation, source/distribution closure, rollback, and URL/log safety.

Blocking findings:

1. template/2026.6.6.2026/jinja/cmn/restart.sh.j2:50 is not renderable by Ansible/Jinja. The Bash expansion ${#candidate_urls[@]} contains Jinja's {# comment opener, and there is no closing #}. The real ansible.builtin.template --check action returns rc=2 with Missing end of comment tag; the identical action against the base template returns rc=0. Therefore every changed copy_restart_sh.yml path fails before installing restarter.sh. The committed behavior test misses this because it renders with sed, not Jinja. Escape/raw-wrap this Bash expression (or avoid that expansion) and add a real Jinja/Ansible render assertion.

2. The changed template is common, but its caller closure is incomplete. The active template tree has nine import_playbook callers of copy_restart_sh; mainnet-rpc/init.yml, mainnet-rpc/init-old.yml, and devnet-rpc/init.yml do not pass restarter_snapshot_network. They therefore render readonly snapshot_network="" and the generated restarter exits through the unsupported-network branch. The CLI directly selects the mainnet and devnet init playbooks. Either scope every caller to a supported network or explicitly keep unsupported surfaces from receiving this common restarter, with render/behavior coverage for each intended caller.

3. Source/dist closure is partial. At base, the active common restarter and the validator, RPC, and gRPC distribution copies all had SHA-256 b077e803.... At this head, source plus validator dist are 8ee141c6..., while RPC and gRPC dist remain b077e803..., even though build-skill-oss.sh copies common Jinja/Ansible dependencies into all three packages. Also, newly added dist/oss-skills/slv-validator/ansible/mainnet-validator/init-allnodes-jito.yml fails ansible-playbook --syntax-check because its package omits install_allnodes_jito.yml (and the local wget dependency is also absent). Reconcile the complete intended distribution closure and assert parity/dependency completeness; do not treat the current validator-only cmp checks as full closure.

4. The promotion is recoverable only while this Bash process reaches its EXIT trap. After live_snapshot is renamed to the timestamp/PID rollback path and before staging is promoted, SIGKILL, power loss, or host failure leaves remote absent. A later invocation initializes rollback="", never scans .rollback.*, and can proceed without restoring the preserved snapshot. Use a stable, network-scoped transaction marker/rollback path and reconcile it under the lock before new download/promotion. The full direct-FRA base is also written to stderr; log the candidate name rather than an unconstrained URL so userinfo/query credentials or private topology cannot enter service/cron logs.

Checks and results:

• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh → PASS / rc 0 (soft stop failure, ledger failure, lock exclusion, blank testnet URL, failed download preservation, resume, and validator-dist parity covered).
• ansible localhost ... -m ansible.builtin.template ... --check → base rc 0; target rc 2, Missing end of comment tag.
• ansible-playbook --syntax-check → changed common copy/wget and top-level mainnet-validator, testnet-validator, and testnet-rpc playbooks pass; the newly added dist allnodes entry fails on missing dependencies. Legacy source init-jito/init-firedancer/init-allnodes and two legacy dist init files fail on both base and head, so those pre-existing failures are not counted as this round's regression.
• git diff --check origin/main...HEAD → PASS / rc 0.
• GitHub SLV Tests and Release / Run Tests → SUCCESS on the fixed target; the macOS and remote release jobs are SKIPPED by their conditions.
• Stem/caller census used repository-wide git grep; active-version import census and source/dist SHA-256 comparisons produced the closure findings above. No workflow files changed, so Bahamut intake was not triggered.

Requirement result: resumable per-source staging, flock, pre-stop archive/slot checks, the 10,800-second Ansible allowance with failure propagation, and inventory validator_type pass-through are present and pass the committed soft-failure contract. Non-regression result: FAIL because the deploy-time renderer is newly broken and common caller/distribution closure is incomplete.

Counterfactual: if this head were deployable, the same Ansible template action that succeeds on base would return rc 0 on the target; it instead returns rc 2 before any host mutation, so the observation distinguishes deployable from non-deployable.

Unverified: no live validator, systemd service, ledger deletion, restart, catch-up, deployment, or real direct-FRA value was exercised. Hard-crash recovery is established by the static state transition above, not by injecting failure on a live host. During a discarded isolated mock-harness attempt, mocks did not import under the invoking shell and a read-only public FRA download began; it was stopped, its exact temporary tree was deleted, no remote state was mutated, and that traffic is not used as behavior evidence. Green GitHub CI cannot substitute for the failing render action because the committed test uses sed instead of Jinja.

Gate: auron
Round: 1
Target: 983115f
Verdict: FAIL
Claim: auron-pr411-r1-cbb31a87

[POPPIN-FUMI]

Auron round 1 claim reacquired solely to correct the posted review's unverified-scope wording; target and FAIL polarity are unchanged.

auron gate claim / round 1 / 対象 983115f / auron-pr411-r1-cbb31a87

[POPPIN-FUMI]

Infra Review v2 round 2 claim acquired for PR #411.
Target is fixed to the exact descendant head; repair verification and full current-diff review are in progress.

auron gate claim / round 2 / 対象 9b1e754 / auron-pr411-r2-0584337a

[POPPIN-FUMI]

cyan gate claim / round 2 / 対象 9b1e754 / cyan-pr411-r2-87f31c6d

[POPPIN-FUMI]

PASS — Auron infra Review v2, round 2 (final round)

Target: 9b1e754a3b4d09a5d0cc8ec5f9dc25082b8d9ced; baseline: current main d68e9342e2cae223bdc9878dbfa3bbd752d36b15. The round-1 target 983115f7e8074b32523f7980a818fa7f6a096fba is an ancestor. I reviewed the full current diff and independently rechecked Jinja rendering, shell behavior/recovery, download bounding, caller/network closure, Ansible failure propagation and syntax, validator-type propagation, source/dist parity, secrets/logging, and workflow scope.

Round-1 repair acceptance:

• The Jinja {# collision is removed. Real ansible.builtin.template renders followed by bash -n passed for jito/mainnet, firedancer-jito/mainnet, jito/testnet, and jito/devnet; service and network selections rendered correctly.
• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh passes. It exercises normal promotion, stop failure, ledger failure rollback, rollback-rename failure with no unsafe start, single interrupted rollback restoration/restart-and-stop-before-forward-retry, lock exclusion, blank testnet URL, failed-download preservation, and resume.
• The active source census has nine copy_restart_sh importers and nine adjacent network bindings: mainnet validator/RPC paths are mainnet, testnet paths are testnet, and devnet RPC is devnet. All corresponding checked-in distribution importers are also bound.
• Common restarter, copy playbook, and wget playbook are byte-identical across the active template and all validator/RPC/gRPC distributions (respectively SHA-256 7ffe9a97..., a2c15844..., and 0a1cb79d...). The broken dist Allnodes entry is removed.
• The restarter now enforces one total download deadline across source candidates using the remaining budget for each timeout ... wget; invalid timeout inputs fail before live mutation, partial files remain resumable, and logging names the candidate rather than printing the direct URL.
• Thirteen targeted ansible-playbook --syntax-check invocations passed: the common copy/wget leaves; source validator and RPC top-level entrypoints; and validator, RPC, and gRPC distribution entrypoints. git diff --check origin/main...HEAD also passed.
• GitHub SLV Tests and Release / Run Tests is SUCCESS on this target; conditional macOS/release jobs are SKIPPED. The PR is OPEN and MERGEABLE. No workflow files changed, so Bahamut intake was not triggered.

Unresolved finding returned under round-2 exhaustion (non-blocking): restore_service at template/2026.6.6.2026/jinja/cmn/restart.sh.j2:144-164 can move the promoted live snapshot before issuing a defensive systemctl stop. The caller at lines 190-192 enters this function when systemctl start returns nonzero. If systemd reports a failed/timed-out start while the unit remains active or activating, the function can rename archives under that process and then treat a subsequent start as recovery. The round-1 correction explicitly called for stopping any partially started service before archive movement, and the new mock does not inject this start-failed-but-active state. The safe follow-up is to stop/confirm inactive before moving a promoted live archive and add an ordering assertion. This is not active in-progress funds/data/public-exposure harm; per gates.gateRoundLimit.default = 2 and gateRoundExhausted = unblock-and-return-findings, it does not continue the gate block in round 2. L1/owner should carry it; Auron is not filing an issue.

Requirement result: the requested staging/resume/flock/network scoping, bounded download, pre-stop archive/slot validation, checked rollback restoration, failure propagation, validator-type pass-through, direct-URL log redaction, and shared source/dist closure are satisfied on the reviewed paths. Non-regression result: PASS for the measured active source and checked-in distribution callers relative to current main; the prior three unscoped callers and partial distributions are repaired.

Counterfactual: if these repairs were textual only, at least one real Jinja render, behavior transition, caller binding, parity hash, syntax check, or current-head CI check would remain red; all of those measured predicates are green on this target. The residual partial-start ordering case is explicitly outside that proven behavior set and is returned above rather than silently generalized away.

Unverified: no live validator, systemd unit, ledger deletion, snapshot download, deployment, or real direct-FRA value was exercised in round 2. The full 10,800-second wall-clock expiry was not waited out; its remaining-budget control flow was inspected and the committed contract checks its wiring. Legacy specialized init files that already failed dependency resolution on both base and round 1 were not promoted into new distribution entrypoints.

Gate: auron
Round: 2
Target: 9b1e754
Verdict: PASS
Claim: auron-pr411-r2-0584337a

[POPPIN-FUMI]

cyan gate — round 2 — target 9b1e754a3b4d09a5d0cc8ec5f9dc25082b8d9ced

Verdict: FAIL. The four round-1 findings are materially repaired, but an independent real Jinja render and a new post-catch-up failure injection found two remaining blockers in the complete target.

Blocking findings

1. High — a normal empty-snapshot_url Jinja render produces an invalid restarter. ansible.builtin.template with restarter_snapshot_network=mainnet and the common copier's default-empty snapshot_url removes lines 46-48 of template/2026.6.6.2026/jinja/cmn/restart.sh.j2. The rendered shell consequently has elif ...; then immediately followed by else; bash -n fails with rc 2 at that else. This affects the three byte-identical distributed templates too. The committed behavioral test does not exercise the real render: lines 125-129 replace the conditional Jinja block with : via sed, masking the empty branch. Required correction: keep an unconditional no-op or equivalent valid command in the testnet/devnet branch and add actual Ansible/Jinja render + bash -n coverage for blank and nonblank URL inputs on every network.

2. High — a cleanup error after successful catch-up can remove both usable snapshot generations while leaving the service running. At lines 201-204 the old rollback and staging root are deleted in one rm -rf while promoted=1 and rollback_required=1 remain armed. I injected an rm that deleted the rollback target and then failed before deleting staging. The EXIT trap moved the healthy live snapshot back into staging, found the rollback already gone, emitted CRITICAL: no live snapshot exists after rollback, and exited with the service still running but /mnt/snapshot/remote absent. Required correction: make successful health/catch-up the explicit commit point before fallible cleanup (or otherwise order cleanup so a failure cannot trigger rollback after the prior generation is deleted), preserve the committed live path, and add this partial-cleanup failure test.

Round-1 correction and independent evidence

• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh: PASS, including the repaired rollback-rename and interrupted-single-rollback paths.
• Real ansible.builtin.template renders with nonempty URLs for mainnet/testnet/devnet followed by bash -n: PASS. The required default-empty mainnet render: FAIL as finding 1 describes.
• Independent partial-cleanup failure injection: reproduced the missing live path/no rollback/service-still-running state in finding 2.
• Common importer census across template and all three OSS packages: 19 importers, 0 missing restarter_snapshot_network bindings.
• Twelve targeted ansible-playbook --syntax-check runs covering common leaves and template/RPC/gRPC caller entrypoints: PASS. The broken new dist Allnodes entrypoint from round 1 is absent.
• Nine template-to-distribution parity checks for the common restarter, copier, and downloader across validator/RPC/gRPC packages: PASS.
• The download phase now has a validated positive total deadline, the direct URL is no longer emitted in the progress log, and failed rollback rename no longer starts without a live snapshot.
• git diff --check d68e9342e2cae223bdc9878dbfa3bbd752d36b15..9b1e754a3b4d09a5d0cc8ec5f9dc25082b8d9ced: PASS.
• GitHub Run Tests: SUCCESS on this target; the PR was OPEN and MERGEABLE immediately before review.
• The 32-file added-line scan found no credential/private-key marker and no workflow/deploy path. No live host, secret, deployment, snapshot endpoint, ledger, or service was accessed.

Requirements, non-regression, and limits

Requirement satisfaction is otherwise strong: stable resumable per-source staging, lock exclusion, network source separation, pre-stop archive/slot checks, exact promotion, bounded download/catch-up, R1 rollback failure retention, interrupted rollback recovery, and active template/dist synchronization are present. However, the generated default configuration is not executable, and recovery is still unsafe for one post-stop cleanup failure, so the target cannot receive an R3 PASS. Separately, the importer and distribution regressions reported in round 1 are closed. Real validator execution remains intentionally unverified and cannot override the deterministic failures above.

Counterfactual: if blank-input Jinja rendering produced valid shell and the post-catch-up commit point could not re-arm rollback after deleting the prior generation, these two blockers would not reproduce. Predicate check: all_supported_renders_parse && committed_live_survives_cleanup_failure is false on this target.

Gate: cyan
Round: 2
Target: 9b1e754
Verdict: FAIL
Claim: cyan-pr411-r2-87f31c6d

[POPPIN-FUMI]

cyan gate claim / round 3 / 対象 1f7f29b / cyan-pr411-r3-3a86d438

[POPPIN-FUMI]

Infra Review v2 round 3 claim acquired for PR #411.
Target is fixed to the exact current head; full-diff and recovery repair verification are in progress under the current unlimited-round policy.

auron gate claim / round 3 / 対象 1f7f29b / auron-pr411-r3-f5177e9d

[POPPIN-FUMI]

cyan gate — round 3 — target 1f7f29bb550eb6b0810836f129455208002ec7ba

Verdict: FAIL. Both round-2 blockers are closed, and the complete safety/caller/distribution matrix otherwise passes. One prior secrecy requirement remains false in the executed download path.

Blocking finding

High — the exact direct snapshot URL is still written to stderr by wget. The progress message at template/2026.6.6.2026/jinja/cmn/restart.sh.j2:123 now emits only the source label, but lines 126 and 129 invoke default-verbose GNU wget with the full ${base} URL and neither quiet it nor capture/sanitize its output. GNU wget prints its request URL before connecting. A local-only reproduction using the target flags against http://127.0.0.1:9/private-source/secret-path/snapshot.tar.bz2 returned rc 4 and printed that complete URL. A rendered-target harness likewise showed child stderr containing the configured direct URL, proving the restarter forwards it. Runtime direct URLs may identify a private source or carry sensitive path material, so replacing only the wrapper's echo does not satisfy “direct URL no longer logged.” Required correction: suppress or capture wget's raw stdout/stderr and emit only a source label plus sanitized status; add a failure-path assertion that the configured direct URL never appears in collected output.

Round-2 closure and independent evidence

• Real ansible.builtin.template render followed by bash -n for all six combinations of mainnet/testnet/devnet × empty/explicit snapshot_url: PASS. The unconditional no-op closes the prior empty-branch parse failure.
• Independent partial-cleanup failure injection that deletes the old rollback and returns nonzero: PASS. The script reports cleanup failure, keeps the promoted snapshot at remote, does not create remote/remote, and does not invoke rollback after the catch-up commit point.
• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh: PASS, including success, cleanup failure, stop failure, ledger rollback, rollback-rename failure, interrupted-single-rollback recovery, lock exclusion, blank testnet URL, failed download preservation, and resume.
• Common importer census across template and all OSS packages: 19 importers, 0 missing network bindings.
• Nine common template/dist parity checks across validator/RPC/gRPC packages: PASS; the broken dist Allnodes entrypoint remains absent.
• Twelve targeted Ansible syntax checks for common leaves and template/RPC/gRPC entrypoints: PASS.
• Inspection confirms one total positive download deadline shared across candidate attempts, bounded catch-up probes, pre-stop archive/slot/readability checks, exact non-nesting promotion, preserved rollback until positive catch-up, and no start when a required rollback rename fails.
• git diff --check d68e9342e2cae223bdc9878dbfa3bbd752d36b15..1f7f29bb550eb6b0810836f129455208002ec7ba: PASS.
• GitHub Run Tests: SUCCESS on this target; PR state OPEN and mergeability MERGEABLE immediately before review.
• The 32-file added-line scan found no committed credential/private-key marker and no workflow/deploy path. No live host, secret, deployment, external snapshot endpoint, ledger, or service was accessed.

Requirements, non-regression, and limits

The transactional snapshot, recovery, network separation, caller closure, distribution closure, and boundedness requirements now pass the reviewed deterministic predicates, including both round-2 corrections. The remaining failure is specifically the runtime output boundary: no secret is committed, but the private direct URL is still observable in normal downloader stderr. Non-regression checks otherwise found no new failing caller, syntax, parity, or CI predicate. Real validator execution remains intentionally unverified and cannot override the deterministic logging failure.

Counterfactual: if the downloader's raw output were suppressed/sanitized and a failing direct request could not reproduce its configured URL in output, this target would have no blocking Cyan finding. Predicate check: transactional_recovery && render_total && caller_dist_total && bounded && direct_url_not_logged is false only on direct_url_not_logged.

Gate: cyan
Round: 3
Target: 1f7f29b
Verdict: FAIL
Claim: cyan-pr411-r3-3a86d438

[POPPIN-FUMI]

PASS — Auron infra Review v2, round 3

Target: 1f7f29bb550eb6b0810836f129455208002ec7ba; baseline: current ValidatorsDAO/slv main d68e9342e2cae223bdc9878dbfa3bbd752d36b15. I reviewed the full 32-file PR diff and the R3 delta from 9b1e754a3b4d09a5d0cc8ec5f9dc25082b8d9ced, covering Jinja/Ansible rendering, shell recovery state, destructive-operation ordering, download/failure propagation, caller/network closure, source/dist parity, secrets/logging, CI reach, and commit-message stability.

Blocking findings: none.

Repair verification:

• Real Ansible/Jinja rendering followed by bash -n passed for six default-empty cases: mainnet/testnet/devnet with snapshot_url undefined or explicitly empty, including Firedancer service selection. The rendered testnet/devnet branch retains the explicit : no-op, so the prior empty-body syntax failure is closed.
• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh passed. Its fired cleanup-failure path removed the previous rollback and then returned nonzero; the restarter reported cleanup failure while retaining the caught-up promoted snapshot at live/remote, with no nested remote/remote rollback.
• The R2 partial-start edge is closed in restore_service: when promoted == 1, checked systemctl stop precedes every move of the promoted live snapshot. I separately fired the stop guard with a local isolated harness: initial stop and promotion succeeded, recovery stop attempts 2 and 3 failed, the command exited 1, no recovery move or start occurred after either failed stop, the promoted live archive remained present, and the old snapshot remained in .rollback.* (RESTORE_STOP_GUARD_FIRED_AND_PRESERVED=PASS).
• The committed behavior contract also passed normal promotion, initial stop failure, ledger failure rollback, rollback-rename failure without unsafe start, single interrupted rollback recovery, lock exclusion, no-URL fail-closed behavior, failed-download preservation, and resume.
• Repository-wide stem census on this target produced 46 copy_restart_sh, 56 wget_snapshot, and 13 restart.sh.j2 hits. Within the active template, all 9 importers have 9 network bindings; across the three checked-in distributions, all 10 importers have 10 bindings.
• Common restarter/copy/wget artifacts are byte-identical across source and validator/RPC/gRPC distributions (SHA-256 groups fa1ba5e0..., a2c15844..., 0a1cb79d...).
• Thirteen targeted ansible-playbook --syntax-check commands passed: common leaves, source validator/RPC top-level entrypoints, and validator/RPC/gRPC distribution entrypoints. git diff --check origin/main...HEAD passed.
• GitHub SLV Tests and Release / Run Tests is SUCCESS on the exact target; the conditional macOS/release jobs are SKIPPED. The existing workflow runs CLI and standalone Solana-key tests, not the new shell behavior contract, so the independent local behavior/render results above—not CI status—carry those claims.
• Added-line scans found no credential/private-key marker or URL-value logging. No workflow file changed, so Bahamut intake was not triggered. The PR title and four commit subjects contain stable change descriptions rather than mutable operational facts.

Requirement result: PASS. Default-empty render safety, post-catch-up cleanup safety, and checked service-stop-before-recovery-move behavior all fire as required. The earlier staging/resume/flock/network scoping, total download deadline, pre-stop archive/slot validation, rollback preservation, validator-type pass-through, failure propagation, direct-URL log redaction, and template/dist closure remain intact.

Non-regression result: PASS for the measured current-main-to-target diff. The R3 changes are confined to the shared restarter copies plus their behavior test; shared copies remain byte-equal, active caller/binding closure remains complete, and all measured syntax/render predicates are green.

Counterfactual: if either Cyan R2 blocker or the Auron R2 recovery edge remained, at least one of the undefined/empty real renders, cleanup-failure live-path assertion, or persistent recovery-stop-failure harness would fail or show an archive move/start after a failed stop. All three discriminating predicates passed on this exact target.

Unverified: no live validator, real systemd unit, ledger deletion, snapshot endpoint/download, deployment, or real direct-FRA value was exercised in round 3. The full 10,800-second timeout was not waited out. The shell behavior contract is not currently reached by GitHub Actions, so its passing result is target-local evidence rather than a durable CI gate. Legacy specialized init files that already failed dependency resolution on the landing base were not treated as newly passing entrypoints.

Gate: auron
Round: 3
Target: 1f7f29b
Verdict: PASS
Claim: auron-pr411-r3-f5177e9d

[POPPIN-FUMI]

cyan gate claim / round 4 / 対象 e1ae6d9 / cyan-pr411-r4-2ff4ecba

[POPPIN-FUMI]

cyan gate — round 4 — target e1ae6d98c632912749062195032cecb5eb359772

Verdict: PASS. I found no blocking defect in the complete base-to-target diff. The sole round-3 logging blocker and both round-2 transactional/render blockers are closed without regressing the earlier safety, caller, or distribution predicates.

Requirement satisfaction

• The restarter uses stable network/source-scoped staging on the snapshot filesystem and a nonblocking flock; interrupted downloads retain wget -c state per source.
• Mainnet candidates remain direct-FRA → public FRA → public AMS, while testnet/devnet accept only their explicit configured URL. A census found all 19 common-template importers bind a network.
• Full/incremental filename, base/slot relationship, nonempty file, and readable archive checks occur before the first service stop. Failed download/validation/stop paths preserve live snapshot and ledger.
• Promotion remains exact live→rollback and staging→live renames, with no remote/remote nesting. Post-stop failures stop a partially started service before rollback; a failed rollback rename retains the rollback and does not start without a live snapshot. A single interrupted rollback is restored/restarted and the invocation exits before a forward retry.
• Positive catch-up is the commit point. Cleanup is performed only after rollback is disarmed, so a partial cleanup failure leaves the promoted live snapshot in place. Download and catch-up are both bounded by validated positive timeouts.
• Both download calls now use wget -q -c. Independent real GNU wget failure against localhost with the target flags produced nonzero rc and zero stderr bytes, so the configured private path was absent from output. All three distributed templates are byte-identical to the source template.

Independent commands and results

• bash template/2026.6.6.2026/tests/validator-restarter-snapshot.sh: PASS across success, cleanup failure, stop failure, ledger rollback, rollback-rename failure, interrupted recovery, lock exclusion, blank testnet URL, failed download preservation, resume, and GNU wget secrecy control.
• Real ansible.builtin.template render + bash -n for mainnet/testnet/devnet × empty/explicit URL (six variants): PASS.
• Independent partial-cleanup failure injection: PASS; healthy remote remained present and rollback was not re-entered.
• Independent timeout 2s wget -q -c --trust-server-names <localhost-private-path>: expected nonzero rc, stderr size 0, private path absent.
• Common importer census: 19 total, 0 missing network bindings. Common template/dist parity: 9/9 PASS. Targeted Ansible syntax checks: 12/12 PASS. The broken dist Allnodes entrypoint remains absent.
• git diff --check d68e9342e2cae223bdc9878dbfa3bbd752d36b15..e1ae6d98c632912749062195032cecb5eb359772: PASS.
• git merge-tree --write-tree d68e9342e2cae223bdc9878dbfa3bbd752d36b15 e1ae6d98c632912749062195032cecb5eb359772: PASS/conflict-free.
• GitHub Run Tests: SUCCESS on this target; PR state OPEN and mergeability MERGEABLE immediately before review.
• The 32-file added-line scan found no credential/private-key marker, workflow/deploy path, or nonquiet restarter wget call. No live host, secret, deployment, external snapshot endpoint, ledger, or service was accessed.

Non-regression and remaining risk

Against the landing base, no reviewed caller, syntax, distribution parity, security-output, rollback, or CI predicate regressed. Remaining risk is operational rather than a code blocker: this review did not execute the destructive flow on a real validator, download a real snapshot, or prove a live deployment. Those require separately authorized operator evidence and are not inferred from the deterministic harness.

Counterfactual: removing -q, re-arming rollback before cleanup, or rendering an empty conditional branch would respectively reproduce the prior URL disclosure, live-snapshot displacement, or shell parse failure. Predicate check: staged_valid_before_stop && exact_promotion && recovery_fail_closed && committed_live_survives_cleanup && render_total && caller_dist_total && bounded && direct_url_not_logged is true on this target.

Gate: cyan
Round: 4
Target: e1ae6d9
Verdict: PASS
Claim: cyan-pr411-r4-2ff4ecba