fix(e2e): preserve initial plan coverage across partial sync retries

[rspurgeon]

Problem

The initial all sync can recover from a partially applied execution, then
fail because its assertions expect the original creation plan. Track this as
the first implementation task under #1943. Keep all in blocking CI.

In PR #2197, attempt 1 applied 42 changes, failed gateway-service resolution
with a connection reset, and skipped one dependent change. Attempt 2 succeeded
and created the remaining API implementation. Assertion assert-002 then
expected all-key-auth to appear as CREATE, although it already existed.
The terminal failure was an assertion mismatch, not an execution timeout.

Failed shard

Captured evidence: all/scenario.yaml, 002-sync-all/000-sync-all;
execution attempts 28.629s and 7.297s; terminal phase assertion;
execution retry stop succeeded. The retry plan contained only the API
implementation. Assertions/backoff added approximately 31 seconds.

Proposed change

1. After reset, generate a read-only sync-mode plan and assert the expected
   CREATE actions, resource references, dependencies, and other initial-plan
   contracts against that immutable plan, before any mutation.
2. Execute sync with bounded recovery. Assert successful execution, then
   verify expected remote resources and fields through readbacks, followed
   by the existing no-op convergence and deletion lifecycle checks.
3. Audit the assertions attached to this retried sync. Move assertions that
   depend on the original remote state to the pre-execution plan check.
   Preserve their coverage; do not replace them with fixed/minimum retry-plan
   counts or a broad CREATE-or-UPDATE allowance. Already-created resources
   can be absent from the retry plan entirely.
4. Avoid retry delays when rechecking the same immutable output. Preserve
   polling for assertion sources that actually perform fresh remote reads.
5. Document this authoring pattern for stateful scenarios.

The initial plan is a coverage artifact, not a saved CREATE plan to blindly
reapply after partial success. Normal execution retries should replan from
current state. Full scenario reset/restart is reserved for tests whose actual
contract requires a clean initial state; do not make it the default recovery.

Acceptance criteria

• A deterministic fault after some creates reproduces the old failure and
  passes after recovery with the new assertions.
• Missing desired resources, wrong fields, incorrect initial plan actions,
  failed convergence, and failed deletion still fail the scenario.
• Successful execution alone cannot satisfy the scenario's resource checks.
• Immutable-output mismatches fail without repeated backoff; fresh readback
  assertions retain bounded eventual-consistency polling.
• All attempt artifacts remain available and identify the final assertion
  source. Extend existing test seams rather than adding production fault
  injection solely for this scenario.
• all remains blocking; no automatic assertion-failure shard reruns.

Related: #1695 covers surfacing assertion mismatches in Actions summaries.

[github-actions]

🎯 Agentic Issue Triage

Summary: The all E2E scenario's inline CREATE-action assertions are evaluated against the sync command's stdout after execution (including retries). When the first sync attempt partially applies changes and a retry succeeds with a reduced plan, the assertions still expect the original full CREATE set — causing a deterministic assertion mismatch, not a flaky timeout. This is the first implementation task under #1943.

Root Cause Analysis

The 002-sync-all / 000-sync-all command has ~40 inline assertions using JMESPath against plan.changes in the sync output. Each asserts action: CREATE for a specific resource_type/resource_ref pair.

Failure sequence (PR #2197):

1. Attempt 1: sync applies 42 changes, then hits a connection reset on a gateway-services GET after 17.617s. One dependent change (api_implementation) is skipped.
2. Attempt 2: harness retries the entire sync command. Kongctl replans from current state — the retry plan contains only the remaining api_implementation CREATE.
3. Assertions run against attempt 2's output. assert-002 expects all-key-auth as CREATE, but that resource was already created in attempt 1 and is absent from the retry plan.
4. Terminal failure: assertion mismatch in the assertion phase, not an execution timeout.

The core problem: assertions on plan actions are evaluated against the final successful attempt's stdout, but they encode expectations about the initial clean-state plan. Any partial application invalidates them.

Proposed Implementation Strategy

Based on the issue's proposed changes and the scenario/harness structure:

Phase 1: Pre-mutation plan check

• After 001-reset-org and before the sync in 002-sync-all, add a read-only plan step (e.g., kongctl plan -f config.yaml) that generates the initial plan without executing.
• Move the ~40 CREATE/UPDATE action assertions to this read-only plan output. This captures the "clean state → desired state" contract immutably.
• The plan output is deterministic after a fresh org reset, so these assertions will not be affected by execution retries.

Phase 2: Post-execution verification

• Replace the current plan-action assertions on the sync command with:
  • Execution success assertions: summary.status: success, summary.failed: 0
  • Resource readback assertions: verify expected resources exist and have correct field values via source.get queries (similar to existing commands 001-009, but potentially more granular)
  • Convergence check: retain 003-sync-no-op as-is (already verifies total: 0)

Phase 3: Assertion timing optimization

• Pre-execution plan assertions use immutable output → no backoff/retry needed; set retry.attempts: 1
• Post-execution readback assertions that query Konnect need bounded eventual-consistency polling → keep existing retry defaults
• This addresses acceptance criterion: "Immutable-output mismatches fail without repeated backoff"

Harness changes needed

• The scenario YAML format already supports all required primitives: separate commands per step, source.get assertions, per-assertion retry overrides
• No new harness features are likely needed — this is primarily a scenario restructuring task

Key Files and Code Paths

Area	Path	Relevance
Scenario definition	test/e2e/scenarios/all/scenario.yaml	All 5 steps and ~40 inline assertions to restructure
Test data	test/e2e/scenarios/all/testdata/config.yaml	Full declarative config (~480 lines, all resource types)
Gateway state	test/e2e/scenarios/all/testdata/kong.yaml	decK state file for _deck integration
Domain overlay	test/e2e/scenarios/all/overlays/randomize-custom-domain/ops.yaml	Dynamic custom domain substitution
Assertion execution	test/e2e/harness/scenario/run.go:894	executeAssertions — how assertions are evaluated and retried
CLI retry logic	test/e2e/harness/scenario/run.go:957	runCLIWithRetry — command-level retry with backoff
Retry classification	test/e2e/harness/retry.go	ShouldRetry(), timeout threshold suppression
Scenario types	test/e2e/harness/scenario/types.go	YAML schema: Assertion, Retry, Command
Artifact preservation	test/e2e/harness/scenario/run.go:1032	preserveAttemptArtifacts() for multi-attempt diagnostics

Debugging & Reproduction Strategy

Deterministic reproduction (per acceptance criteria):

1. Introduce a test seam that simulates a fault after N successful creates in the all scenario (e.g., a connection reset after 42 of 43 changes)
2. Verify the current assertions fail with the expected mismatch
3. Apply the restructured assertions and verify they pass after recovery
4. Remove the fault and verify the scenario still passes end-to-end

Artifact inspection for the existing failure:

• Failed shard from PR #2197
• Check all/002-sync-all/000-sync-all/attempts/ for both attempt outputs
• Compare plan.changes arrays between attempt 1 and attempt 2 to confirm the reduced plan

Negative testing (per acceptance criteria):

• Missing desired resources → readback assertions fail
• Wrong field values → readback assertions fail
• Incorrect initial plan actions → pre-mutation plan assertions fail
• Failed convergence → 003-sync-no-op fails
• Failed deletion → 004-delete-all fails

Implementation Checklist

• Add a read-only kongctl plan -f config.yaml command before 000-sync-all in step 002-sync-all
• Move all ~40 CREATE/UPDATE action assertions to the new plan command's output
• Set retry.attempts: 1 on the plan assertions (immutable output, no polling needed)
• Simplify 000-sync-all assertions to execution success + summary checks
• Add granular source.get readback assertions for key resources after sync
• Verify 003-sync-no-op and 004-delete-all / 005-verify-deleted remain unchanged
• Add a deterministic-fault test (using existing test seams) that reproduces the old failure and passes with new assertions
• Verify all attempt artifacts remain available and identify the final assertion source
• Document the "pre-mutation plan check + post-execution readback" authoring pattern
• Run the full all scenario in CI and confirm it passes consistently

Related Issues

• Parent: Improve E2E timeout and retry behavior for stateful scenarios #1943 — Improve E2E timeout and retry behavior for stateful scenarios
• Sibling: fix(e2e): separate request deadlines and bound recovery of stalled reads #2203 — Separate request deadlines and bound recovery of stalled reads (addresses the network-level failures that trigger partial application)
• Sibling: task(e2e): capture HTTP connection phases for interrupted requests #2204 — Capture HTTP connection phases for interrupted requests (follow-up diagnostics)
• Related: Surface E2E assertion mismatch details in GitHub Actions summaries #1695 — Surface E2E assertion mismatch details in GitHub Actions summaries

This issue is independent of #2203 (different failure layer) but both are needed to fully resolve the all scenario instability. #2202 fixes the assertion contract; #2203 fixes the recovery mechanism that exposes the contract violation.

Generated by Opened Issue Triage for #2202 · opus46 · 117.4 AIC · ⌖ 9.46 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb