Improve E2E timeout and retry behavior for stateful scenarios

[rspurgeon]

Problem

The E2E scenario shard containing test/e2e/scenarios/all is sometimes perceived as timing out because all performs a large declarative lifecycle against one Konnect organization. Recent CI evidence suggests that disabling the scenario or adding artificial delays would address the symptom poorly:

• PR task(sdk): update sdk-konnect-go to v0.62 patch #1942 ran all successfully twice at about 21.7 seconds.
• Other recent failed shard runs completed all in roughly 33-35 seconds, then timed out in later scenarios.
• A historical all failure took about 74 seconds but was not a timeout. The first stateful sync partially applied changes, the harness retried the whole command, and CREATE-action assertions no longer matched the converged plan.
• all already sets maxConcurrency: 1, so declarative mutations are serialized.

The harness currently gives each CLI subprocess a blanket 60-second timeout. CI also sets KONGCTL_E2E_HTTP_TIMEOUT=0s. For CLI subprocesses this causes the harness to omit http-timeout from the profile, leaving kongctl's default 60-second HTTP timeout; direct harness HTTP operations have separate semantics. A near-full subprocess timeout is intentionally not retried. This can make a single stuck request consume the entire command deadline and provides limited information about the underlying cause.

Proposed improvements

• Keep scenarios/all enabled because it exercises cross-resource planning, references, execution, no-op convergence, and deletion.
• Preserve initial CREATE-action coverage in a read-only plan check before mutation; use stable convergence, summary, and readback assertions after execution retries.
• Separate the effective HTTP request deadline from the 60-second subprocess timeout and budget for bounded read recovery.
• Prefer bounded per-request HTTP deadlines and appropriate transport retries over artificial sleeps.
• Support targeted longer command timeouts for unusually large sync/delete operations when CI evidence demonstrates they need additional time.
• Improve failure reporting so the scenario, command, timeout layer, retry classification, and elapsed duration are immediately visible.
• Measure failure frequency after these changes before considering scheduled/advisory execution.

Acceptance criteria

• A partially successful stateful command can be retried without causing all to fail solely because its plan actions changed from CREATE.
• E2E HTTP and subprocess timeout behavior is documented and internally consistent.
• Timeout diagnostics identify whether the failure came from an HTTP request, CLI subprocess deadline, Go test deadline, or Actions job deadline.
• No unconditional sleeps are added.
• scenarios/all remains part of blocking CI unless measured post-change instability justifies quarantine.

Evidence

• PR task(sdk): update sdk-konnect-go to v0.62 patch #1942: task(sdk): update sdk-konnect-go to v0.62 patch #1942
• Successful scenario runs:
  • https://github.com/Kong/kongctl/actions/runs/32493224185
  • https://github.com/Kong/kongctl/actions/runs/32493500887
• Historical run showing retry-sensitive all assertions and an unrelated later timeout:
  • https://github.com/Kong/kongctl/actions/runs/32009270550

Implementation breakdown — September 16, 2026

PR #2172's capture now confirms both failure mechanisms across three PRs.
Implement the following independently reviewable tasks:

• fix(e2e): preserve initial plan coverage across partial sync retries #2202 — Preserve initial plan coverage across partial sync retries.
  First priority: move original-state plan assertions before mutation; verify
  remote resources and convergence after recovery without losing coverage.
• fix(e2e): separate request deadlines and bound recovery of stalled reads #2203 — Separate request deadlines and bound recovery of stalled reads.
  First priority: make request failures observable before subprocess death
  and permit bounded recovery of safe reads within the command budget.
• task(e2e): capture HTTP connection phases for interrupted requests #2204 — Capture HTTP connection phases for interrupted requests.
  Follow-up: distinguish DNS/connect/TLS/request/response stalls and link
  recovered attempts separately from terminal failures.

Keep #1695 as the existing task for displaying assertion mismatch details.

New evidence

• PR refactor(declarative): register ordered collection validation #2197, all: a gateway-services GET reset after 17.617s. The first
  sync applied 42 changes; the second succeeded with only the remaining API
  implementation in its plan. CREATE assertions then failed on absent entries.
  Shard
• PR refactor(simplify): streamline vault matching and persistent flags #2195, deck/multi-file: control-plane GET with no recorded completion;
  killed at 60s, one of six command attempts used, retry policy stopped recovery.
  Shard
• PR fix(declarative): show styled nested field changes in text diffs #2201, ai-gateway/vault: first GET in a new kongctl process stalled
  while listing AI Gateways, before retrieving the vault. Same 60s deadline
  and retry suppression. Prior name-based retrieval succeeded immediately.
  Shard

Comparisons: the multi-file plan passed in 1.214s on #2197 and 1.481s on
#2201; vault retrieval by ID passed in 315ms on #2195 and 311ms on #2197.
The precise network/service cause of the two stalls remains unconfirmed.

Corrections to earlier implementation suggestions

• CLI HTTP timeouts were not disabled by the CI 0s setting: profile
  omission retained the CLI's 60-second default. Direct harness HTTP helpers
  must be considered separately.
• Each subprocess receives its own deadline. Earlier scenarios do not consume
  a later command's 60-second allowance; overall test/job budgets are separate.
• Fixed/minimum retry-plan counts are also invalid after partial application.
  Already-created resources can disappear from the plan, not just become UPDATE.
• Preserve CREATE coverage before mutation. Do not merely loosen assertions
  or blindly reapply a saved CREATE plan after partial success.

Immediate PR handling

A bounded manual rerun of the failed shards on #2195, #2197, and #2201 is
reasonable given the captured failures and successful comparisons. Require
normal E2E coverage/status verification to pass before merging. If a rerun
fails, retain and compare its new evidence instead of repeating until green.
These follow-up issues do not waive checks or authorize automatic retries.

[github-actions]

🎯 Agentic Issue Triage

Well-scoped issue targeting a real CI reliability gap: the interaction between disabled per-request HTTP timeouts, a blanket 60s subprocess deadline, retry-sensitive CREATE assertions, and limited timeout-layer diagnostics. The evidence from PR #1942 and historical runs clearly demonstrates the failure modes.

📍 Codebase orientation — key files and current behavior

Area	File	Key lines
Subprocess timeout (60s default)	test/e2e/harness/cli.go	L97, L157, L392-396, L423-425
HTTP timeout policy & KONGCTL_E2E_HTTP_TIMEOUT	test/e2e/harness/http_policy.go	L15 (default 15s), L64-65, L86-101, L116-123
CI disables HTTP timeout (0s)	.github/workflows/e2e.yaml	L679
Retry logic & 90% timeout threshold	test/e2e/harness/retry.go	L17-28, L118-169 (especially L149-155)
CLI retry wrapper	test/e2e/harness/scenario/run.go	L897-938, L570
Per-command timeout override	test/e2e/harness/scenario/run.go	L940-958 (commandTimeout())
scenarios/all — CREATE assertions (34 of them)	test/e2e/scenarios/all/scenario.yaml	L86-280
maxConcurrency: 1 in all	test/e2e/scenarios/all/scenario.yaml	L33
Retry artifact preservation	test/e2e/harness/scenario/run.go	L964-1015

🔍 Root cause analysis

There are two distinct failure modes described in the evidence:

Mode 1 — Retry-sensitive assertions (the 74s historical failure):
The first sync partially applies (some resources created), then the harness retries the entire command. On retry, already-created resources produce UPDATE plans instead of CREATE, so the 34 action: CREATE assertions fail. This is NOT a timeout — it's a semantic mismatch between retry behavior and assertion specificity.

Mode 2 — Timeout cascade (the 33-35s runs):
With KONGCTL_E2E_HTTP_TIMEOUT=0s, a single stuck HTTP request can block until the 60s subprocess deadline. The 90% threshold (DefaultTimeoutRetryThreshold = 0.9, i.e. 54s) means a request that takes 55s will consume the subprocess and NOT be retried. When all finishes near the deadline, later scenarios in the same shard have reduced time budget. The timeout diagnostic only says "command timed out" — it doesn't identify which HTTP request or timeout layer caused it.

🛠️ Implementation checklist

Based on the acceptance criteria and codebase structure:

• Replace CREATE-action assertions with convergence assertions (scenarios/all/scenario.yaml L86-280)

  • Assert summary.failed: 0 and summary.status: success (already done at L83-85)
  • Assert expected resource count in the plan rather than per-resource action: CREATE
  • Add a no-op re-sync step that asserts summary.total: 0 (convergence check — partially exists at step 003-verify-noop, L282-313)
  • Consider readback assertions (GET resource, verify fields) instead of plan-action assertions

• Reconcile HTTP and subprocess timeouts (http_policy.go, cli.go)

  • Set a bounded per-request HTTP timeout (e.g. 30s) instead of 0s in CI
  • Ensure the subprocess timeout is meaningfully larger than (max_resources × http_timeout) / concurrency — with maxConcurrency: 1 and ~40 resources, that's ~20 minutes at 30s each in the worst case, but typical is much less
  • Consider a per-command timeout: override in scenario.yaml for all (already supported via commandTimeout() at scenario/run.go L940-958) rather than raising the global 60s

• Improve timeout diagnostics (cli.go, retry.go)

  • When TimedOut is set, log which timeout layer fired (HTTP request, subprocess context, Go test deadline, Actions job)
  • Include elapsed duration per HTTP request in trace/debug logging
  • Surface the retry classification (ClassifyRetry() at retry.go L181-223) in the artifact output

• Preserve retry context in artifacts (scenario/run.go)

  • preserveAttemptArtifacts() (L964-1015) already copies per-attempt artifacts; verify this includes the plan JSON so a partial-apply → retry sequence is diagnosable
  • Add scenario name and elapsed duration to the timeout warning at cli.go L423-425

• Measure post-change stability

  • Track all pass rate across N CI runs after changes land
  • Compare against the current baseline before deciding on quarantine

🔗 Related issues

• Surface E2E assertion mismatch details in GitHub Actions summaries #1695 — Surface E2E assertion mismatch details in GitHub Actions summaries (open, triaged) — complementary: better failure reporting in CI summaries would directly help with the "improve failure reporting" acceptance criterion here.
• Task: Defer E2E scenario suites until pull requests are ready for review #1741 — Defer E2E scenario suites until pull requests are ready for review (open, triaged) — related: if all stability improves, the urgency of deferring E2E suites decreases.

Neither is a duplicate — #1695 focuses on Actions summary rendering, #1741 on CI triggering policy, while this issue targets the underlying timeout/retry/assertion mechanics.

💡 Implementation notes and suggestions

On replacing CREATE assertions: The simplest migration is to keep the step 002-sync-all but replace the 34 per-resource action: CREATE checks with:

1. summary.total >= expected count (proves the planner found all resources)
2. summary.failed: 0 (proves execution succeeded)
3. Step 003-verify-noop already asserts convergence — strengthen it to check summary.total: 0 (no pending changes)
4. Optionally add readback assertions in a new step that GETs key resources and verifies field values

On HTTP timeout reconciliation: The current setup (KONGCTL_E2E_HTTP_TIMEOUT=0s + 60s subprocess) means a single request can silently consume the entire deadline. A bounded per-request timeout (e.g. 15-30s) with 1-2 transport retries is safer — it fails fast on stuck connections while still tolerating transient slowness. The DefaultHTTPTimeout of 15s in http_policy.go is already a reasonable starting point; the CI override to 0s could simply be removed.

On the 90% threshold: The DefaultTimeoutRetryThreshold = 0.9 in retry.go L21 means commands consuming ≥54s of a 60s deadline won't be retried. This is correct behavior (retrying a near-timeout is likely to timeout again), but it means the first attempt must succeed. With bounded HTTP timeouts and transport retries, this becomes less of a concern.

Labels applied: triaged, medium-priority (in addition to existing e2e, code improvement).

@rspurgeon — this is a thorough and well-evidenced issue. One clarifying question: for the "bounded per-request HTTP deadlines" improvement, do you envision this as restoring the default 15s DefaultHTTPTimeout in CI (removing the 0s override), or introducing a new timeout tier specific to declarative sync operations?

Generated by Opened Issue Triage for #1943 · opus46 · 144.7 AIC · ⌖ 28.4 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb