fix(e2e): separate request deadlines and bound recovery of stalled reads

[rspurgeon]

Problem

Two unrelated E2E commands stalled on HTTP reads until the 60-second harness
deadline killed kongctl. Both had six available command attempts but stopped
after one with retry_stop=retry_policy and cause=subprocess_deadline.
This is the second implementation task under #1943.

PR	Scenario / command	Last recorded request	Comparison
#2195	deck/multi-file, 000-plan-create	GET /v2/control-planes during reference resolution	Same command passed in 1.214s on #2197 and 1.481s on #2201
#2201	ai-gateway/vault, 004-get-vault-by-id	GET /v1/ai-gateways, first request of the process	Same command passed in 315ms on #2195 and 311ms on #2197

• PR 2195 shard
• PR 2201 shard
• PR 2197 shard
  also recorded a confirmed connection reset on a gateway-services GET after
  17.617s, with retry_connection_errors=false.

No response or transport error was recorded for the two stalled requests
before termination. Their precise network/service cause remains unknown.

Important correction to the older #1943 discussion:
KONGCTL_E2E_HTTP_TIMEOUT=0s makes the harness omit http-timeout from the
generated CLI profile. Kongctl retains its default 60-second HTTP timeout.
The HTTP request and whole-command budgets therefore compete. Direct harness
HTTP operations have separate configuration semantics and must be documented
separately. Each command has its own deadline; earlier scenarios do not
consume a later command's 60-second allowance.

Proposed change

• Configure an explicit bounded request timeout below the subprocess budget.
  Trial 15 seconds as a starting point, then choose values using live timing
  evidence. Budget for retry attempts, backoff, and other work in the command;
  use targeted longer command limits where warranted.
• Add bounded retry of transient connection/read failures for safe GETs at
  the HTTP boundary, where failed requests are known. Avoid multiplying SDK,
  HTTP-wrapper, and harness retries into an excessive total retry budget.
• Do not enable blanket transport retries for mutations or unconditionally
  retry every killed subprocess. Retain fail-fast behavior for deterministic
  errors and genuine command hangs.
• Make zero/default timeout semantics explicit and consistent in config
  generation, documentation, and diagnostic output.
• Preserve terminal failures and attempts even when later recovery succeeds.

Connection recycling is already enabled in these runs. Increasing shard/job
timeouts or enabling recycling again does not address this failure mechanism.

Acceptance criteria

• A local test server that stalls a GET produces a classified request timeout
  before the harness kills the command; bounded recovery can then succeed.
• A connection reset on a GET can recover within the configured total budget.
• Persistent failures and genuine subprocess hangs stop within documented
  limits; non-retryable errors fail promptly.
• Tests establish that unsafe mutations do not inherit the new read retries.
• Both imperative reads and declarative planner/executor reads are covered.
• Diagnostics report effective request/command limits and retry stop reasons.
• Record live results for the three motivating scenarios and compare timeout
  frequency and elapsed time before/after; retain artifacts on failure.

The assertion task under #1943 remains necessary: command-level recovery of
partially applied mutations must not invalidate scenario expectations.

[github-actions]

🎯 Agentic Issue Triage

Well-scoped implementation task — the second of three under #1943. The core problem is that kongctl's default 60s HTTP timeout and the harness's 60s subprocess deadline are effectively the same budget, so a single stalled GET consumes the entire command allowance with no opportunity for recovery. The harness then records retry_stop=retry_policy (the 90% timeout-consumed threshold in ShouldRetry suppresses further attempts) and the command is lost.

Codebase analysis and implementation notes

Root cause chain

1. CI sets KONGCTL_E2E_HTTP_TIMEOUT=0s (.github/workflows/e2e.yaml:814), which causes the harness to omit http-timeout from the generated profile config (test/e2e/harness/cli.go:774-777).
2. kongctl falls back to its own 60s default (internal/konnect/httpclient default timeout), matching the subprocess deadline exactly.
3. A stalled read saturates the entire budget. When the HTTP request finally times out at ~60s, the subprocess deadline has also expired (or is within the 90% threshold).
4. The harness retry policy suppresses recovery because DefaultRetryTimeoutThreshold = 0.9 in test/e2e/harness/retry.go:28 — if ≥90% of the timeout was consumed, ShouldRetry returns false, setting retry_stop=retry_policy.
5. The production HTTP client does not retry transport errors by default — DefaultRetryConnectionErrors = false in internal/konnect/httpclient/retry_config.go:30. The shouldRetryError logic for ECONNRESET, EOF, and Timeout() on idempotent methods exists (retrying_client.go:239-283) but is gated behind this flag.

Key files to modify

Area	File	What to change
HTTP timeout config	internal/cmd/root/products/konnect/common/common.go:326-335	ResolveHTTPTimeout — ensure explicit bounded default below 60s
Retry connection errors	internal/konnect/httpclient/retry_config.go:29-30	Enable RetryConnectionErrors for safe GETs, or make it configurable per-request
Retry gating	internal/konnect/httpclient/retrying_client.go:239-283	Already has idempotent-method checks — may just need the config flag enabled selectively
E2E HTTP timeout	test/e2e/harness/http_policy.go:15,64-66	DefaultHTTPTimeout = 15s for harness direct calls; consider injecting a bounded value into CLI profiles too
E2E profile injection	test/e2e/harness/cli.go:774-777	When KONGCTL_E2E_HTTP_TIMEOUT=0s, inject an explicit bounded timeout (e.g. 15s) instead of omitting
Subprocess timeout	test/e2e/harness/cli.go:101,161	May need headroom above the HTTP timeout (e.g. 60s subprocess with 15s HTTP timeout allows ~3 retry windows)
Retry threshold	test/e2e/harness/retry.go:28	DefaultRetryTimeoutThreshold = 0.9 — with a 15s HTTP timeout inside a 60s subprocess, this threshold would no longer suppress recovery

Architectural consideration: retry layering

The issue explicitly warns against multiplying SDK, HTTP-wrapper, and harness retries. Current retry layers:

1. Production RetryingHTTPClient (retrying_client.go) — retries on HTTP status codes (403, 429, 5xx) with exponential backoff, max 3 attempts. Transport errors gated by RetryConnectionErrors.
2. E2E harness subprocess retry (scenario/run.go:957-1006) — retries whole CLI commands, max 6 attempts, with pattern matching on combined stderr/stdout.
3. E2E harness HTTP retry (scenario/run.go:309-402) — retries direct harness HTTP calls (create/delete resources), separate from CLI subprocess retries.

Enabling RetryConnectionErrors at layer 1 for safe GETs means layer 2 should see fewer transport-induced failures. The total retry budget needs documentation: e.g., 3 HTTP retries × 15s timeout = 45s worst case per request, fitting within a 60s subprocess deadline with margin.

Suggested implementation checklist

• Configure bounded HTTP request timeout below subprocess budget. Set an explicit 15s (or similar) HTTP timeout for E2E CLI profiles instead of omitting it. Verify the 60s subprocess deadline provides adequate headroom for retries + backoff + other command work.
• Enable transport retry for safe GETs at the HTTP client layer. Either flip RetryConnectionErrors to true for idempotent methods in RetryConfig, or make it configurable so the caller can opt in. The shouldRetryError logic already handles ECONNRESET, EOF, and Timeout() for idempotent methods.
• Ensure mutations do NOT inherit read retries. The existing idempotent-method guard in shouldRetryError (retrying_client.go:246-248) covers this, but add explicit test coverage.
• Add a local test server for stalled-read simulation. Use httptest.Server with a handler that blocks on GET to verify: (a) request timeout fires before subprocess deadline, (b) retry succeeds on a subsequent attempt.
• Add a local test for connection-reset recovery. Simulate ECONNRESET on GET and verify bounded recovery within the configured budget.
• Document effective timeout/retry semantics. Update E2E README and/or harness docs with the timeout layering: HTTP request timeout → retry budget → subprocess deadline → test deadline → job deadline.
• Record and compare live results for the three motivating scenarios. Run deck/multi-file 000-plan-create, ai-gateway/vault 004-get-vault-by-id, and the refactor(declarative): register ordered collection validation #2197 gateway-services case with the new configuration. Compare timeout frequency and elapsed time before/after.
• Preserve terminal failures in diagnostics. Even when recovery succeeds, log the original failure (stall duration, error classification) alongside the successful attempt.

Related issues and context

• Parent issue: Improve E2E timeout and retry behavior for stateful scenarios #1943 — Improve E2E timeout and retry behavior for stateful scenarios
• Sibling task 1: fix(e2e): preserve initial plan coverage across partial sync retries #2202 — Preserve initial plan coverage across partial sync retries (first priority, independent)
• Sibling task 3: task(e2e): capture HTTP connection phases for interrupted requests #2204 — Capture HTTP connection phases for interrupted requests (follow-up, depends on this work)
• Existing assertion work: Surface E2E assertion mismatch details in GitHub Actions summaries #1695 — Display assertion mismatch details (retained as separate task)
• All three sibling tasks are independently reviewable per the Improve E2E timeout and retry behavior for stateful scenarios #1943 implementation breakdown.

Debugging strategies for future stalls

• Enable trace logging (--log-level trace) on failing shards to capture HTTP request/response timing.
• Check TCP_USER_TIMEOUT — the harness supports it on Linux via KONGCTL_E2E_HTTP_TCP_USER_TIMEOUT env var (test/e2e/harness/http_transport_linux.go:13-28). This can detect dead TCP connections at the kernel level faster than application-layer timeouts.
• Connection recycling is already enabled (RecycleConnectionsOnError), so stale connections from prior requests should not carry over. Verify this is active in failing runs.
• Compare timing across PRs — the issue already provides comparison data showing the same commands completing in 300ms–1.5s on other PRs, confirming the stalls are transient network/service issues, not inherent command cost.

Generated by Opened Issue Triage for #2203 · opus46 · 153.2 AIC · ⌖ 9.35 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb