task(e2e): capture HTTP connection phases for interrupted requests

[rspurgeon]

Problem

PR #2172 successfully distinguished recovered execution errors from terminal
assertion failures and subprocess deadlines. Its records and existing logs
still cannot locate the network phase of a request interrupted by process
termination. This is a lower-priority follow-up under #1943, complementary to
#1695's assertion summaries.

• PR 2195:
  deck/multi-file ends with khttp-000008, GET /v2/control-planes,
  at 2026-09-16 04:21:46.687 UTC; killed about 59 seconds later.
• PR 2201:
  ai-gateway/vault ends with khttp-000001, GET /v1/ai-gateways,
  at 2026-09-16 15:13:54.727 UTC; killed about 60 seconds later.
• PR 2197:
  all captures a connection reset on a gateway-services GET. The second
  execution succeeds; the eventual assertion failure is a separate event.

The #2201 stall is the first request of a new kongctl process. Reuse of an
earlier connection within that process cannot explain every observed stall.
Packet capture was disabled in all three inspected runs.

Proposed change

• Add opt-in HTTP connection-phase events/timings, using existing logging
  conventions and Go HTTP tracing: DNS, connect, TLS, connection acquisition
  and reuse, request write, and first response byte.
• Persist phase starts/progress as they happen so SIGKILL does not discard
  all evidence of an unfinished request. A final completion-only record is
  insufficient for this use case.
• Correlate request records with the scenario, logical command, execution
  attempt, and artifact-relative paths. Request IDs can restart in each CLI
  process; do not treat them as globally unique.
• Surface effective HTTP and subprocess budgets plus the last unfinished
  request/phase in diagnostics where supported. Keep evidence from recovered
  attempts distinct from the terminal failure.
• Document a targeted diagnostic-run procedure, including the existing
  packet-capture option, and how to compare a failure against a successful
  command without automatically rerunning shards.

Acceptance criteria

• Controlled delayed-response and connection-error cases identify the last
  observed phase and elapsed time, including when the process is terminated.
• Missing events remain unknown; neither a subprocess deadline nor an open
  request is labeled a confirmed server timeout.
• Events contain no credentials, request/response bodies, or sensitive query
  values; endpoint representation follows existing redaction conventions.
• Existing normal-verbosity behavior remains bounded and tracing is opt-in.
• Attempt links distinguish the first failed execution from final success
  and any later assertion mismatch.
• This work does not change retry eligibility or duplicate Surface E2E assertion mismatch details in GitHub Actions summaries #1695's assertion
  diff rendering work.

[github-actions]

🎯 Agentic Issue Triage

Summary: This is a well-scoped follow-up enhancement under #1943 to add opt-in HTTP connection-phase tracing so that when a CLI process is killed mid-request (SIGKILL), the last observed network phase and elapsed time are preserved in diagnostics. Currently, terminated requests leave no evidence of where they stalled (DNS, TLS, connect, first byte, etc.), making it impossible to distinguish server-side stalls from network issues.

This is the third of three decomposed tasks from #1943's September 16 implementation breakdown:

• fix(e2e): preserve initial plan coverage across partial sync retries #2202 (high-priority) — preserve initial plan coverage across partial sync retries
• fix(e2e): separate request deadlines and bound recovery of stalled reads #2203 (high-priority) — separate request deadlines and bound recovery of stalled reads
• task(e2e): capture HTTP connection phases for interrupted requests #2204 (this issue, medium-priority) — capture HTTP connection phases for interrupted requests

Complementary to #1695's assertion summary rendering work; no overlap.

Implementation Notes & Debugging Strategy

Key Go Facilities

The core mechanism is [net/http/httptrace.ClientTrace]((pkg.go.dev/redacted), which provides hooks for:

• DNSStart / DNSDone — DNS resolution phase
• ConnectStart / ConnectDone — TCP connection phase
• TLSHandshakeStart / TLSHandshakeDone — TLS negotiation
• GotConn — connection acquisition (includes Reused flag to distinguish pooled vs new)
• WroteRequest — request write completion
• GotFirstResponseByte — first response byte received

Attach via req = req.WithContext(httptrace.WithClientTrace(ctx, trace)) in the HTTP client layer.

Persistence Design

The critical requirement is that phase events are persisted as they happen (not buffered until completion), so SIGKILL mid-TLS-handshake still leaves evidence of the DNS and connect phases completing. Options:

• Write-ahead log to the artifact directory (append-only, fsync per phase)
• Structured JSON lines file alongside existing khttp-* request records
• Leverage existing logging conventions with a dedicated opt-in log level or structured field

Correlation Requirements

Request records need to carry:

• Scenario name, step, command identity
• Execution attempt number (distinguishing recovered vs terminal)
• Artifact-relative path
• The khttp-NNNNNN request ID (noting these restart per CLI process and are not globally unique)

What NOT to Do

Per the acceptance criteria:

• Don't log credentials, request/response bodies, or sensitive query values
• Don't label an open request or subprocess deadline as a "confirmed server timeout"
• Don't change retry eligibility
• Don't duplicate Surface E2E assertion mismatch details in GitHub Actions summaries #1695's assertion diff rendering
• Keep normal-verbosity output bounded; tracing must be opt-in

Suggested Implementation Checklist

• Add httptrace.ClientTrace hook wiring in the HTTP client layer (likely internal/konnect/httpclient/)
• Make tracing opt-in via a flag, env var, or log level (e.g., --trace-http or KONGCTL_HTTP_TRACE=1)
• Implement write-ahead phase event persistence (append per phase, not buffered)
• Add correlation fields: scenario, step, command, attempt number, artifact path
• Surface effective HTTP timeout and subprocess budget in diagnostic output
• Handle connection reuse detection (GotConn.Reused) — the fix(declarative): show styled nested field changes in text diffs #2201 stall was the first request in a new process, ruling out reuse as the cause
• Add redaction for endpoint representation (follow existing conventions)
• Distinguish recovered attempt evidence from terminal failure evidence
• Add unit tests for phase event recording and redaction
• Add a controlled delayed-response test case that verifies last-observed phase and elapsed time
• Add a connection-error test case
• Document the diagnostic-run procedure (including existing packet-capture option)
• Document how to compare a failure against a successful command

Related Issues & Context

Issue	Relationship	Status
#1943	Parent tracking issue for E2E timeout/retry improvements	Open
#2202	Sibling: preserve initial plan coverage across partial sync retries	Open (high-priority)
#2203	Sibling: separate request deadlines and bound recovery of stalled reads	Open (high-priority)
#1695	Complementary: surface assertion mismatch details in Actions summaries	Open
#2058	Related: prototype replay-backed PR scenario validation	Open

Motivating CI Evidence

Three recent PR runs demonstrated the diagnostic gap:

• PR refactor(simplify): streamline vault matching and persistent flags #2195 (deck/multi-file): GET /v2/control-planes had no completion record; killed at 60s
• PR fix(declarative): show styled nested field changes in text diffs #2201 (ai-gateway/vault): GET /v1/ai-gateways stalled on the first request of a new process
• PR refactor(declarative): register ordered collection validation #2197 (all): connection reset on gateway-services GET after 17.6s; second execution succeeded

In all cases, packet capture was disabled and no transport-level evidence survived the termination.

Relevant Code Areas

• internal/konnect/httpclient/ — HTTP client setup; where httptrace hooks would be wired
• test/e2e/harness/ — E2E test harness, request recording, artifact persistence
• test/e2e/scenarios/ — scenario definitions referenced in the motivating evidence
• internal/log/ — structured logging; connection-phase events should follow existing conventions

Labels applied: triaged (existing: enhancement, e2e, medium-priority — all appropriate)

Priority assessment: Medium priority is correct. This is an observability/diagnostic improvement that depends on insights from #2203's request deadline separation. The higher-priority siblings (#2202, #2203) should land first as they address the active failure mechanisms, while this issue adds the diagnostic depth to prevent future blind spots.

Generated by Opened Issue Triage for #2204 · opus46 · 134.7 AIC · ⌖ 28.6 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb