Size the agent's heap budget from its container

[shawnburke]

Why

Agent containers looked like they were leaking — growing over time, and one had reached 200MB before it was killed. Three heap profiles taken 15 minutes apart say otherwise:

	13:58:35	14:08:30	14:13:21
Live heap	4.69 MB	4.04 MB	4.70 MB
Live objects	7,089	7,173	7,066
Cumulative alloc	358.84 MB	417.96 MB	531.67 MB

Live heap delta across the full 15 minutes: +9.51 kB, and the only two entries in the diff are single 512kB samples that cancel out. A goroutine profile taken alongside showed 59 goroutines across 25 stacks (largest single stack: 15) — no goroutine leak either.

What actually moved was the allocation rate: 102 KB/s → 400 KB/s, tracking RSS 19MB → 27MB.

That's the runtime behaving as configured. Go doesn't read cgroup limits, so with only GOGC to go on it sizes the heap at 2× live and returns pages lazily. A process with a small live set and a high allocation rate lets RSS ratchet toward the container limit and stay there. Nothing leaks; the container just never gives it back.

What this changes

1. Derive GOMEMLIMIT from the cgroup limit (agent/memlimit)

The agent reads its own cgroup memory limit at startup (v2, falling back to v1) and targets 80% of it. Container sizing becomes the only dial — raise the limit and the working budget follows, with nothing to keep in step.

GOMEMLIMIT is a soft limit: the runtime never refuses an allocation because of it, so this cannot cause an OOM kill that wouldn't have happened anyway. It makes the collector work before the kernel does. An explicit GOMEMLIMIT always wins; no cgroup limit leaves the runtime untouched; a limit too small to target safely (<64MiB after headroom) is refused rather than honored.

2. Bound the frame size the agent accepts (agent/server/grpctunnel, server/config)

Each in-flight call allocates one read buffer of the server-announced frame size (call_handler.go:292), so the agent's ceiling is frame size × in-flight cap. That frame size arrives in ServerHello — it is not configured where the agent runs. Without a bound, a server-side MAX_FRAME_BYTES change silently multiplies the memory ceiling of every agent in the fleet, somewhere their operators can't see or override.

Bounded at both ends: the server now refuses values it can't announce in the int32 the hello carries (previously int32(cfg.MaxFrameBytes) at tunnel/service.go:169 would wrap negative, and agents would silently fall back to their own default), and the agent clamps what it accepts at 4MiB. Frames above the clamp are still carried — just read in more than one chunk.

3. Document the sizing model in the helm chart. Resources are unchanged (256Mi/512Mi, already what shipped).

What this deliberately does not change

MAX_INFLIGHT_REQUESTS is left unclamped. It's a genuine operator dial, and capping it would silently throttle anyone who sized up for throughput — a functional regression, not a memory one. Container sizing stays the dial.

The oversized per-call buffer is left alone. buf := make([]byte, maxFrame) allocates 1MiB to move ~57KB of payload — 96% of all allocation in the sampled window, at 17-19× overhead. Shrinking it to ~64KB would cut churn and drop the worst-case ceiling ~16×, but that's an efficiency change rather than a safety one, and it belongs in its own PR against its own benchmark.

Verification

Unit tests cover cgroup v2/v1 parsing, the max and near-MaxInt64 unlimited sentinels, missing files, v2→v1 fallthrough, explicit-env precedence, and that Configure actually moves debug.SetMemoryLimit rather than only reporting that it would. Frame clamping is tested at and above the ceiling, at the pass-through values, and with a nil logger.

End-to-end in real containers:

512m (chart default)  GOMEMLIMIT set to 409.6MiB (80% of v2 cgroup limit 512MiB)
2g                    GOMEMLIMIT set to 1.6GiB (80% of v2 cgroup limit 2GiB)
128m                  GOMEMLIMIT set to 102MiB (80% of v2 cgroup limit 128MiB)
explicit GOMEMLIMIT   GOMEMLIMIT=100MiB set explicitly; leaving runtime limit alone
no limit set          no cgroup memory limit detected; leaving GOMEMLIMIT unset

go build, go vet, and go test ./... pass clean on both the agent and server modules. (gofmt flags agent/test/load/loadgen.go and server/metrics/metrics.go, both pre-existing and untouched here.)

🤖 Generated with Claude Code