ateom-gvisor: measure the sandbox cgroup for GetWorkloadStats

[Tim Bai (baizhenyu)]

Second of the three PRs for Phase 0 of #550, stacked on #667 and to be merged after it.

Review the second commit only. The base branch lives on my fork, so GitHub cannot use it as a base here and this PR is opened against main; it therefore shows #667's commit as well. Once #667 merges I will rebase and this collapses to a single commit. Direct link to the commit under review: 36879514.

Fills in the measurement half of GetWorkloadStats for the gVisor runtime, so it returns real numbers instead of Unimplemented. The micro-VM runtime keeps its stub until the guest-agent read lands in PR 3, which closes #594.

Where the numbers come from

/sys/fs/cgroup/pause, the sandbox's cgroup v2 leaf.

gVisor runs every container of a sandbox inside one host process — the sentry — and runsc places that process in the cgroup of the container that created the sandbox. That is pause, the first container RunWorkload and RestoreWorkload create. The actor's own containers get leaves too, but their memory and CPU are the sentry's and are charged there instead. This is why a sample is attributed to the actor and not to a container, as the proto already says.

The path follows the "/" + containerName convention runsc.ensureContainerCgroupsPath writes into the OCI spec, resolved against the pod's own cgroup scope that setupCgroupDelegation prepares (the worker runs in a private cgroup namespace, so /sys/fs/cgroup is the pod's cgroup, not the host root).

The one thing I could not verify locally: that the sentry really lands in that leaf on a live node is a runsc placement behavior. The convention is derived from our own spec-writing code, but if someone can confirm against a real worker before PR 3 builds on it, that would be worth doing.

The read

New cmd/ateom-gvisor/internal/cgroupstats. It takes the cgroup directory as an argument rather than reaching for an absolute path, so the parsing is testable from a fixture tree without root or a live sandbox — and it carries no build tag, so unlike the rest of cmd/ateom-gvisor those tests run on every platform.

It fails only when memory.current is missing or unparseable, wrapping fs.ErrNotExist in the first case so the handler can tell "the sandbox is gone" (FailedPrecondition) from "the format is not what we parse" (Internal).

Every other file degrades to zero on its own field, because each has a legitimate reason to be absent on a healthy node: memory.peak predates kernel 5.19, and setupCgroupDelegation enables controllers best-effort, so a cgroup with memory but no cpu is reachable. Reporting no memory numbers because the node could not report CPU seemed like the wrong trade — pushback welcome if you'd rather it be all-or-nothing.

Working set is memory.current − memory.stat:inactive_file, saturating at zero rather than wrapping. The two files are read a moment apart and are not a consistent snapshot, so inactive_file can legitimately exceed the memory.current read just before it; on uint64 the naive subtraction gives ~1.8e19 instead of ~0.

Locking

AteomService.activeActor becomes an atomic.Pointer. The three lifecycle RPCs still hold AteomService.lock for their whole bodies and keep doing so — the change is for the reader.

GetWorkloadStats is polled on a timer for a workload's whole lifetime, while lock is held across entire boots and checkpoints (including the readiness wait). A lock-guarded read would park each poll behind a multi-second runsc call and let pollers pile up. Holding the lock across the cgroup read would be worse in the other direction: a CheckpointWorkload queued behind telemetry. The field is only ever assigned or cleared as a whole pointer, never mutated in place, which is what atomic.Pointer is for.

The handler reloads the pointer after the read and compares identity, so a checkpoint plus a fresh run completing underneath the read is reported as a failed precondition rather than misattributed to the wrong actor.

TestGetWorkloadStatsDoesNotTakeLock pins this: it holds s.lock across the call, so a handler that ever reaches for the lock deadlocks.

Panic recovery

Both ateoms gain ateinterceptors.RecoveryUnaryInterceptor, chained ahead of InternalServerUnaryInterceptor.

grpc-go serves each RPC on its own goroutine and does not recover handler panics, and the Go runtime kills the process on an unrecovered panic in any goroutine. grep -rn "recover()" finds nothing in our non-test code, so today a nil dereference in any handler ends every other RPC the ateom is serving — including an in-flight checkpoint, and the worker's socket with it.

That gap predates this change. I've included it here because adding a caller that polls on a timer for the life of every workload is what makes it reachable, and the stats paths are exactly the shape that hits it: the "no workload here" state is a nil pointer, and cgroup files are parsed line by line. Happy to split it out if you'd rather review it separately. Scoped to the two ateoms; the other five gRPC servers in the repo have the same gap and are worth a follow-up.

Follow-up worth filing separately

The readiness wait is held under s.lock in all four boot paths (main.go:364, :639, and the micro-VM equivalents). It is not a runsc subcommand, so it sits outside what the lock's own comment claims to protect, and it is the longest hold in the file. Shrinking it changes lifecycle behavior, so it does not belong in a telemetry PR — but it is the highest-leverage locking fix here.

Testing

• cgroupstats: nine-case table over fixture trees — all files present, missing memory.peak, missing cpu.stat, missing memory.stat, memory.stat without inactive_file, inactive_file above memory.current, all-zero, malformed/blank/over-long lines, unparseable optional files — plus missing-cgroup and malformed-memory.current cases that pin which one matches fs.ErrNotExist.
• GetWorkloadStats: happy path against a fixture cgroup root, a five-case error table (empty actor_uid → InvalidArgument; available, UID mismatch, and vanished cgroup → FailedPrecondition; malformed cgroup → Internal), and the no-lock regression test.
• RecoveryUnaryInterceptor: recovers a nil-deref panic into Internal, keeps the panic text out of the response and in the log with a stack, and passes success / status-error / plain-error through untouched.
• All eight runnable local verifiers pass. shellcheck.sh needs docker, which is not available here; no shell scripts changed.

The cmd/ateom-gvisor handler tests are //go:build linux, so locally they are only compile-checked (GOOS=linux go test -c); CI's run-tests executes them.