Conversation
On the k8s backend PID 1 in every sandbox pod is the agent-env chart's default command, `tail -f /dev/null`, which never calls wait(). Each sandbox exec runs in a fresh process tree, so anything it orphans is reparented to PID 1 and stays there as a zombie, holding its process slot for the life of the pod. Once enough slots are held, fork() returns EAGAIN and every subsequent command in the sample fails, permanently. PortBench hit exactly this in production (epoch-research/PortBench#676); roughly 6% of samples across three eval sets died to it there. Local Docker never shows it: the compose file's `init: true` puts docker-init above tail, and docker-init reaps. Kubernetes has no init equivalent (kubernetes/kubernetes#84210), and our values file is written in chart vocabulary directly, so there the fix has to be the command itself: `/usr/bin/tini -s -- tail -f /dev/null`. `-s` makes tini a subreaper where something else is already PID 1, so the same command is correct on both backends -- both writers now draw it from the shared SANDBOX_COMMAND constant, per that block's no-drift rule. tini is installed in the Dockerfile's base stage rather than a service stage: every service image (agent, agent_corpus, comparator, generate) builds FROM base, and the command is shared by both services in each backend's config, so a service whose image lacked tini would fail to start at all. It is folded into base's existing apt line. This is a workaround; remove it if the upstream issue is fixed: UKGovernmentBEIS/inspect_k8s_sandbox#251 Note this invalidates the base layer, so every image rebuilds from scratch, and ECR tags are immutable and keyed on apn.__version__, so the cluster only picks the fix up at the next version bump.
Both places that link inspect_k8s_sandbox#251 now carry an explicit TODO to remove the tini workaround if it is fixed upstream.
ECR tags are immutable and keyed on apn.__version__, so without a bump the cluster would keep running the pre-tini images.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports epoch-research/PortBench#676 to this repo.
On the k8s backend, PID 1 in every sandbox pod is the agent-env chart's default command,
tail -f /dev/null, which never callswait(). Anything a sandbox exec orphans is reparented to PID 1 and stays there as a zombie holding its process slot for the life of the pod; once enough accumulate,fork()returns EAGAIN and every later command in the sample fails, permanently. PortBench measured this killing ~6% of samples across three eval sets, and diagnosed/verified the fix on a live pod (see the PR above and the upstream issue).Local Docker never shows it because the compose
init: trueis honoured there (docker-init reaps). Kubernetes has no init equivalent, so the fix has to be the pod command itself.The port differs from PortBench in shape because this repo writes the k8s config natively in chart vocabulary instead of letting hawk convert the compose file:
SANDBOX_COMMANDconstant (/usr/bin/tini -s -- tail -f /dev/null), keeping the two artifacts from drifting.-smakes tini a subreaper wherever something else is already PID 1 (the local-docker case), so the one command is correct on both backends.agent,agent_corpus,comparator,generate) build FROM base and the command is shared by both services, so an image lacking tini would fail to start at all. It is folded into base's existing apt line.Relevant note: this invalidates the base layer, so every image rebuilds from scratch. ECR tags are immutable and keyed on
apn.__version__, so the PR includes a bump to 0.1.10 — without it the cluster would keep running the pre-tini images.