Skip to content

application: report a pid-isolated worker that died before it was created - #421

Open
andypost wants to merge 1 commit into
masterfrom
fix/310-pid-isolated-worker-record
Open

andypost wants to merge 1 commit into
masterfrom
fix/310-pid-isolated-worker-record

Conversation

@andypost

Copy link
Copy Markdown

Under "isolation": {"namespaces": {"pid": true}} a worker that died in the
CREATING state left main holding the record, the port and the descriptor it made
at WHOAMI time until the prototype exited. Measured under rootlesskit with a
worker killed inside the WHOAMI round trip, five times: main's open descriptors
went 14, 15, 16, 17, 18 without this and stayed at 14 with it. The bound is the
prototype's lifetime, which is the application's, so in practice this is an fd
leak in the most privileged, longest-lived process.

REMOVE_PID cannot carry it -- a pid is its whole payload, and the only pid a
prototype has for such a worker is the namespace-local one, which names an
unrelated process everywhere else. Main is the one process that holds both
names: the WHOAMI arrives with the global pid in SCM_CREDENTIALS and the
worker's namespace-local pid in the message header, and workers are forked with
no clone flags of their own, so that is the number the prototype got from
fork(). Main keeps the pair when the two disagree, and
NXT_PORT_MSG_REMOVE_CHILD_PID names the worker by it.

/proc/<pid>/status NSpid, which #310 proposed, is not needed. The header pid
is only ever looked for among the children of the sender the kernel names, and a
worker naming a live sibling is refused, so a forger can give up its own record
and nothing more. On a platform with no credential nothing is recorded, so
nothing can be resolved.

The slot is appended, the only index-stable edit to nxt_port_handlers_t.

src/test/nxt_main_remove_child_pid_test.c drives both halves, and the
pid-isolated cases of src/test/nxt_proto_creating_wedge_test.c now assert the
message instead of its absence. Five mutations each fail the matching assertion.

Validation: debug and release builds, 0 warnings; build/tests 51 passed (50 on
master); build/unit_port_recv_test 22 unchanged. Two failures in
test_process_abrupt_teardown.py and test_process_teardown_churn.py reproduce
identically on a clean master build here and are unrelated.

Closes #310.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ba675e723f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/nxt_main_process.c Outdated
@andypost
andypost force-pushed the fix/310-pid-isolated-worker-record branch from ba675e7 to 686ba5e Compare September 18, 2026 16:50
@andypost

Copy link
Copy Markdown
Author

@codex review
@claude review

@claude

claude Bot commented Sep 20, 2026

Copy link
Copy Markdown

Claude finished @andypost's task in 5m 22s —— View job


Review of #421 — pid-isolated worker record

  • Gather context (repo guidelines, diff)
  • Review src/nxt_main_process.c (recording + REMOVE_CHILD_PID handler)
  • Review src/nxt_application.c (prototype-side reporting)
  • Review headers and tests
  • Post review

The design holds up: main is indeed the only process that holds both names, the slot is appended so the wire numbering is stable, the teardown sequence in the handler is a faithful copy of the children arm of nxt_main_process_sigchld_handler() (src/nxt_main_process.c:1492-1499) including the unreferenced-nxt_process_close_ports() rationale, and nxt_runtime_process_free() unlinks a child from its parent's queue (src/nxt_runtime.c:1617-1624) so the walk can't reach a freed record. The Codex wire_pid == pid point was fixed correctly in 686ba5e — recording under #if (NXT_USE_CMSG_PID) is the right gate.

One thing survived that fix, on the other half of the pair.

The zero payload resolves against the "unnamed" sentinel

src/nxt_main_process.c:1140

nxt_queue_each(child, &pprocess->children, nxt_process_t, link) {
    if (child->parent_ns_pid != pid) {
        continue;
    }

parent_ns_pid is 0 for "no such name" (nxt_process_new() zallocs, src/nxt_process.c:117), and the handler never rejects a non-positive pid. So a REMOVE_CHILD_PID carrying 0 matches the first unnamed child of the prototype and retires it: nxt_port_remove_notify_others() broadcasts a REMOVE_PID for a live worker's global pid, main closes its end of that worker's port, and the record goes. The recorder guards its own input (wire_pid <= 0, line 934) precisely because zero is the sentinel; the resolver doesn't.

Children with parent_ns_pid == 0 are not hypothetical — naming is skipped for a forged wire_pid <= 0, and skipped again on the sibling-collision refusal at line 939.

The sharper case is the one the function's own comment rules out (src/nxt_main_process.c:1082-1083):

on a platform with no sender credential nothing is ever recorded, so nothing can ever be resolved either

Not with a zero payload. Without NXT_USE_CMSG_PID (macOS, NetBSD, OpenBSD, illumos — named as such at src/nxt_port.c:320) every child is unnamed, and nxt_recv_msg_cmsg_pid(msg) degrades to msg->port_msg.pid, so sender is chosen by the sender. Main's port write end is inherited by every descendant, so any app worker there can send {sender = the prototype's pid, payload = 0} and have main retire an arbitrary live sibling — port closed, router told to remove it. The handler is registered unconditionally, so it is reachable even though no prototype on those platforms ever sends the message.

The precedent for the fix is a few hundred lines away, for the same reason:

/* nxt_port_process_ready_handler(), src/nxt_port.c:664 */
* Reject rather than compare, so that a cmsg_pid of 0 or the -1 that
* the shared-memory queue path leaves in place ... cannot match by accident.

Rejecting pid <= 0 right after the nxt_memcpy() closes it. Wrapping the resolution in #if (NXT_USE_CMSG_PID) as well would make the comment's claim structurally true rather than incidentally so. src/test/nxt_main_remove_child_pid_test.c already has the shape for the case — it asserts the naming side refuses 0 and -1 (lines 287-295) — but no report of 0 is ever driven through the handler, which is why this passes.

Fix this →

Two smaller notes

The forger's reach is slightly wider than the comment says. src/nxt_main_process.c:923 — "a forger can give up its own record and nothing more". A worker that claims a local pid no sibling holds yet keeps it, and when the sibling that really has that number sends its WHOAMI, line 939 refuses the legitimate claim. That sibling stays unnamed, so its death is exactly the #310 leak again. It's bounded — one name per forger, since a second WHOAMI overwrites the first — and the nxt_alert() names both parties, so this is a wording fix rather than a behaviour one.

nxt_exiting is not honoured. nxt_main_process_sigchld_handler() skips nxt_port_remove_notify_others() entirely while exiting (src/nxt_main_process.c:1471-1488) and only closes ports; the new handler always notifies. Almost certainly harmless, but the asymmetry is worth a deliberate decision either way.

Otherwise

  • The ordering argument in nxt_proto_report_child_pid() checks out for the reason it gives and one it doesn't: main's own port carries no shared-memory queue (only app ports and router ports get one), so there is no queue-versus-socket path to reorder the WHOAMI against the report. Might be worth a half-sentence, since the argument would not hold if main's port ever gained a queue.
  • child->stream is 0 on main's record for a prototype's worker (created by nxt_runtime_process_port_create() in the WHOAMI handler, never by a start), so the REMOVE_PID the handler triggers can't be retyped into a stray RPC_ERROR by nxt_router_remove_pid_handler(). Good.
  • The omission of nxt_port_rpc_remove_peer() relative to nxt_port_remove_pid() matches what the sigchld path already does for a prototype's children — consistent, not a gap.
  • The wedge-test inversion (absence → asserted message) is the right way round, and the mutation-testing note is a useful thing to have in the description.

I could not build or run build/tests here — ./configure isn't permitted in this environment — so the above is from reading only; the finding is a missing guard visible in the source rather than an observed failure.
· branch fix/310-pid-isolated-worker-record

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 686ba5e4b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

nxt_main_test_run_name_child(task, proto, child, NXT_MAIN_RCP_NS_PID,
child_pid);

if (nxt_slow_path(child->parent_ns_pid != NXT_MAIN_RCP_NS_PID)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate credential-dependent test assertions

On macOS, NetBSD, OpenBSD, and illumos, NXT_USE_CMSG_PID is false, so nxt_main_process_name_child() is compiled as a no-op and child->parent_ns_pid remains zero. This unconditional assertion therefore makes build/tests fail whenever the test suite runs on those supported platforms; the naming/removal expectations need to be conditional on credential support, with the no-credential path instead verifying that no child name is recorded or resolved.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in b5a23d23. Forcing NXT_HAVE_UCRED and NXT_HAVE_SOCKOPT_SO_PASSCRED off locally, build/tests failed at exactly this line (exit 1, 41 passed) — and five assertions in the file were credential-dependent, not one.

The naming and resolution arms are now under #if (NXT_USE_CMSG_PID), and the #else arm asserts the inert shape you suggested rather than skipping: nothing is named, the report the prototype does send resolves nothing, the record stays and the router is told nothing (exit 0, 50 passed).

src/test/nxt_proto_creating_wedge_test.c is not affected. nxt_proto_report_child_pid() is gated on rt->is_pid_isolated, not on the credential, so the message is sent on every platform, and that test passed in the no-credential build.

Writing those inert assertions turned up a real gap, so this carries a one-line production change too: the handler matched child->parent_ns_pid == pid with no floor, and 0 is how "no name" is stored — so a report naming 0 retired the first child holding no name. It now refuses pid <= 0, mirroring the naming side, with a test case that reddens with credentials on as well as off.

Worth noting for the platform question: rt->is_pid_isolated can only be set where Linux namespaces and CLONE_NEWPID exist, and every such platform has SO_PASSCRED and struct ucred — so no prototype ever sends this message where the credential is missing, and the #310 window cannot open there. That reasoning is now in the handler comment.

Under "isolation": {"namespaces": {"pid": true}} a worker that died in the
CREATING state left the main process holding the record, the port and the
descriptor it had made for it at WHOAMI time, until the prototype itself
exited.  With the start RPC now retired and requests retrying, a worker that
keeps dying in that window costs main one of each per attempt, and main is
the process that can least afford to run out of descriptors.

The gate that left it there is right about REMOVE_PID.  A pid is that
message's whole payload, and the only pid a prototype has for a worker that
has not completed the PROCESS_CREATED handshake is the namespace-local one
it got from fork().  nxt_proc_remove_notify_matrix pairs a dying APP with
MAIN and the router, a prototype's local counter climbs with every worker
straight into the range the daemon's own pids occupy, and
nxt_port_remove_pid() closes the ports of whatever process holds the number
-- so broadcasting it would drop a live sibling's, the router's or the
controller's in-flight work.  Losing requests is worse than the leak.

Main is the one process that can resolve that pid, because it is the one
that holds both names.  A worker's WHOAMI arrives with the kernel-translated
global pid in SCM_CREDENTIALS, which is what main keys its record on, and
with the worker's own namespace-local pid in the message header -- workers
are forked with no clone flags of their own, so that is the same number the
prototype got from fork().  Measured, not assumed: main logs "whoami" from
global pid 29 with header pid 2 while the prototype logs fork() returning 2.
Main keeps the pair, and NXT_PORT_MSG_REMOVE_CHILD_PID then names the worker
by the only pid the prototype has.

The pair is kept whether or not the two numbers differ.  The counters are
independent, so a global pid that has wrapped can land on the small number a
namespace-local one holds, and refusing the pair on that equality would
leave behind exactly the record this path exists to retire.  What the pair
needs is two independently sourced pids, which is what NXT_USE_CMSG_PID
says: without it nxt_recv_msg_cmsg_pid() is the header pid itself, so
nothing is recorded and nothing can ever be resolved.  Nothing is lost by
that: a pid namespace needs Linux unshare() and CLONE_NEWPID, and a platform
with both has SO_PASSCRED and struct ucred, so no prototype ever sends this
message there.  The tests assert that inert shape rather than skipping it.

/proc/<pid>/status NSpid would give main the same number from the kernel
rather than from the sender, at the cost of an open and a parse per start.
It is not needed.  The header pid is only ever looked for among the children
of the sender the kernel names, so a prototype can report its own workers
and nothing else, and a worker that names a live sibling instead of itself
is refused rather than believed -- which leaves a forger able to give up its
own record and nothing more.  Both ends refuse a pid of 0 or less: zero is
how "no name" is stored, which a child keeps wherever the pair was refused,
so a report of it would retire the first such child of the sender.

What is not checked is that the worker is dead, and nothing in main can
check it: the pid names a process in a namespace main does not share, and
the prototype is its only reaper.  A compromised prototype can therefore
orphan a live worker of its own from main's bookkeeping.  That is not a new
trust boundary -- it is the real parent, it already kills its own children
on a failed start, and outside a pid namespace it can say the same thing
with REMOVE_PID -- but it is a new way to desync that bookkeeping, and it is
said where the handler trusts the report.

Main tells the router by the global pid once it has one, so the router sees
what an ordinary REMOVE_PID would have given it.  It has nothing to retire
in practice -- a worker reaches it through the PROCESS_READY this one never
sent -- but it keeps the two paths identical from its side.

The message slot is appended, which is the only index-stable edit to
nxt_port_handlers_t: every _NXT_PORT_MSG_* value is that struct's offset.

Measured under rootlesskit with "pid": true and a worker killed inside the
WHOAMI round trip, five times: main's open descriptors went 14, 15, 16, 17,
18 without this and stayed at 14 with it.

Closes #310.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@andypost
andypost force-pushed the fix/310-pid-isolated-worker-record branch from 686ba5e to b5a23d2 Compare September 20, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

application: under pid isolation a worker that dies before PROCESS_CREATED leaves main holding its record until the prototype exits

1 participant