Skip to content

x86 TCP-recv blocked thread never rescheduled → boot-test hang (pre-existing) #545

Description

@ryanbreen

Symptom

x86 boot test hangs: thread 18 (tcp_socket_test) issues sys_read(fd=22) on a TCP
socket, blocks in TCP_BLOCK waiting for data, and is never rescheduled. The kernel
falls into a permanent idle loop (HLT forever, no further progress, test times out).

Confirmed pre-existing on main at a38ce4a4b2e4f1008f5421a0726067c97e954f46
(3/3 reproductions, identical signature/backtrace) — i.e. independent of and not
introduced by PR #470 PR-2 (fix/470-pr2-leaves@9319d4bb), which is otherwise fully
green (aarch64 prove-to-zero: clean-gate 300 + starved-gate 100, 0 faults, wired
92/92; x86 build clean, 0 warnings; Parallels 3/3). Branch fix/470-pr2-leaves
actually reproduces the hang less often than main (2/3) in local runs — consistent
with a scheduler/wakeup race rather than anything PR-2 touches.

Root cause (mechanism) — CONFIRMED

Scenario (a): the peer's TCP segment is produced but never delivered because
loopback delivery is pull-based, and nothing pumps it once the last runnable user
thread blocks in recv.

The localhost path enqueues packets into LOOPBACK_QUEUE
(kernel/src/net/mod.rs:234, log line "NET: Loopback detected, queueing packet for deferred delivery" at kernel/src/net/mod.rs:837) and only delivers them when
something calls net::drain_loopback_queue() (kernel/src/net/mod.rs:331).

That drain is currently called from specific socket syscalls (sendto/recvfrom/
connect/accept/write — see kernel/src/syscall/socket.rs:463,1475 and
kernel/src/syscall/handlers.rs:440,3341,3532,3710) and from the special
single-test idle loops
(dns_test_only / blocking_recv_test /
nonblock_eagain_test at kernel/src/main.rs:819, :887, :958) — but it is
not called from the general kernel idle loop, idle_thread_fn
(kernel/src/main.rs:1751).

So: once the last user thread blocks in recv waiting on a socket, and the peer's
reply is sitting in LOOPBACK_QUEUE un-drained, the kernel enters the general idle
loop (not one of the special test-only idle loops) and nothing ever pumps the queue
again. The blocked thread is never woken. Permanent hang.

Fix scope — Small / low difficulty

Primary fix: call crate::net::drain_loopback_queue() from the general kernel
idle loop idle_thread_fn (kernel/src/main.rs, ~line 1751), exactly mirroring what
the special test-mode idle loops already do
(kernel/src/main.rs:819, :887, :958). This guarantees pending loopback packets
are delivered whenever the system goes idle, so a blocked recv-waiter gets woken.

Optional hardening: also drain inside the blocking HLT loops themselves — e.g.
TCP recv in kernel/src/syscall/handlers.rs (~lines 1272-1308), and similarly for
TCP accept / UDP recvfrom / TCP connect — so delivery isn't solely dependent on
reaching the top-level idle loop.

Files:

  • kernel/src/main.rs
  • kernel/src/syscall/handlers.rs
  • kernel/src/syscall/socket.rs

Root cause is known; this does not touch any Tier-1/Tier-2 prohibited files, and the
fix is a straightforward addition of an existing, already-used drain call to one more
call site (plus optional hardening at a couple more).

Status

This is a follow-up to be fixed after the current #470 (aarch64 leaf-frame
custody) work settles. Filed per operator ruling on PR #470 PR-2
(fix/470-pr2-leaves) review — the hang is orthogonal to PR-2's leaf-custody changes
and does not block that merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions