Skip to content

DAOS-623 test: instrument NLT fault-injection sweep hang - #18756

Draft
mjmac wants to merge 15 commits into
masterfrom
mjmac/DAOS-623-nlt-fi-hang
Draft

DAOS-623 test: instrument NLT fault-injection sweep hang#18756
mjmac wants to merge 15 commits into
masterfrom
mjmac/DAOS-623-nlt-fi-hang

Conversation

@mjmac

@mjmac mjmac commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Draft — diagnostic instrumentation, not for review or merge.

Purpose: capture evidence for the intermittent NLT hang that has aborted the NLT stage
~13 times in 5 days across 10 unrelated PRs (#18709, #18731, #17550, #18707, #18729,
#18718, #18733, #18737, #18740, #18741). Each occurrence costs the full stage timeout and
discards all artifacts, so no build so far has produced any diagnostic data.

What hangs

test_alloc_fail_cont_create in the NLT stage (which runs mode all). dfuse stops
answering FUSE requests it has already accepted; every client using the mount then blocks
in the kernel indefinitely, and NLT's sweep loop waits forever on children that can never
finish. Reproduced twice on a 16-core cloud client; intermittent (a second cluster ran the
same sweep 9 times without hanging).

The cause is not known. The obvious explanation — dfuse's threads all tied up
servicing stalled clients — is contradicted by the only thread stacks captured so far
(from a --thread-count 2 wedge, so not necessarily the same bug): the dfuse worker was
idle in read() while the kernel held 5 queued requests, i.e. requests accepted and never
answered rather than threads busy. Getting the equivalent stacks from a
default-configuration hang is exactly what this branch is for.

What this branch adds

  • A stall watchdog in the fault-injection sweep loop: if no child completes for
    NLT_FI_STALL_SECS (default 300), dump NLT thread stacks, per-child /proc state
    (wchan/syscall/State) and gdb backtraces of dfuse, the agent, the engine and the
    stalled children — then kill the wedged children so the run fails fast with artifacts
    intact instead of being aborted with none. Bounded by a wall-clock deadline, writes to a
    file as well as stdout, skips targets gdb cannot attach to, resumes any target a debugger
    left stopped, never blocks on children that outlive SIGKILL, and reports its own failure
    rather than aborting silently.
  • A bounded DFuse.stop() query. Teardown currently queries the very mount that may be
    wedged with no timeout, which is why a hang consumes the whole stage rather than one test.
  • Fail loudly when mode fi/all is requested but the build has no fault injection
    compiled in. Today that prints one line and exits green with the requested tests
    silently skipped.
  • Make dfuse's log level and thread count settable for FI runs; the sweep pins dfuse to
    WARN, so the prime suspect in a dfuse-side wedge logs nothing at all.

NLT-repeat: 5 is set to get multiple chances at an intermittent failure per build. Skip
pragmas restrict this to builds plus NLT.

Independent of this investigation, the watchdog and the bounded teardown would convert any
future NLT hang from a lost CI run into a fast, self-documenting test failure. If they look
useful beyond the debugging, they are worth proposing separately against master.

mjmac added 5 commits July 30, 2026 11:38
@mjmac
DAOS-623 test: diagnose NLT fault-injection sweep hang
797b7b3 
The alloc-fail sweep spawns children with piped stdout/stderr but only
drains the pipes after exit, and its completion loop has no timeout: a
child that never exits (blocked writing a full pipe, or genuinely
wedged in the client) leaves NLT spinning silently until the Jenkins
stage timeout kills the run with no artifacts. This signature has been
killing the majority of PR-gate NLT runs across unrelated PRs.

Add a stall watchdog: after five minutes with no completions, dump
NLT thread stacks, /proc wchan/syscall state and gdb backtraces for
the stuck children and daemons to stdout so the evidence lands in the
console even if the stage is later aborted, then kill the children so
the run fails fast and artifacts still archive. Run NLT unbuffered so
the console shows exactly where output stops.

NLT-repeat: 5
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: fail NLT when requested fault injection is unavailable
d930345 
A release-type build lacks the fault injection feature, and NLT
responded to an explicit fi/all mode request by printing one line and
exiting green with the requested tests silently skipped. Surfaced by a
cloud NLT run that reported 93/93 passed while never executing the
sweeps it was asked to run.

Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: bound the fault-injection stall dump
14bc05e 
The first version of the stall dump detected correctly and then wedged
inside itself: gdb cannot attach to a process in uninterruptible sleep
or one another debugger already holds, and a killed gdb leaves its
target stopped so a later kill never lands. The dump also blocked
reaping children that survive SIGKILL, and wrote only to a stdout that
a hostage pipeline can swallow. A cloud run detected its stall exactly
on time and still produced no usable evidence.

Give the dump a wall-clock deadline, collect /proc state before
anything that can block, skip and report unattachable targets, resume
any target a debugger left stopped, dump daemons before children, write
to a file as well as stdout, and report rather than wait on children
that outlive SIGKILL.

Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: make dfuse debug level and thread count settable for F…
252f154 
…I runs

The fault-injection sweeps drive every request through dfuse, but the
sweep sets dfuse's log mask to WARN, so a dfuse that stops answering
leaves an empty log behind. The sweep also runs three quarters of the
host's cores in parallel against a dfuse fixed at four threads, which
is the suspected wedge; varying it is how that gets confirmed or ruled
out.

Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: make the stall dump self-reporting and file-first
670b7d7 
A cloud run detected its stall on time, printed the banner, and then
produced nothing further with no indication of why - the worst possible
outcome for a diagnostic. Two causes, both fixed here: the dump wrote
to stdout before its local file, so anything blocking the consumer of
stdout could swallow the evidence; and an exception anywhere in the
dump propagated into the test flow, taking the run down instead of
reporting itself.

Write the local file first, tolerate a failing stream, and catch
everything: a dump that cannot complete now says so and leaves the run
to fail on its own terms.

Also stop a timed-out JSON command from raising a decode error that
hides the timeout that caused the empty output.

Signed-off-by: Michael MacDonald <github@macdonald.cx>
@github-actions

github-actions Bot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown

Ticket title is 'Generic ticket for minor code cleanup and improvement'
Status is 'Resolved'
Labels: 'request_for_2.6.5,request_for_2.6.6,request_for_2.8,scrubbed_2.6.5'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-623

@github-actions github-actions Bot added the priority Ticket has high priority (automatically managed) label Jul 30, 2026
@daosbuild3

daosbuild3 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/1/execution/node/868/log

@mjmac mjmac changed the title DAOS-623 test: instrument the NLT fault-injection sweep hang (DRAFT - do not review) DAOS-623 test: instrument NLT fault-injection sweep hang Jul 30, 2026
@mjmac
DAOS-623 test: dump thread stacks to a real file descriptor
31f6e02 
The stall dump asked the traceback dumper to write to sys.stdout, which
this script replaces with a wrapper that captures per-thread output and
has no fileno(). The resulting AttributeError escaped a too-narrow
except clause and took the whole dump down with it, which is why two
earlier captures produced nothing past the banner.

Write to the real stdout underneath the wrapper, skip any stream that
cannot supply a descriptor, and never let this step abort the dump.

Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
mjmac force-pushed the mjmac/DAOS-623-nlt-fi-hang branch from f29d915 to e3bad76 Compare July 30, 2026 19:38
@mjmac
DAOS-623 ci: carry CI pragmas on the branch tip
f74506c 
The pragmas were only on the first commit of the series and were not
picked up: NLT ran once per build rather than repeatedly, which is a
fraction of the chances of catching an intermittent hang. Repeat them
here, on the commit CI actually reads.

Ten repeats: the observed hang rate is roughly one run in seven, so a
single build of ten runs is a better use of one CI node than several
builds of fewer.

NLT-repeat: 10
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
mjmac force-pushed the mjmac/DAOS-623-nlt-fi-hang branch from e3bad76 to f74506c Compare July 30, 2026 22:26
@mjmac
DAOS-623 test: exercise the stall watchdog deliberately
1e4d15e 
The reporting path for a hang was only ever tested by the hang itself,
which happens on roughly one run in seventeen. That is a poor way to
find out whether the reporting works: two captures were lost to bugs in
the dump before anyone could read them.

Wedge one sweep iteration on demand so the whole path - detect, dump
stacks, kill, continue, fail with artifacts intact - runs on every
build. The Jenkinsfile setting is temporary and belongs to this
diagnostic branch only.

Signed-off-by: Michael MacDonald <github@macdonald.cx>

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
mjmac force-pushed the mjmac/DAOS-623-nlt-fi-hang branch from b306f51 to 1e4d15e Compare July 30, 2026 22:46
@mjmac
DAOS-623 test: release children stuck in unanswered FUSE requests
3015a03 
A CI run showed the recovery path does not recover. Children wedged in
the sweep sit in an unanswered FUSE request, which is uninterruptible
sleep: SIGKILL is recorded but never delivered, because delivery waits
for a request the wedged mount will never answer. The sweep therefore
found them still running, stalled again two minutes later, and repeated
until the stage timed out - the exact outcome this mechanism exists to
prevent, and it went unnoticed because the only previous occurrence
died before reaching this point.

Abort the backed-up FUSE connection when children survive SIGKILL in a
FUSE wait, which fails their outstanding requests and lets them die.

Also capture stacks with eu-stack, falling back to gdb: gdb spent
longer loading symbols for these binaries than a bounded dump can
afford and was killed part-way through, so the one dump that reached
dfuse recorded a thread list and no frames.

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@daosbuild3

daosbuild3 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/8/execution/node/535/log

mjmac added 3 commits July 31, 2026 09:11
@mjmac
DAOS-623 test: wedge once per run and stop a sweep whose mount was ab…
178ff49 
…orted

The deliberate wedge matched on the iteration number alone, and every
sweep counts iterations from the same base, so it fired in each of them
in turn: one run stalled three times and ended in an error rather than
the single clean failure it was meant to demonstrate.

Fire it once per run. Also stop a sweep once clearing its stall required
aborting the mount, since the iterations that follow have nothing left
to run against and only turn a deliberate failure into a confusing one.

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: read wedged threads from /proc rather than a debugger
822abfa 
A run captured the real hang and still produced no stacks for dfuse:
both stack tools timed out against it. That is the failure mode, not a
tooling accident - a debugger stops its target first, and a thread in
uninterruptible sleep cannot be stopped, which is precisely the state a
wedged process is in. The tools go quiet exactly when the information
is wanted.

Report every thread from /proc instead: name, wait channel, state and
kernel stack. It needs no debugger, answers immediately, and names the
function a thread is blocked in even when nothing can attach to it.

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: stop paying for stack tools that cannot read a wedged …
7f21d75 
…process

Neither stack tool can read a process whose threads are in
uninterruptible sleep, and that is the only state this dump is ever
taken in. One run spent eleven of its sixty available minutes waiting
for them to time out, and the run was killed before it could finish.
/proc already reports every thread, so cap the attempt at fifteen
seconds and drop the debugger fallback entirely.

Five repeats, to get more than one chance at an intermittent hang now
that a dump is affordable again.

NLT-repeat: 5
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@daosbuild3

daosbuild3 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18756/11/execution/node/534/log

mjmac added 3 commits July 31, 2026 13:01
@mjmac
DAOS-623 ci: hunt the real hang, twenty runs to a build
50eb4c0 
The deliberate wedge has served its purpose: every part of the
reporting path is now proven, on a real hang for the recovery and on a
forced one for the rest. Keeping it only guarantees that every build
fails, which hides the failure actually being hunted.

Remove it, and raise the repeat count so a build is a real attempt
rather than a single roll: at the rate this hang appears, one run per
build is close to no attempt at all. The stage timeout is pinned rather
than derived from the repeat count, so a wedge that somehow escapes the
watchdog costs six hours instead of twenty.

NLT-repeat: 20
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 ci: one run to a build, which is where the hang actually app…
14674df 
…ears

Twenty runs in a single build found nothing, and the tally across this
branch explains why: three of the four builds that ran NLT once hung,
and none of the forty runs that followed a previous run in the same
build did. The failure tracks the first run on a freshly prepared node,
not the number of runs.

So stop paying for repeats. One run to a build costs about twenty-five
minutes instead of five hours and, on the evidence so far, is far more
likely to reproduce.

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
@mjmac
DAOS-623 test: do not let a wedged mount outlive the run
879ee0c 
A run finished its work, printed its last line and called sys.exit, and
the process then stayed alive for five and a half hours until CI killed
the stage. Everything worth keeping had already been written; the exit
itself was what hung.

Interpreter shutdown runs cleanup that touches the filesystem: waiting
on compression, removing the working directory, stopping a dfuse
instance. None of that can complete against a mount holding requests
nobody will answer, and the calls do not time out.

Do the cleanup where it can be abandoned, fail the outstanding requests
of any backed-up mount if it does block, and leave through _exit so no
remaining handler can take the process hostage on the way out.

This bounds the cost of a wedge to the test that hit it rather than the
whole stage. It cannot be absolute: a thread already blocked in the
kernel keeps the process alive whatever userspace does.

NLT-repeat: 1
Skip-unit-test: true
Skip-unit-test-bdev: true
Skip-unit-test-with-memcheck: true
Skip-unit-test-bdev-with-memcheck: true
Skip-fault-injection-testing: true
Skip-test: true
Skip-test-hardware: true
Skip-test-rpms-on-el-9: true
Skip-test-rpms-on-leap-15: true
Signed-off-by: Michael MacDonald <github@macdonald.cx>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

priority Ticket has high priority (automatically managed)

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mjmac @daosbuild3