Enforce subprocess timeouts while streaming stdout and stderr

[VishnuR23]

Description

run_subprocess_safely waits until the child has exited before calling wait(timeout=...), so its timeout never limits a running command. A command sleeping for 0.5 seconds with a 0.05-second timeout returns success after roughly 0.58 seconds. Reading a partial line can also block the main loop, and the baseline stalled while draining large stdout/stderr output.

Drain stdout and stderr concurrently with reader threads while the main thread calls wait(timeout=...). On timeout, kill and reap the child, finish collecting its output, and return the existing timeout error structure with partial output. Preserve live output and the existing success/nonzero-exit/not-found result formats. Regenerate the shared copies in Eden and phage recipes.

Type of changes

• Bug fix

Validation

• Three timeout regressions fail against the baseline (silent, line-terminated, and unterminated output).
• All seven CPU tests pass after the fix, including large output on both pipes, live output capture, failure, and missing-command cases.
• All applicable pre-commit hooks pass.
• Full GPU recipe suites were not run locally.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: NVIDIA-BioNeMo/bionemo-recipes/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 77cb1692-26cd-4957-9d29-351e3c32f46e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}