Skip to content

fix(inference): close the log file if inference fails to start - #89

Open
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:fix/rollout-close-log-on-start-failure
Open

fix(inference): close the log file if inference fails to start#89
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:fix/rollout-close-log-on-start-failure

Conversation

@cn0303

@cn0303 cn0303 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

handle_start_inference opens the rollout log file before it spawns the
inference subprocess, and the stdout-pump thread only takes ownership of that
handle once it starts. If anything in between raised — most commonly Popen
itself failing to launch the process — the except block released the inference
slot and returned a 500, but never closed the log file. That leaks the file
handle. On Windows it also keeps the just-created log file locked.

The cause

log_path = log_dir / f"{int(time.time())}.log"
log_handle = log_path.open("w", buffering=1)      # opened here
...
proc = subprocess.Popen(cmd, ...)                 # if this raises...
...
threading.Thread(target=_pump_stdout, args=(proc, log_handle), ...).start()
except Exception as exc:
    logger.exception("Failed to start inference")
    with _state_lock:
        inference_active = False
    return {...}                                  # ...log_handle is never closed

The fix

Track the handle from before the try, hand ownership to the pump thread by
clearing the reference once the thread starts, and close it on the failure path
only when it is still ours (never opened, or already handed off, both stay safe):

log_handle = None
try:
    ...
    log_handle = log_path.open("w", buffering=1)
    ...
    threading.Thread(target=_pump_stdout, args=(proc, log_handle), ...).start()
    log_handle = None            # pump thread owns and closes it from here on
except Exception as exc:
    logger.exception("Failed to start inference")
    if log_handle is not None:
        log_handle.close()       # opened before we failed — don't leak it
    with _state_lock:
        inference_active = False
    return {...}

Before / after

Simulating a Popen failure (patch subprocess.Popen to raise):

  • Before: handle_start_inference returns 500, but the log file opened moments
    earlier stays open until garbage collection; on Windows the file is locked.
  • After: the handle is closed on the way out. The slot is still released and the
    same 500 is returned.

Testing

New test test_handle_start_inference_closes_log_when_popen_fails patches
Popen to raise and asserts the opened log handle ends up closed and the
inference slot is released. It fails against the current code and passes with the
fix. pytest tests/test_rollout.py: 23 passed. ruff check and ruff format green;
pre-commit (mypy, bandit, typos) green.

handle_start_inference opens the rollout log file before spawning the
subprocess, and the stdout pump thread only takes ownership of that handle once
it starts. If anything in between raised — most likely Popen itself — the except
block released the inference slot and returned, but never closed the log file,
leaking the handle. On Windows that also keeps the log file locked.

Track the handle from before the try, hand ownership to the pump thread by
clearing it once the thread starts, and close it on the failure path when it is
still ours.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant