fix(inference): close the log file if inference fails to start - #89
Open
cn0303 wants to merge 1 commit into
Open
fix(inference): close the log file if inference fails to start#89cn0303 wants to merge 1 commit into
cn0303 wants to merge 1 commit into
Conversation
handle_start_inference opens the rollout log file before spawning the subprocess, and the stdout pump thread only takes ownership of that handle once it starts. If anything in between raised — most likely Popen itself — the except block released the inference slot and returned, but never closed the log file, leaking the handle. On Windows that also keeps the log file locked. Track the handle from before the try, hand ownership to the pump thread by clearing it once the thread starts, and close it on the failure path when it is still ours.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
handle_start_inferenceopens the rollout log file before it spawns theinference subprocess, and the stdout-pump thread only takes ownership of that
handle once it starts. If anything in between raised — most commonly
Popenitself failing to launch the process — the
exceptblock released the inferenceslot and returned a 500, but never closed the log file. That leaks the file
handle. On Windows it also keeps the just-created log file locked.
The cause
The fix
Track the handle from before the
try, hand ownership to the pump thread byclearing the reference once the thread starts, and close it on the failure path
only when it is still ours (never opened, or already handed off, both stay safe):
Before / after
Simulating a
Popenfailure (patchsubprocess.Popento raise):handle_start_inferencereturns 500, but the log file opened momentsearlier stays open until garbage collection; on Windows the file is locked.
same 500 is returned.
Testing
New test
test_handle_start_inference_closes_log_when_popen_failspatchesPopento raise and asserts the opened log handle ends upclosedand theinference slot is released. It fails against the current code and passes with the
fix.
pytest tests/test_rollout.py: 23 passed. ruff check and ruff format green;pre-commit (mypy, bandit, typos) green.