Skip to content

fix(vLLM): harden native IPC synchronization - #494

Open
CjhHa1 wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
CjhHa1:fix/vllm-native-ipc-p1
Open

CjhHa1 wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
CjhHa1:fix/vllm-native-ipc-p1

Conversation

@CjhHa1

@CjhHa1 CjhHa1 commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Hardens the native vLLM packed-IPC path added in #480 against rank-local failures and memory spikes.

  • records the canonical outputs produced by each FSDP materialization and reaches rank consensus before the next materialization collective;
  • also gates IPC-buffer allocation and packed-chunk handle gathering, so one rank cannot leave peers blocked in another process group;
  • drops full-tensor references after their packed copy instead of retaining the previous tensor while materializing the next one;
  • requires the Gloo control timeout to exceed the vLLM weight-update timeout, and fixes the TP4 reference recipe accordingly.

Related Issue

N/A

Test Plan

  • python3 -m pytest -q tests/test_vllm_native_engine.py tests/test_ipc_weight_sync.py (8 passed)
  • SKIP=no-commit-to-branch pre-commit run --all-files --show-diff-on-failure
  • push-time pre-commit suite
  • GPU smoke test not run; this environment has no allocated GPU. The focused tests cover source/chunk consensus gates, failure routing to the next safe gate, buffer-allocation consensus, boundary validation, and timeout ordering.

Compatibility / Risk

  • No checkpoint or wire-format changes.
  • Native vLLM IPC now performs a Gloo consensus at each raw state-dict materialization boundary; this adds control-plane synchronization to prevent cross-process-group deadlocks.
  • Existing valid timeout configurations remain compatible. Native IPC configurations where control_timeout_s <= weight_update_timeout_s now fail during construction instead of timing out asymmetrically during publication.

Reviewer Notes

Follow-up to #480. Please focus on VLLMNativeIPCTrainerEngine._send_planned_packed: source boundaries guard the next FSDP collective, while chunk boundaries guard the default-group IPC-handle gather. The two gates are intentionally distinct.

Checklist

  • I reviewed the changed code and removed unrelated/generated artifacts.
  • I updated tests, docs, and configs where needed, or explained why not.

@github-actions github-actions Bot added the need review Ready and waiting for review label Sep 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need review Ready and waiting for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant