[PERF] Unify distributed communication for real-time LingBot-World v2

[lzx1413]

Description

This PR optimizes TeleFuser's multi-GPU communication and LingBot-World v2 streaming pipeline to support real-time
four-H100 generation.

It introduces direct tensor channels between worker groups, centralizes distributed collectives, adds spatially
sharded VAE decoding, keeps bounded session data resident on GPU, and removes redundant work from the streaming DiT
path. It also improves CPU-only CI coverage and documents the validated four-GPU performance envelope.

Motivation

Large intermediate tensors previously crossed worker and orchestration boundaries through paths that introduced
avoidable synchronization, copies, and CPU involvement. LingBot-World v2 also repeated condition preparation,
device transfers, and cache-related work across chunks.

These costs prevented the four-GPU pipeline from consistently meeting its 16 FPS target. The new communication and
residency design keeps large tensors on their producer/consumer devices, overlaps independent communication where
possible, and reduces per-chunk computation without changing the public pipeline contract.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Other (please describe):

Changes Made

• Added WorkerTensorChannel for direct worker-to-worker tensor transfer using pooled CUDA IPC/P2P buffers or shared
  CPU storage, including lifecycle, cancellation, acknowledgement, sharding, and fallback handling.
• Centralized high-throughput collectives under telefuser.distributed, submitting independent communication before
  synchronization and reusing the same primitives across sequence-parallel paths.
• Added four-GPU spatial VAE decoding with height sharding and halo exchange, colocated with DiT workers to avoid
  orchestrator-mediated latent transfers.
• Kept reference-image conditions, RoPE tables, scheduler inputs, KV state, and reusable output buffers resident for
  the bounded session lifetime.
• Optimized LingBot normalization, modulation, attention dispatch, denoising, cache updates, and streaming lifecycle.
• Hardened tensor handoff and deterministic SDPA regression paths, and added focused unit/integration coverage.
• Updated local CI to enforce CPU-only execution without requiring installation of a separate CPU PyTorch build.
• Added reproducible direct and one-minute AIPerf benchmark workflows and bilingual communication/performance docs.

Testing

• Full test suite executed with exactly pytest tests/
• Manual testing performed
• Benchmarks added/updated

Test commands:

# GitHub-compatible CPU CI
bash scripts/run_ci_tests.sh --skip-install
# Result: 940 passed, 3 skipped, 95 deselected; server suite: 62 passed

# Focused communication and LingBot tests
python -m pytest <focused communication and LingBot test selection>
# Result: 58 passed

# Four-GPU direct real-time gate
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python tools/validation/benchmark_lingbot_world_v2_direct.py \
  --pipeline examples/lingbot/lingbot_world_v2_image_to_video_h100.py \
  --image examples/data/lingbot_world_fast/image.jpg \
  --control-trace benchmarks/telefuser_aiperf/data/stream_lingbot_controls.json \
  --output work_dirs/lingbot_world_v2_4gpu_77frames.json \
  --gpu-num 4 --frame-num 77 --fps 16 --chunk-size 4

# One-minute LiveKit/AIPerf replay
bash benchmarks/telefuser_aiperf/scripts/run_stream_bench.sh
# Result: 1/1 session succeeded; 957 target frames in 60 chunks

# Documentation
python -m mkdocs build --strict
git diff --check

GPU regression coverage also exercised the previously failing example cases. Wan pipeline-parallel behavior is
unchanged and remains outside this PR's scope.

Checklist

• Code follows the project's coding standards (ruff)
• pre-commit run --all-files executed on the final HEAD
• Full pytest tests/ suite executed
• New tests added for new functionality
• Documentation updated
• Commit messages are clear and descriptive
• PR title follows the convention: [TYPE] Brief description

Related Issues

N/A

Additional Notes

The public pipeline and service contracts remain compatible. Optimized tensor transport is method-scoped and retains
fallback behavior where direct CUDA transfer is unavailable.

Target compute and client delivery metrics are intentionally reported separately. LiveKit pacing, encoding, network
delivery, and client rendering are not included in chunk_compute_fps.

GPU Architecture Support

• SM80 (Ampere, Ada Lovelace)
• SM90 (Hopper H100)
• SM100+ (Blackwell)

The new kernel and four-GPU paths were validated on H100/SM90. Other architectures have not been validated as part of
this PR.

Performance Impact

Validated on 4 x H100 80 GB, PyTorch 2.11.0+cu128, BF16 DiT, FP32 VAE, FlashAttention-4, with FSDP and
torch.compile disabled:

• 77-frame direct gate: 17.1399 steady compute FPS at 832x480 against a 16 FPS target.
• Steady chunk mean/p50/p90/max: 0.9335 / 0.9409 / 0.9410 / 1.0058 s.
• One-minute replay: 16.1979 steady compute FPS, completing all 957 target frames and 60 chunks.
• One-minute chunk p99/max: 1.0932 / 1.1149 s.
• One-minute LiveKit delivery: 13.1967 stream FPS and 803 client-received frames.

The average target-side compute rate meets the 16 FPS real-time gate. This does not claim that every individual chunk
finishes within one second or that end-to-end client delivery sustains 16 FPS.