Skip to content

Improve metal memory usage for SDPA D256 - #4505

Open
dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-prenax-d256
Open

dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-prenax-d256

Conversation

@dhiltgen

Copy link
Copy Markdown
Contributor

This helps improve the memory usage of D256 attention on M4 and older GPUs with models such as qwen, as well as a small prompt speedup.

Testing with mlx-community/Qwen3.8-27B-nvfp4 and mlx-community/Qwen3.5-2B-nvfp4 via mlx_lm.benchmark

System Model Prompt length Build Prompt tps Generation tps Peak memory
M4 Pro, 64 GB Qwen3.8-27B 16K main 115.8 14.3 46.92 GB
M4 Pro, 64 GB Qwen3.8-27B 16K this branch 125.5 14.3 35.46 GB
M1, 16 GB Qwen3.5-2B 4K main 444.0 47.1 2.78 GB
M1, 16 GB Qwen3.5-2B 4K this branch 454.6 47.0 2.70 GB
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: co-developed with a coding agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant