Skip to content

Improve pre-M5 metal memory usage for SDPA D512 - #4518

Draft
dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-prenax-d512
Draft

dhiltgen wants to merge 1 commit into
ml-explore:mainfrom
dhiltgen:sdpa-prenax-d512

Conversation

@dhiltgen

Copy link
Copy Markdown
Contributor

This helps improve the memory usage of D512 attention on M4 and older GPUs with models such as gemma4.

Testing with mlx-community/gemma-4-31b-it-nvfp4 and mlx-community/gemma-4-12B-it-nvfp4 via mlx_lm.benchmark g128

System Model Prompt Build Prompt tps Generation tps Peak memory
M4 Pro, 64 GB Gemma 4 12B 16K main 246.530 29.410 9.736 GB
M4 Pro, 64 GB Gemma 4 12B 16K this branch 246.767 29.418 9.032 GB
M3 Ultra, 512 GB Gemma 4 31B 16K main 326.389 26.949 23.952 GB
M3 Ultra, 512 GB Gemma 4 31B 16K this branch 327.970 26.930 22.531 GB
M3 Ultra, 512 GB Gemma 4 12B 65K main 664.250 53.256 13.919 GB
M3 Ultra, 512 GB Gemma 4 12B 65K this branch 710.878 53.681 10.290 GB
  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: co-developed with a coding agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant