Skip to content

Add global scale for nvfp4 quantized MoEs - #1844

Open
nastya236 wants to merge 2 commits into
mainfrom
global-scale-nvfp4
Open

nastya236 wants to merge 2 commits into
mainfrom
global-scale-nvfp4

Conversation

@nastya236

@nastya236 nastya236 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Follow up of ml-explore/mlx#4458 that adds global scale to gather_qmm with nvfp4 weights.
[should be merged after mlx pr]

Tested with mlx-community/Qwen3.6-35B-A3B-nvfp4:

mlx_lm.chat --model mlx-community/Qwen3.6-35B-A3B-nvfp4 --max-tokens 1024

Input: What is the highest mountain? Be brief.
Output:
Here's a thinking process:

  1. Analyze User Input:
  • Question: "What is the highest mountain?"
  • Constraints: "Be brief."
  • Key requirement: Direct, concise answer without elaboration or reasoning.
  1. Identify Core Fact:
  • The highest mountain above sea level is Mount Everest.
  • Height: 8,848.86 meters (29,031.7 feet) above sea level.
  1. Apply Constraints:
  • "Be brief." → Keep it to one short sentence or just the name.
  1. Formulate Response:
  • "Mount Everest." (or "Mount Everest, 8,848.86 meters above sea level.")
  1. Final Check:
  • Matches fact? Yes.
  • Brief? Yes.
  • No extra thinking/reasoning? Yes.
  • Ready. Output: "Mount Everest."✅

Mount Everest.

@nastya236 nastya236 changed the title global scale for nvfp4 moes Add global scale for nvfp4 quantized MoEs Sep 4, 2026
@gabe-l-hart

gabe-l-hart commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

This branch was proposed as a fix for the underflow issues I'm seeing with Granite in #1836 (both dense and moe). I tested it out, but it doesn't seem to fix the issue for either model, and the MoE path raises a stack trace: #1836 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants