[FEAT] Speculative-cascade acceptance (opt-in, off by default): token-specific rule holds HumanEval at +29% decode; peak rule does not by davidtai · Pull Request #485 · youssofal/MTPLX

davidtai · 2026-09-08T17:25:24Z

Speculative-cascade acceptance: a second opt-in speed optimization for Qwen3.8 Flash-Next

This is a second lossy acceptance mode, NOT distribution-exact, and an ALTERNATIVE to typical acceptance (#478), not an addition on top of it. It changes the emitted token stream. It is OFF by default, selected by --cascade-threshold, and mutually exclusive with #478's --typical-threshold: pick at most one acceptance mode. It is judged on task quality, not on matching the model's own distribution. Turn it on only when you want the extra speed and accept the trade.

This pull request adds one optional decode optimization, speculative-cascade acceptance, behind a single operator knob, --cascade-threshold (the deferral cost alpha), that is unset by default. Its base is commit 27d5ff6b, the #475 optimizations on upstream MTPLX 2.11.2 and the same base as #478's typical acceptance, so cascade is a sibling of #478, a second selectable acceptance mode on the shared exact base, not a change stacked on top of typical. When the knob is unset, the exact speculative-sampling law runs unchanged, token for token, and the output is byte-identical to the base. When it is set, the verify step keeps a drafted token whenever the draft distribution is good enough against the target, and defers to the exact law otherwise, so decode commits more tokens per cycle.

Headline figures at 16,384 tokens of context, 3 seeds, cold prefill:

Decode: 82.80 tok/s with the optimization off, 107.10 tok/s with the TokenV3 rule at alpha 0.95, +29.3%.
Task quality at that point: strict pass@1 0.9695, completed-task 1.0000 (Section 3), against 0.9634 for the exact law on the same base.
Defaults: the mode is off. Alpha unset means the exact speculative law runs. When an operator sets alpha, the rule selector defaults to TokenV3; opt, tokenv1 and tokenv2 stay selectable via --cascade-rule.
The OPT rule's own alpha grid is in Section 2.1 and both rules' quality is in Section 3.

Definitions

Three different fractions in this pull request could each be called an acceptance rate, and they are not interchangeable. Every table column and every sentence below names one of these three terms. No table column and no sentence in this document says "accept rate" on its own. Where the raw field names mtp_accept_rate and accept_rate appear, they are named as fields and are bound to (a) and (c) respectively by the definitions below.

(a) MTP draft acceptance. Accepted draft tokens divided by drafted tokens.

MTP draft acceptance = accepted draft tokens / drafted tokens

Source: the benchmark receipt's mtp_accept_rate field, written by the harness for every window. This is the same quantity #475 and #478 report, on the same population (the throughput windows), so it is the only one of the three that is comparable across the three pull requests. It is what every speed table in Section 2 shows, under the column heading MTP draft acceptance. With the mode off it is the exact law's own figure, about 0.493 on this model at 16,384 tokens.

(b) Cascade deferral rate r. Positions at which the rule deferred to the target, divided by positions the rule decided.

r = positions deferred to the target / positions the rule decided

This is the paper's own quantity: r is the deferral indicator of Narasimhan et al., and their Lemma 3 gives the rejection probability as r times D_TV, which is why the deferral cost carries the alpha * D_TV term. No run in this pull request reports r. The [cascade-accept] verdict line counts positions, accepted and resamples, and a resample can only happen at a deferred position whose coin failed, so the counters bound r from below but do not determine it: a deferred position whose coin succeeded is counted as an accept and is indistinguishable from a position that was never deferred. A defer_rate field is being added to the verdict line so r can be reported directly. Until a run carries that field, r is not available and this document reports (c) instead. Nothing in this document is a measurement of r.

(c) Cascade kept-draft rate. Draft tokens kept, which is the no-defer keeps plus the deferred positions whose exact coin accepted, divided by positions the rule decided.

kept-draft rate = cascade_accepted / cascade_positions

Source: the accept_rate field of the served request's [cascade-accept] verdict line, where cascade_accepted counts both keep paths and cascade_positions counts every position the rule decided. The two counters satisfy accepted + resamples = positions at every position, so the printed accept_rate is exactly the fraction above. This is the quantity the HumanEval quality table reports, under the column heading Kept-draft rate (HumanEval cell), and the quantity the acceptance chart in Section 2.1 plots. It is a within-cascade engagement figure on one HumanEval run, measured on a different population from (a). The two are never set against each other as if they were the same measurement; where both are named in one sentence it is to keep them apart.

Other terms:

decode: the token-by-token generation phase.
verify cycle: one model pass that checks a block of drafted tokens at once.
tokens/cycle: committed tokens divided by verify cycles; higher is faster.
p, q: the target row and the draft row at a position.
D_TV(p, q): total variation sum_v max(0, p(v) - q(v)) over the scored top-k.
alpha: the one operator dial (--cascade-threshold), the deferral cost. Higher alpha means fewer deferrals, so faster and lossier. The two rules read alpha differently and equal alphas are not comparable between them; see Section 1.
pass@1: the fraction of coding problems the first sample solves.
tok/s: tokens per second.

1. Strategy and references

Typical acceptance (Cai et al., #478) and speculative-cascade acceptance (Narasimhan et al., this pull request) are two selectable lossy acceptance modes on the same exact base, the #475 optimizations. An operator picks at most one, and with neither set the exact speculative-sampling law runs. This pull request is the cascade mode.

The exact speculative rule keeps a draft token by a rejection coin min(1, p(x)/q(x)) that reproduces the target distribution. The cascade rule first asks whether to defer at all. When it does not defer, the cascade target is pi = q, so Algorithm 4's per-position accept probability min(1, pi/q) is 1 and the draft token is kept with no coin. That probability is a per-position quantity and is none of the three rates defined above. When it defers, pi = p, which is exactly the lossless speculative-decoding law: the min(1, p/q) coin plus the residual norm(max(0, p - q)). The cascade path is therefore a strict superset of the exact rule with a keep shortcut in front of it, and with the knob unset it is byte-identical to the exact rule. More draft tokens survive per verify cycle, so decode is faster and the kept stream is no longer distribution-exact.

q on this pack is the native MTP head's scored rows, the FR-Spec frequency-ranked subset, the same q the exact rule already uses, so the rule adds no draft forward.

1.1 The OPT rule (Equation (10))

defer  <=>  max_v q(v) < max_v p(v) - alpha * D_TV(p, q)

This is the paper's r-hat_OPT (Section 4.3, Equation (10)), executed with Algorithm 4. It is the plug-in ESTIMATOR of the optimal speculative-cascade deferral rule (Lemma 4, Equation (9)): it replaces the ground-truth expected 0-1 losses that rule needs with one minus each model's max probability. It is therefore not the optimal rule itself and not an oracle, since the oracle requires expectations under the ground-truth distribution. The separate Diff rule (Equation (5), max q < max p - alpha, with no total-variation term) is the SEQUENTIAL-cascade oracle, which this pull request does not implement.

On the sign of the divergence term: the cost alpha * D_TV is subtracted, so more disagreement lowers the bar and defers less, not more. The rule pays the expensive verification only when the target clearly leads. A diverging draft that has also lost peak confidence defers; a draft that stays confident on a wrong token is kept. This is the paper's rule as written.

The paper names the resulting failure itself, in its Section 4.4: comparing max probabilities can keep a drafted token that does not maximize q, when q is merely more peaked than p. That failure is what the OPT quality curve in Section 3 measures.

1.2 The TokenV3 rule (Equation (15))

defer  <=>  p(v) < max_w p(w) * (1 - alpha)

TokenV3 is the paper's answer to that failure: a token-specific rule (Section 4.4, Equation (15), target pi_Token of Equation (11), executed with Algorithm 6) that judges the drafted token v itself rather than comparing the two peaks. It defers when the target's probability on the drafted token is low relative to the target's own peak.

TokenV3's alpha is a fraction of the target's peak probability, while OPT's alpha scales a total-variation margin. The two alphas are different quantities. Equal alphas are not comparable between the two rules; compare the rules at equal decode speed instead, which is what Sections 2.4 and 3 do.

1.3 Selector and defaults

Both rules ship behind one selector, --cascade-rule (env MTPLX_FABLE_CASCADE_RULE, values opt|tokenv1|tokenv2|tokenv3), on the same alpha knob. The selector defaults to tokenv3. opt, tokenv1 and tokenv2 remain selectable. The default is the rule, not the mode: the mode stays off until alpha is set, and with alpha unset the exact speculative law runs whatever the selector says.

Reference: Narasimhan, Jitkrittum, Rawat, Kim, Gupta, Menon, and Kumar, "Faster Cascades via Speculative Decoding," arXiv:2405.19261 v2 (2024), Section 4.3 Equation (10) (r-hat_OPT), Lemma 4 Equation (9), Equation (5), Algorithm 4, Lemma 3, and Section 4.4 Equations (12) to (15) with Algorithm 6.

2. Speed results

The base arm (optimization off) and the cascade arms ran on one machine, each arm receiving the same request body per cell. Section 4 gives the settings. Prefill, TTFT and peak memory are flat across alpha, because the optimization touches only the decode keep decision.

Every throughput cell is the fastest of its seeds with the slowest-to-fastest range. Every acceptance column in this section is (a) MTP draft acceptance, from the receipt's mtp_accept_rate, so it is comparable down the column and across #475 and #478.

2.1 OPT rule, decode at 16,384 tokens (higher is better)

Decode tok/s by cascade alpha: the fastest seed with its slow-to-fast band. The two reference lines are cascade off on the #475 base at 82.80 tok/s and #478 typical acceptance at 0.2 on 99.16 tok/s, both from the pooled 16,384-token windows in #478.

Two charts: cascade kept-draft rate (c) by alpha, and tokens per verify cycle by alpha, the two quantities the dial trades against decode rate. Both are read from the run's [cascade-accept] verdict lines, not from the receipts, which do not carry them. The kept-draft rate plotted here is a different quantity from the MTP draft acceptance column in the table below.

alpha	Decode tok/s (fastest seed)	Tokens/cycle	MTP draft acceptance	Mean D_TV	vs off
off (exact)	82.80 (72.29-82.80) n=6	2.88	0.493	n/a	--
0.0	97.83 (91.46-97.83) n=3	3.53	0.728	0.3459	+18.2%
0.25	104.78 (101.15-104.78) n=3	3.61	0.838	0.3335	+26.5%
0.5	112.41 (106.36-112.41) n=3	3.81	0.900	0.3533	+35.8%
0.75	120.60 (115.27-120.60) n=3	4.27	0.976	0.3356	+45.7%
1.0	139.10 (118.07-139.10) n=3	4.87	1.000	0.3308	+68.0%
2.0	139.31 (120.20-139.31) n=3	4.87	1.000	0.3308	+68.3%

The alpha grid is set from the paper's continuous alpha sweep (Section 6, Figure 2) and this model's measured scale. The exact law's MTP draft acceptance of about 0.493 gives D_TV of about 0.51, so alpha * D_TV spans about 0 to 1.0 across the grid, from maximal deferral at alpha 0 to near all-keep at alpha 2.

2.2 OPT rule, prefill, TTFT and peak memory at 16,384

Metric at 16,384	off	0.0	0.25	0.5	0.75	1.0	2.0
Prefill tok/s	1397.6 (1366.9-1397.6) n=6	1396.5 (1317.8-1396.5) n=3	1395.7 (1321.7-1395.7) n=3	1396.1 (1324.6-1396.1) n=3	1398.1 (1322.6-1398.1) n=3	1398.2 (1321.8-1398.2) n=3	1395.2 (1322.6-1395.2) n=3
TTFT s	11.889 (11.889-12.165) n=6	11.898 (11.898-13.131) n=3	11.924 (11.924-12.577) n=3	11.902 (11.902-12.549) n=3	11.912 (11.912-12.567) n=3	11.888 (11.888-12.575) n=3	11.928 (11.928-12.567) n=3
Peak memory GB	93.36 (87.88-93.36) n=6	92.74 (87.88-92.74) n=3	92.27 (87.88-92.27) n=3	92.27 (87.88-92.27) n=3	93.36 (87.88-93.36) n=3	93.41 (87.88-93.41) n=3	92.37 (87.88-92.37) n=3

2.3 TokenV3 rule, decode at 16,384 tokens

The token-specific rule (--cascade-rule tokenv3) at 16,384 tokens, 3 seeds, fastest-of with the min-max band. vs exact is against the pooled exact law at 82.80 tok/s. This is a separate rule from Section 2.1 and the two tables are not comparable cell for cell at equal alpha, because the two alphas mean different things (Section 1.2).

alpha	Decode tok/s (fastest seed)	Tokens/cycle	MTP draft acceptance	Mean D_TV	vs exact
0.25	84.09 (80.17-84.09) n=3	3.01	0.503	0.3222	+1.6%
0.5	88.70 (84.66-88.70) n=3	3.08	0.547	0.3183	+7.1%
0.75	95.30 (90.12-95.30) n=3	3.23	0.637	0.2850	+15.1%
0.9	96.52 (90.39-96.52) n=3	3.39	0.667	0.3469	+16.6%
0.95	107.10 (96.70-107.10) n=3	3.38	0.722	0.3593	+29.3%

The TokenV3 HumanEval cell runs at alpha 0.95 (107.1 tok/s), the TokenV3 setting closest in speed to OPT alpha 0.25 (104.8 tok/s), so Section 3 compares the two rules at equal decode speed.

2.4 Paired ABAB: typical acceptance 0.09 against TokenV3 alpha 0.95

An interleaved A/B/A/B run of the two lossy modes at their comparable speeds, reported both as each arm's fastest-of and as the paired per-seed delta (B minus A, matched by round and seed) with a 95% interval. The paired delta is the comparison of record; a difference of two fastest-of values is not.

Arm	Decode tok/s (fastest, min-max)
A: typical acceptance 0.09	104.46 (99.16-104.46) n=9
B: TokenV3 alpha 0.95	107.11 (96.55-107.11) n=9

Paired B minus A	95% interval	pairs
-1.62 tok/s	-4.30 to +1.05	9

In a three-round interleaved run at 16,384 tokens the two quality-neutral modes decode at the same speed within noise: typical acceptance at 0.09 reaches 104.46 tok/s and TokenV3 at alpha 0.95 reaches 107.11 tok/s, and the paired mean difference is -1.62 tok/s with a 95% confidence interval of -4.30 to +1.05.

2.5 Context sweep

The context sweep runs the exact arm and each cascade arm across 1,024 to 261,120 tokens, with the same fastest-of rule and min-max band as the 16K cell. The 261,120 row overflows on this base: cascade sits on the #475 base (27d5ff6b) without the #482 verify-KV fix, so the first decode step exceeds the memory knob. #482 fixes it. See #475 Section 1.5.

The 16,384 row of every context table is the Section 2.1 and Section 2.3 window for that arm. The exact column is arm-C-caspair, this run's own paired exact windows, so at 16,384 it reads 82.85 rather than the pooled 82.80 in Section 2.1; the 0.05 tok/s between them is inside either band. The 131,072 exact cell is two seeds: the third hit a server crash unrelated to memory and was dropped.

Decode tok/s by context, one line per arm, each point the fastest seed with a min-max variance bar. The 261,120 cell is absent, over the knob on this base.

Decode tok/s by context

Cell	off	0.0	0.25	0.5	0.75	1.0	2.0	TokenV3 0.95
1K	89.10 (85.27-89.10) n=3	104.15 (94.42-104.15) n=3	116.02 (109.55-116.02) n=3	123.84 (116.94-123.84) n=3	131.32 (119.29-131.32) n=3	134.73 (124.84-134.73) n=3	135.47 (125.48-135.47) n=3	106.46 (104.54-106.46) n=3
8K	85.70 (75.65-85.70) n=3	86.43 (85.08-86.43) n=3	101.91 (92.75-101.91) n=3	107.47 (100.58-107.47) n=3	103.28 (102.68-103.28) n=3	128.41 (108.17-128.41) n=3	128.85 (108.28-128.85) n=3	98.81 (87.30-98.81) n=3
16K	82.85 (72.57-82.85) n=3	97.83 (91.46-97.83) n=3	104.78 (101.15-104.78) n=3	112.41 (106.36-112.41) n=3	120.60 (115.27-120.60) n=3	139.10 (118.07-139.10) n=3	139.31 (120.20-139.31) n=3	107.10 (96.70-107.10) n=3
32K	81.40 (71.64-81.40) n=3	96.61 (87.58-96.61) n=3	108.35 (101.60-108.35) n=3	126.82 (107.13-126.82) n=3	121.30 (108.94-121.30) n=3	125.77 (117.40-125.77) n=3	126.67 (117.43-126.67) n=3	102.32 (95.90-102.32) n=3
64K	84.51 (70.98-84.51) n=3	92.78 (90.11-92.78) n=3	99.24 (91.70-99.24) n=3	112.35 (107.47-112.35) n=3	147.00 (110.63-147.00) n=3	117.27 (114.62-117.27) n=3	117.43 (114.67-117.43) n=3	95.80 (91.59-95.80) n=3
128K	68.69 (65.46-68.69) n=2	92.33 (81.33-92.33) n=3	94.70 (89.38-94.70) n=3	102.17 (96.40-102.17) n=3	107.11 (103.92-107.11) n=3	112.27 (108.58-112.27) n=3	112.21 (107.88-112.21) n=3	85.88 (80.59-85.88) n=3
255K	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)

At 64K the alpha 0.75 fastest seed (147.00 (110.63-147.00) n=3) lands above alpha 1.0. Cascade keeps are stochastic at temperature 1, so decode varies seed to seed and this band overlaps alpha 1.0's. The point inversion at this one context is seed noise, not a reversal of the alpha ordering.

Prefill tok/s by context

Cell	off	0.0	0.25	0.5	0.75	1.0	2.0	TokenV3 0.95
1K	912.7 (759.0-912.7) n=3	916.9 (760.8-916.9) n=3	915.9 (745.2-915.9) n=3	916.7 (762.7-916.7) n=3	914.2 (749.3-914.2) n=3	913.0 (759.0-913.0) n=3	911.0 (762.5-911.0) n=3	910.3 (749.8-910.3) n=3
8K	1360.6 (1305.2-1360.6) n=3	1361.0 (1303.0-1361.0) n=3	1359.0 (1239.8-1359.0) n=3	1361.5 (1306.0-1361.5) n=3	1363.0 (1240.1-1363.0) n=3	1363.0 (1304.6-1363.0) n=3	1364.1 (1307.8-1364.1) n=3	1357.8 (1238.0-1357.8) n=3
16K	1398.2 (1328.7-1398.2) n=3	1396.5 (1317.8-1396.5) n=3	1395.7 (1321.7-1395.7) n=3	1396.1 (1324.6-1396.1) n=3	1398.1 (1322.6-1398.1) n=3	1398.2 (1321.8-1398.2) n=3	1395.2 (1322.6-1395.2) n=3	1395.9 (1318.7-1395.9) n=3
32K	1260.1 (1244.2-1260.1) n=3	1259.9 (1239.6-1259.9) n=3	1258.3 (1217.8-1258.3) n=3	1261.1 (1243.5-1261.1) n=3	1258.6 (1218.9-1258.6) n=3	1259.9 (1240.6-1259.9) n=3	1260.9 (1244.1-1260.9) n=3	1258.5 (1219.8-1258.5) n=3
64K	1178.6 (1175.9-1178.6) n=3	1176.9 (1161.1-1176.9) n=3	1170.3 (1166.1-1170.3) n=3	1179.3 (1177.5-1179.3) n=3	1169.1 (1168.5-1169.1) n=3	1177.3 (1175.9-1177.3) n=3	1177.2 (1176.3-1177.2) n=3	1168.9 (1151.8-1168.9) n=3
128K	1130.1 (1129.8-1130.1) n=2	1131.4 (1121.9-1131.4) n=3	1125.7 (1125.5-1125.7) n=3	1130.6 (1129.0-1130.6) n=3	1126.5 (1123.9-1126.5) n=3	1129.2 (1128.8-1129.2) n=3	1130.4 (1129.3-1130.4) n=3	1126.7 (1116.9-1126.7) n=3
255K	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)

TTFT s by context

Cell	off	0.0	0.25	0.5	0.75	1.0	2.0	TokenV3 0.95
1K	1.247 (1.247-1.478) n=3	1.237 (1.237-1.473) n=3	1.239 (1.239-1.503) n=3	1.241 (1.241-1.470) n=3	1.240 (1.240-1.494) n=3	1.243 (1.243-1.477) n=3	1.244 (1.244-1.471) n=3	1.246 (1.246-1.499) n=3
8K	6.165 (6.165-6.428) n=3	6.169 (6.169-6.450) n=3	6.179 (6.179-6.771) n=3	6.166 (6.166-6.433) n=3	6.165 (6.165-6.766) n=3	6.159 (6.159-6.441) n=3	6.154 (6.154-6.428) n=3	6.174 (6.174-6.774) n=3
16K	11.905 (11.905-12.506) n=3	11.898 (11.898-13.131) n=3	11.924 (11.924-12.577) n=3	11.902 (11.902-12.549) n=3	11.912 (11.912-12.567) n=3	11.888 (11.888-12.575) n=3	11.928 (11.928-12.567) n=3	11.900 (11.900-12.598) n=3
32K	26.262 (26.262-26.577) n=3	26.251 (26.251-26.779) n=3	26.266 (26.266-27.139) n=3	26.213 (26.213-26.598) n=3	26.251 (26.251-27.112) n=3	26.243 (26.243-26.661) n=3	26.219 (26.219-26.585) n=3	26.250 (26.250-27.088) n=3
64K	55.946 (55.946-56.139) n=3	56.028 (56.028-56.809) n=3	56.324 (56.324-56.527) n=3	55.940 (55.940-55.993) n=3	56.377 (56.377-56.447) n=3	55.996 (55.996-56.056) n=3	55.991 (55.991-56.054) n=3	56.390 (56.390-57.220) n=3
128K	116.575 (116.575-116.641) n=2	116.451 (116.451-117.461) n=3	117.008 (117.008-117.040) n=3	116.540 (116.540-116.699) n=3	116.904 (116.904-117.207) n=3	116.660 (116.660-116.673) n=3	116.506 (116.506-116.642) n=3	116.916 (116.916-117.942) n=3
255K	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)

Wall s by context

Cell	off	0.0	0.25	0.5	0.75	1.0	2.0	TokenV3 0.95
1K	13.48 n=1	11.07 (11.07-12.31) n=3	10.07 (10.07-10.13) n=2	9.52 (9.52-9.76) n=2	9.04 (9.04-9.06) n=2	8.84 (8.84-9.16) n=2	8.81 (8.81-9.03) n=2	10.86 (10.86-11.29) n=2
8K	18.90 n=1	18.05 n=1	16.23 (16.23-17.69) n=2	n/a	16.10 (16.10-16.74) n=2	15.22 n=1	15.20 n=1	n/a
16K	24.89 (24.89-25.49) n=2	n/a	n/a	21.03 n=1	20.42 (20.42-21.12) n=2	21.27 n=1	n/a	22.46 n=1
32K	39.58 (39.58-40.64) n=2	36.93 (36.93-38.47) n=2	35.84 (35.84-37.30) n=2	35.16 (35.16-36.23) n=2	34.77 n=1	34.92 n=1	34.93 n=1	37.00 n=1
64K	n/a	68.45 n=1	n/a	n/a	63.63 (63.63-65.97) n=2	65.05 n=1	64.99 n=1	n/a
128K	n/a	n/a	128.92 n=1	127.93 n=1	128.06 n=1	n/a	n/a	n/a
255K	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)

Peak memory GB by context

Cell	off	0.0	0.25	0.5	0.75	1.0	2.0	TokenV3 0.95
1K	86.88 (84.85-86.88) n=3	87.34 (84.85-87.34) n=3	86.88 (84.85-86.88) n=3	87.35 (84.85-87.35) n=3	86.88 (84.85-86.88) n=3	86.87 (84.85-86.87) n=3	86.41 (84.85-86.41) n=3	86.88 (84.85-86.88) n=3
8K	90.99 (86.94-90.99) n=3	90.10 (86.94-90.10) n=3	90.10 (86.94-90.10) n=3	90.10 (86.94-90.10) n=3	91.88 (86.94-91.88) n=3	90.63 (86.94-90.63) n=3	90.63 (86.94-90.63) n=3	90.10 (86.94-90.10) n=3
16K	93.36 (87.88-93.36) n=3	92.74 (87.88-92.74) n=3	92.27 (87.88-92.27) n=3	92.27 (87.88-92.27) n=3	93.36 (87.88-93.36) n=3	93.41 (87.88-93.41) n=3	92.37 (87.88-92.37) n=3	92.28 (87.88-92.28) n=3
32K	96.44 (88.94-96.44) n=3	96.44 (88.94-96.44) n=3	96.44 (88.94-96.44) n=3	97.52 (88.94-97.52) n=3	95.45 (88.94-95.45) n=3	96.00 (88.94-96.00) n=3	96.00 (88.94-96.00) n=3	96.44 (88.94-96.44) n=3
64K	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3	89.67 (89.67-89.67) n=3
128K	93.04 (93.04-93.04) n=2	93.05 (93.05-93.05) n=3	93.05 (93.05-93.05) n=3	93.05 (93.05-93.05) n=3	93.05 (93.05-93.05) n=3	93.05 (93.05-93.05) n=3	93.05 (93.05-93.05) n=3	93.04 (93.04-93.04) n=3
255K	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)	exceeds the memory knob (fixed by #482)

2.6 Comparison with #475 and #478

Both charts put this pull request's operating points beside the two that precede it on the same base: #475 (base, acceptance mode off) and #478 (typical acceptance at 0.2). Every point is the fastest seed with a min-max band, and each arm's acceptance mode is in its label, because a decode figure is not readable without it.

Decode tok/s at 16,384 tokens for release 2.11.2, #475, #478 and four operating points from this pull request, each bar annotated with the same arm's HumanEval strict pass@1.

Arm	Acceptance mode	Decode tok/s at 16,384	HumanEval strict pass@1
release 2.11.2	acceptance mode off	71.97 (70.98-71.97) n=9	0.9695
#475 (base)	acceptance mode off	82.84 (72.02-82.84) n=9	0.9634
#478	typical 0.2	99.16 (95.30-99.16) n=9	0.9695
this PR, OPT rule	cascade alpha 0.25	104.78 (101.15-104.78) n=3	0.8537
this PR, TokenV3 rule	cascade alpha 0.75	95.30 (90.12-95.30) n=3	0.9695
this PR, TokenV3 rule	cascade alpha 0.95	107.10 (96.70-107.10) n=3	0.9695

TokenV3 at alpha 0.95 reaches 107.10 tok/s against 99.16 for #478 typical acceptance at 0.2 and 82.84 for the #475 base with acceptance mode off, at the same strict pass@1 the release build scores. The paired ABAB comparison in Section 2.4 is the controlled version of the TokenV3-against-typical figure and puts the two at the same speed within noise; the bars above are the unpaired sweep windows. OPT at alpha 0.25 lands at 104.78 tok/s, within noise of typical acceptance, but at strict pass@1 0.8537.

Decode tok/s by context size, 1,024 to 131,072 tokens, for #475 (base) with cascade off, #478 at typical 0.2, this pull request's OPT alpha 0.25 and its TokenV3 alpha 0.95. No arm has a 261,120 point: on the #475 base without #482 every arm on this pack exceeds the memory knob.

3. Quality results

HumanEval at the cell's own sampler (temperature 1, top-p 0.95, top-k 20, xhigh), one seed 20260829, re-scored offline. Each cell reports strict pass@1, completed-task pass@1 and the truncation rate.

The acceptance column here is (c) the kept-draft rate, measured on this HumanEval run itself from its [cascade-accept] verdict lines. It is a within-cascade engagement figure for the cell in its own row. It is not the MTP draft acceptance of the speed tables in Section 2, which is a different quantity on a different population, and the two are never compared.

HumanEval pass@1 per operating point: strict (truncated tasks count as failures) and completed-task (truncated tasks excluded). The two references on the left are cascade off on the #475 base (0.9634 strict) and #478 typical acceptance at 0.2 (0.9695 strict, 1.0000 completed-task), both from the #478 sweep on the same base, seed and sampler. The OPT rule loses accuracy at every alpha, from 0.9024 at alpha 0.0 down to 0.7073 at 0.75; the TokenV3 rule holds 0.9695 at both 0.75 and 0.95, level with typical 0.2. OPT alpha 1.0 did not finish and has no bar.

alpha	Strict pass@1	Completed-task pass@1	Truncation (rate; mean/max tok)	Kept-draft rate (HumanEval cell)	Wall / mean output tok
off (exact)	0.9634	0.9814	1.8% (mean 2734 tok)	n/a	1h 29m / 2,734
0.0	0.9024	0.9427	4.3% (mean 3419 tok)	0.871	1h 42m / 3,419
0.25	0.8537	0.8974	4.9% (mean 3837 tok)	0.919	1h 49m / 3,837
0.5	0.7805	0.8533	8.5% (mean 5633 tok)	0.953	2h 41m / 5,633
0.75	0.7073	0.7785	9.1% (mean 6133 tok)	0.977	2h 49m / 6,133
1.0	did not finish: 153 of 164 tasks in 3 h	did not finish: 153 of 164 tasks in 3 h	13.1% over the 153 generated	0.9999	3 h (limit) / 7,207 over the 153 generated
tokenv3 0.75	0.9695	0.9876	1.8% (mean 2652 tok)	0.820	1h 19m / 2,652
tokenv3 0.95	0.9695	1.0000	3.0% (mean 3037 tok)	0.881	1h 22m / 3,037

Quality cells were run at alpha 0.0, 0.25, 0.5, 0.75 for OPT (and at 1.0, which did not finish inside the window) and 0.75, 0.95 for TokenV3; other alphas have speed windows only.

The off (exact) row is the #478 sweep's own exact cell, served from that tree with the typical rule off: the same exact speculative law on the same #475 base, one seed 20260829, re-scored the same way. It is the comparison point for the alpha rows, not a cascade-tree measurement. Cascade is off there, so it has no kept-draft rate.

3.1 The two rules at equal decode speed

TokenV3 at alpha 0.95 decodes at 107.10 tok/s with strict pass@1 0.9695 (159 of 164), the same as the release build and one task above the exact law on this base. OPT at alpha 0.25 decodes at 104.78 tok/s with strict pass@1 0.8537. At matched speed the two rules separate by 19 tasks of 164, which is the failure the paper's Section 4.4 token-specific rules were designed to fix.

On the 16,384-token speed windows those two arms differ in (a) MTP draft acceptance as well: TokenV3 alpha 0.95 reads 0.722 and OPT alpha 0.25 reads 0.838, both from the Section 2.1 and Section 2.3 tables. TokenV3 therefore reaches the same speed while keeping fewer drafts, which means it defers on different tokens rather than on fewer of them.

3.2 The OPT alpha curve

Strict pass@1 across the OPT sweep is 0.9634 for the exact law, 0.9024 at alpha 0.0, 0.8537 at 0.25, 0.7805 at 0.5 and 0.7073 at 0.75. Alpha 1.0 did not finish 164 tasks in three hours.

At alpha 0.0, the mildest setting, strict pass@1 is 0.9024 against 0.9634 for the exact law on the same base, ten tasks of 164; completed-task pass@1 is 0.9427 against 0.9814 and truncation 4.3% against 1.8%. At alpha 0.5, strict pass@1 is 0.7805 (128 of 164), completed-task 0.8533, truncation 8.5% and mean output 5,633 tokens, while the kept-draft rate on the cell rises from 0.871 at alpha 0.0 to 0.953 at 0.5. Quality falls as the cell keeps more drafts.

Generation length grows with alpha and is itself a result. Mean output is 2,734 tokens for the exact law, 3,419 at alpha 0.0 and 5,633 at alpha 0.5. At alpha 1.0 mean output reached 7,207 tokens with a kept-draft rate of 0.9999, and 153 of 164 tasks finished inside the three-hour window.

3.3 The TokenV3 alpha curve

TokenV3 holds strict pass@1 at 0.9695 at both measured alphas: 0.75 at 95.30 tok/s and 0.95 at 107.10 tok/s. It is the only rule measured here that holds HumanEval at the release build's own score.

4. Benchmark method

Setting	Value
Model	`Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed`, revision `29ba90f82124961d0d902a9ea9bbb1034972af2f`
Arms	base (release 2.11.2 + #475 @ `27d5ff6b`, optimization off); base + cascade acceptance at alpha 0.0, 0.25, 0.5, 0.75, 1.0, 2.0
Engine build	MTPLX 2.11.2 on MLX 0.32.2
Profile	cli-resolved (Turbo); a fresh SSD session cache directory per window
Prompt / cell	16,384 tokens of context
Output length	1,024 tokens maximum per request
Sampler	temperature 1, top-p 0.95, top-k 20
Reasoning effort	`xhigh`
MTP depth	native, depth 3
Seeds	20260829, 20260830, 20260831
16,384 protocol	interleaved base/candidate, 3 windows per arm, 3 seeds each; each arm's value is its fastest window
Cell statistic	throughput reports the fastest of the seeds with the slowest-to-fastest range; TTFT and wall the fastest (lowest); peak memory the highest
Quality gate	HumanEval at the cell's own sampler, one seed 20260829; strict + completed-task pass@1 + truncation
Memory cap	100 GiB on every arm
Prefill state	cold; cross-request prefix restore off; every cell cold by receipt
Thermal state	fans at maximum, a 40 degree Celsius gate before every cell

5. Switch and observability

One optimization, one dial, off by default, mutually exclusive with typical acceptance.

The knob is --cascade-threshold (env MTPLX_FABLE_CASCADE_THRESHOLD). Unset is off and byte-identical to the exact law. Any set value turns the mode on, and higher alpha is faster and lossier. The rule selector is --cascade-rule (env MTPLX_FABLE_CASCADE_RULE, values opt|tokenv1|tokenv2|tokenv3), which picks the deferral rule on that same alpha knob; it defaults to tokenv3, the paper's Section 4.4 token-specific rule, and opt (Equation (10)), tokenv1 and tokenv2 remain selectable.

GET /health reports the resolved state under cascade_acceptance. Each active request prints one [cascade-accept] NOT distribution-exact line carrying rule, threshold, alpha, positions, accepted, resamples, accept_rate, mean_divergence, tokens_per_cycle, accepted_by_depth, generated and verify_calls. The accept_rate field is (c) the kept-draft rate as defined above, and accepted + resamples equals positions at every position. The line does not yet carry (b) the cascade deferral rate r; a defer_rate field is being added for it, and until then no run reports r.

The mode ships off by default. When an operator enables it, the rule defaults to TokenV3, the only measured rule that holds HumanEval, and alpha is the operator's choice. The measured TokenV3 points are alpha 0.75 (95.30 tok/s) and alpha 0.95 (107.10 tok/s), both at strict pass@1 0.9695.

6. Provenance and file map

The [cascade-accept] verdict lives in the served generate_mtpk loop (generation.py:8028; server entry openai.py:24502). A re-parent had moved it into the uncalled generate_mtpa (generation.py:13858), so between the re-parent and ef0e83e8 no served request could emit it; 351c889c removes the dead copy from generate_mtpa, leaving that function byte-identical to the base. The cascade rule itself runs solely in generate_mtpk (cascade_defer_decision, generation.py:12383/12497). The 16K OPT sweep ran on the pre-re-parent code and the HumanEval cells on the fixed code, both with the verdict present; the context ladder ran on the un-fixed code with engagement verified through /health and through (a) MTP draft acceptance rising monotonically with alpha across the ladder receipts, from 0.494 on the exact arm to 0.706 at alpha 0.0, 0.816 at 0.25, 0.910 at 0.5, 0.945 at 0.75 and 1.000 at 1.0 and 2.0 (mean over the ladder's 15 records per arm).

Measured on served code 2eac2fee. The OPT rule on the branch of record (perf/qwen38-cascade-acceptance, base 27d5ff6b) is byte-identical to it by AST code hash, with the proof in docs/perf/qwen38-cascade-acceptance.md. The TokenV3 numbers were measured on served code 3bfdb685, whose mtplx/ content equals a2531bde, the commit on this branch that adds the rule selector. ef0e83e8 and 351c889c differ only inside generate_mtpa, a function with no callers on any tree, so every HumanEval cell served an identical decode path and rule regardless of which blob it imported; the per-cell blob hashes in the sweep report are recorded for the record, not relied on.

Full writeup: docs/perf/qwen38-cascade-acceptance.md.

… 2.11.2 MTPLX 2.11.2 re-landed nine of PR 391's twelve Flash-Next decode keys; this ports three of the unlanded five onto upstream's re-landed structure, wired into the fixed-M4 auto-arm block with the MTPLX_QWEN4_*/MTPLX_QSA_* namespace and a per-key =0 opt-out (the old MTPLX_FABLE_* names kept as aliases). - HC_M4 (MTPLX_QWEN4_HC_M4): the verify-width (2..8 row) hyper-connection read run as one multi-threadgroup GEMV (kernels/qwen4_m4_hyper_read). Reader in runtime_options read once at import; GatedResidual gains the geometry eligibility check, pack validation and the fused read; install validation runs after the M4-stage3 install and reports at /health qwen4_install_reports.hc_m4. Rounding-class. - prefill causal-mask fuse (MTPLX_QWEN4_PREFILL_MASK_FUSE): the dense QSA prefill chunk goes through MLX's fused SDPA instead of a materialized score tensor MLX's head-dim-256 heuristic declines; a per-shape-class capability cache keeps a verify step MLX refuses from disarming a wide chunk. Rounding-class (exact visible set). - QSA prefill query tile (MTPLX_QSA_PREFILL_QUERY_TILE): tiles only the dense QSA attention query rows so a wider prefill chunk keeps the narrow chunk's attention peak and cost. Value companion, default 2048 (inert at the production 2,048 chunk width). Rounding-class (exact visible set). Each is default-on for a served fixed-M4 Flash-Next pack (server auto-arm lane_defaults, gated on the fixed-M4 config predicate) with a per-key kill switch through the existing pop loop, and registered in the boot-time runtime-env validator. All three are rounding-class, so quality-gated on HumanEval. Two of the five remain and are documented in docs/perf/qwen38-391-remainder.md: the QSA sparse split-K decode (a native kernel whose build and parity probe need the GPU) and the graph-build overlap (its prefix/suffix split of the fixed-M4 verify has no substrate on 2.11.2's single-graph verify). CPU tests (venv mlx 0.32.2, no GPU): tests/test_qwen4_hc_m4.py 53 passed, tests/test_qwen4_prefill_mask_fuse.py 40 passed.

The fourth of PR 391's five unlanded lanes: MTPLX_QSA_SPARSE_DECODE, the native split-K sparse-GQA attention for the M=4 fixed verify. It reads the selected KV rows of the fixed QSA cache once per verify cycle instead of materializing a gathered [1,2,4,2052,256] K/V pair per layer, which is where the shipped lane's bytes are. Rounding class: fp32 online softmax over the exact visible set. - kernels/qsa_sparse_decode.py + native_extensions/qsa_sparse_gqa (package mtplx_native_qsa: the split-K Metal kernel, steel headers and a nanobind binding); mtplx/native loads it, runtime_options reads MTPLX_QSA_SPARSE_DECODE (+_TILE 128:32, +_SPLITS 17); the old MTPLX_FABLE_* names are honoured as aliases when the new key is unset. - graphbank.TensorOffsetQSACache validates the lane ONCE at cache install (a real parity probe, outside any mx.compile trace); the twin re-promotion sites and the compiled verify_step carry it, and the verify body asserts the lane is in the traced graph. models/qwen4_exp routes the fixed-capacity verify width to the kernel (QSAIndexer._sparse_decode_route) or declines to stock for a request shape it cannot serve. - Server auto-arm: default ON for the fixed-M4 pack ONLY when the native extension is built; a wheel without mtplx_native_qsa declines to stock with a logged verdict and still serves. An explicit MTPLX_QSA_SPARSE_DECODE=1 reaches the fail-closed install (armed and unbuilt raises). Registered in the boot-time runtime-env validator; kill switch through the existing pop loop. - scripts/bundle_native_runtime_wheel.py signs and packages mtplx_native_qsa alongside mtplx_qsa_kernels (Developer ID, hardened runtime, secure timestamp), with tests. - The mask-fuse refusal test now accepts either MLX build's native wording: the lane logs a version-independent per-class line and never raises under default arming (it falls to the stock dense SDPA). Load-time parity on stock mlx 0.32.2 with the native kernel built: vs the fp32 reference worst rel_l2 3.1e-05 with the top-1 token identical, vs the stock gather path rel_l2 4.6e-03 (rounding class), across the 4093 and 2052 probe cells that stand in for the 16K and 261,120 serving regimes. CPU tests (venv mlx 0.32.2): tests/test_qsa_sparse_decode.py, tests/test_qsa_sparse_decode_wiring.py, tests/test_qsa_sparse_gqa_native.py and tests/test_bundle_native_runtime_wheel.py all green.

Served via `mtplx serve` (cli-resolved Turbo, no lane flags), the QSA split-K decode lane did not engage: qsa_sparse_decode_enabled() read the environment at IMPORT and cached the default (False), but the fixed-M4 auto-arm stamps MTPLX_QSA_SPARSE_DECODE (native-gated) into the environment AFTER runtime_options is imported, so the cache froze the default before the stamp landed -- the lane was absent from /health with neither an "armed:" nor a "declined to stock" line. hc_m4 escaped only because its reader is read on a path where the module was imported after the stamp. Resolve the flag lazily on the FIRST read (which is the graphbank cache install, after the overrides are applied), then cache; the _QSA_SPARSE_DECODE module global stays (tests force it to a bool) and the native-gated default in the server auto-arm is unchanged. The env is frozen once serving starts, so a lazy first read is still a single cached bool on the hot path. Regression tests, the shape that would have caught this: - the reader picks up a stamp applied AFTER import (an import-frozen reader fails it), - the fixed-M4 auto-arm block stamps the lane when the native extension is built, or prints the declined-to-stock verdict and leaves it unstamped when it is not.

Second arming failure (battery, 2026-09-07): served as `mtplx serve` launches it, hc_m4 was OFF for the same reason the QSA decode lane was in commit 3 -- its reader froze the environment at import (default off) while the fixed-M4 auto-arm stamps the lane keys into the environment AFTER runtime_options is imported. The earlier claim that hc_m4 read on a post-stamp path did not hold for the served path. Resolve every remainder-lane flag at USE (the install / route path, which runs after the overrides are applied), never at import: - runtime_options: qwen4_hc_m4_enabled, qsa_sparse_decode_tile and qsa_sparse_decode_splits (qsa_sparse_decode_enabled was fixed in commit 3); each keeps its module global as a test override (None = read env). - models/qwen4_exp: _prefill_mask_fuse_enabled drops @lru_cache (its body already reads os.environ), so a stamp landing after import is seen. - qwen4_prefill_chunk.resolve_query_tile_rows already read at use. The native-gated default and the MTPLX_FABLE_* aliases are unchanged. Upstream's own MTPLX_QWEN4_OPDIET / MTPLX_QWEN4_VERIFY_GLUE readers are left as-is (not part of this remainder set). Test reproducing the served order (tests/test_qwen4_remainder_arming.py): import mtplx.runtime + mtplx.server.openai FIRST, assert all four readers off, run _server_runtime_env_overrides for the fixed-M4 pack, apply it to os.environ, then assert all four arm -- the decode lane armed when the native extension is built, else an explicit declined-to-stock verdict, never silent absence. The hc_m4 read-once test is rewritten to assert read-at-use, and the mask-fuse test drops its now-defunct cache_clear() calls.

Arming audit (battery, 2026-09-07): `mtplx serve` imports generation / runtime / model modules before parse_args stamps the auto-arm env, so any flag whose reader resolves at module import freezes its default before the stamp lands and the auto-arm's setdefault is a silent no-op as launched. A column-0 scan of every module holding a stamped key's reader found exactly four such readers among the ~31 auto-armed keys; every other stamped key reads the environment at use or is consumed from config.json at model load. All four are decode-verify lanes, so the release control (71.17 tok/s at 16K) ran without them -- a plausible slice of the 71->81 gap. Resolve all four at use (read the environment each call; the module global stays a test/force override; the env is frozen once serving starts, so two traces of one graph still read the same value): - MTPLX_QWEN4_DRAFT_K20_PRESCATTER: qwen4_draft_k20_prescatter._ENABLED, and generation.py's cached _QWEN4_DRAFT_K20_PRESCATTER (removed; the one draft consult site calls the reader). - MTPLX_QWEN4_BLOCK_VERIFY: qwen4_block_verify._ENABLED, and generation.py's cached _QWEN4_BLOCK_VERIFY (removed; the accept-loop consult calls the reader). - MTPLX_QWEN4_OPDIET (+ _ITEMS): runtime_options. - MTPLX_QWEN4_VERIFY_GLUE (+ _ITEMS): runtime_options (reset hook kept, now forcing the globals). No default value changed; keys that already read at use are untouched. Upstream's STRICT_CLAIMS and BATCH_PAGED_OFFSETS are also import-frozen but are not auto-armed (operator sets them pre-launch), so they are left as-is. Tests: tests/test_qwen4_remainder_arming.py extended to assert all four arm in the served order (import first, stamp, read) and that the fixed-M4 auto-arm stamps OPDIET / BLOCK_VERIFY / VERIFY_GLUE and their readers then arm. The block-verify and draft-k20 source-inspection tests and the opdiet read-once test are rewritten to assert read-at-use.

The arming audit's lesson: gate on the install verdict, not the env. Three decode-verify lanes had no per-window observable in /health qwen4_install_reports -- draft_k20_prescatter, block_verify, opdiet -- so a served window could not confirm they engaged. Add read-only reports (no behaviour change, no defaults touched): - draft_k20_prescatter: {armed (read at use), engaged (first-use latch set when claim_draft_route installs the route), receipt (the last install receipt)}. - block_verify: {armed, engaged (latched when a block verifier is built for the accept loop)} plus a one-shot "[mtplx] MTPLX_QWEN4_BLOCK_VERIFY armed:" log. - opdiet: {armed, items (configured selection), applied (first-use latch of the items that actually ran at a gated site)}. Each appears only when ARMED (read at use, gate-able without a request), so an unarmed lane stays absent (== off) like the other lanes; the engaged/applied latch rides inside the armed report. CPU test: tests/test_qwen4_remainder_arming.py asserts the three reports are absent when off / =0 and present with armed True under a served-order stamp.

… 391 remainder port Rebase of PR youssofal#475 (cached async PLE + pooled-key rowsel) onto the PR 391 remainder port head (perf/qwen38-391-remainder-main = upstream main 2.11.2 + the four remainder lanes: HC_M4, prefill mask fuse, QSA query tile, and the QSA split-K sparse-GQA decode extension. The two lanes here add to the same fixed-M4 lane_defaults / _QWEN4_PORT_KEYS block, and their native loader (ple_cpu_rows) and wheel-signer entry union with the QSA sparse-decode extension's (mtplx_native_qsa), all cleanly additive). PR 391 is closed and its Fable delivery stack (mtplx/full_stack_env.py, the turbo-full-stack profile, mtplx/native's QSA sparse-GQA loader) was never merged, but the maintainer independently re-landed the Flash-Next stack under the MTPLX_QWEN4_*/MTPLX_QSA_* namespace, so both lanes' base contracts are present upstream and both apply: the fixed-M4 compiled-verify auxiliary plane (mtplx/qwen4_fixed_verify.py) and the QSA indexer pooled-key kernel (mtplx/kernels/qsa_indexer_prepare._pool_keys_kernel). The work here is re-siting the arming and native loading off the absent full_stack_env onto upstream's own machinery. Lanes (armed by default for a served fixed-M4 Flash-Next pack, per-lane opt-out): - ple_cached_aux: a native CPU-stream provider stages the fixed 64 M4 n-gram rows and the auxiliary embedding plane is produced with mx.async_eval outside the compiled verifier. The stock owner-side row cache is preserved; declines to stock with a printed reason when the ple_cpu_rows extension is not built. - qsa_pooled_rowsel: the twelve QSA indexers' pooled-key preparation binds the pool kernel metadata once per indexer and shares one inv_freq object. Rebase changes vs the closed-PR commit: - mtplx/qwen4_aux_lanes.py rewritten off full_stack_env: primary keys are MTPLX_QWEN4_PLE_CACHED_AUX / MTPLX_QSA_POOLED_ROWSEL, the PR 391 MTPLX_FABLE_* names kept as aliases (primary wins when both set). - mtplx/server/openai.py: the two keys join the fixed-M4 lane_defaults and _QWEN4_PORT_KEYS (so the existing pop-loop kill-switch honours KEY=0), with an alias pre-step mirroring an operator's MTPLX_FABLE_* export onto the primary. - mtplx/qsa_pooled_rowsel.py: op-diet contract re-pointed from the absent fable_opdiet_enabled to upstream's qwen4_opdiet_enabled (MTPLX_QWEN4_OPDIET). - mtplx/runtime.py: the two installs run after the fixed-M4 verify install, logging instead of the removed _print_install_receipt. - mtplx/profiles.py: the two keys added to MODEL_RUNTIME_ENV_OVERRIDE_KEYS so normalize_runtime_env_overrides accepts the server-stamped values. - mtplx/native/__init__.py: a minimal PLE-only loader (load_ple_cpu_rows_extension / ple_cpu_rows_unavailable_reason); PR 391's qsa_sparse_gqa loader is not reproduced (upstream loads native QSA via kernels/qsa_prefill_direct.py). - scripts/bundle_native_runtime_wheel.py: accepts and Developer-ID signs the new mtplx_native_ple_cpu_rows extension alongside mtplx_qsa_kernels, so a notarized release wheel carries a signed ple_cpu_rows Mach-O. - scripts/fable/setup_over100_venv.sh: builds only ple_cpu_rows. 772f5be's opt-in interleaved n-gram row cache (MTPLX_NGRAM_ROW_FILE, default off) touches the same _SidecarGather rows but at the disk-layout layer; it is orthogonal to this runtime-scheduling lane and does not subsume it. CPU tests (venv mlx 0.32.2, no GPU): 84 passed across the six lane test files plus the wheel-bundler test.

Implements the plug-in speculative-cascade deferral rule from Narasimhan et al., "Faster Cascades via Speculative Decoding" (arXiv:2405.19261 v2), Section 4.3 Equation (10), with the speculative execution of Algorithm 4, beside the existing typical-acceptance lane. At each draft position with target distribution p and draft distribution q, Equation (10) defers to the target iff max_v q(v) < max_v p(v) - alpha * D_TV(p, q) with D_TV(p, q) = sum_v max(0, p(v) - q(v)) over the scored top-k support. Not deferring means the draft is good enough (the cascade target pi = q, so Algorithm 4's min(1, pi/q) = 1): accept the draft token with no coin. Deferring sets pi = p and runs the exact speculative law unchanged (min(1, p(x_t)/q(x_t)) coin + residual norm(max(0, p - q))), so the cascade path is a strict superset of the exact rule with a draft-accept shortcut. The decision is deterministic; the coin only appears on the deferred path, so with the lane off the RNG stream and verify path are byte-identical. q is the native MTP head's scored rows already in the verify loop as draft_probs[depth_index] (a SparseDistribution over the head's FR-Spec scored vocabulary, the same q the exact rule uses), so the rule reads max_q and the top-k mass for D_TV from that object and adds no draft forward. The knob is --cascade-threshold / MTPLX_FABLE_CASCADE_THRESHOLD (the deferral cost alpha), read at use, default OFF (unset); any set value including 0 turns it on, and higher alpha defers less. It is mutually exclusive with --typical-threshold and fails loud (SystemExit at serve start, ValueError in the verify setup) if both are set. A /health cascade_acceptance install report and a per-request [cascade-accept] verdict line (positions, accepted, resamples, accept_rate, mean_divergence) mirror the typical lane. Note per the paper's Lemma 3: alpha * D_TV is subtracted, so a larger disagreement defers less; a diverging draft that also loses peak confidence defers, while a confident-but-wrong draft is accepted. That behavior is pinned explicitly in the tests. CPU tests cover the rule, arming (read at use, served order, default off), mutual exclusion, and exact-mode-off equivalence. docs/perf/pr478-cascade-acceptance.md carries the citation and the Problem/Change/Effect/Exactness/Files/Switch writeup.

Rename the speculative-cascade perf note to docs/perf/qwen38-cascade-acceptance.md (matching the qwen38-* perf-note naming) and repoint the flag help and /health note references to it. Add a "Recommended alpha grid for the 16K sweep" section: the paper varies alpha continuously (Section 6, Figure 2) with no fixed grid, so the grid is set from Equation (10)'s structure and this model's measured scale (exact MTP acceptance 0.43-0.47 -> D_TV ~= 1 - accept ~= 0.55; target-row entropy 0.83-1.81 nats -> target peak ~0.35-0.65). Grid {0.0, 0.5, 1.0, 2.0} from maximal deferral to near all-accept; recommended HumanEval operating point alpha = 0.5.

… arm-G sweep

Add the provenance bridge to docs/perf/qwen38-cascade-acceptance.md: arm G and the pending HumanEval cell were measured on served code 2eac2fee; this branch re-parents the cascade mode onto the youssofal#475 served base 27d5ff6 (the same base youssofal#478 uses), so youssofal#478 (typical) and this PR (cascade) are alternative-mode peers. Per-function sha256 proof shows the cascade rule (total_variation, _peak_probability, cascade_defer_decision, the batched/lazy verify branches, the readers, the verdict, the /health payload, the --cascade-threshold arg) is byte-identical to 2eac2fee, so the measurements transfer; the three forced glue deltas (lazy elif->if, exact-block re-indent + target_p_for_cache hoist, the mutual-exclusion reading the typical env name directly) are non-behavioral.

Add the cascade context-ladder decode chart (cascade_decode_by_context.svg): exact pairing and cascade alpha 0.0/0.5/1.0/2.0 across 1K-128K, fastest of seeds with min-max variance bands; 16,384 merged from the arm-G sweep; 261,120 absent (every arm OOMs on the youssofal#475 base without youssofal#482). Update the charts manifest.

… path The re-parent that dropped the typical lane deleted the [cascade-accept] verdict emission from generate_mtpk together with the adjacent [typical-accept] block, between _attach_runtime_diagnostics and the return. The rule still engaged on the served path (cascade_* counters and VerifyStats cascade fields moved, /health reported enabled), but the verdict line never printed, so the engagement gate that parses threshold=/positions= off it saw nothing at every alpha. The block had survived only in generate_mtpa, which is not the served loop. Restore the emission in generate_mtpk, guarded by if _cascade_active, byte-identical to the pre-re-parent block. The acceptance rule is untouched (per-function sha256 of cascade_defer_decision, total_variation, _peak_probability and the generation.py cascade-accept branch all match d8efc3f). Add a served-order CPU test that arms the lane via the env reader at use, drives a temperature>0 mocked verify loop through generate_mtpk, and asserts one verdict line with threshold == alpha and positions > 0; it fails on the unfixed tree.

…te_mtpa generate_mtpa is an upstream function (present on base 27d5ff6) with zero callers anywhere in the tree. The cascade feature commit d8efc3f accidentally added a [cascade-accept] verdict block to it while moving the block around during the re-parent; it references cascade counters that generate_mtpa never defines, so it is both dead and broken (would NameError if the function were ever called). It never ran -- the served verdict is in generate_mtpk. Remove only that copy, restoring generate_mtpa byte-identical to its upstream base form. The acceptance rule is untouched (per-function sha256 unchanged); the served generate_mtpk verdict stays.

… precisely The reference carried a wrong author list and an unverified venue. The paper's authors are Narasimhan, Jitkrittum, Rawat, Kim, Gupta, Menon and Kumar; cite it as arXiv:2405.19261 v2 (2024) and drop the ICLR 2025 claim, which is not verifiable from the paper itself. Name the implemented rule as the paper does: it is r-hat_OPT, Equation (10), the plug-in ESTIMATOR of the optimal speculative-cascade deferral rule (Lemma 4, Equation (9)), which replaces that rule's ground-truth expected 0-1 losses with one minus each model's max probability. It is neither the optimal rule nor an oracle; the oracle needs expectations under the ground-truth distribution. The Diff rule (Equation (5)) is the sequential-cascade oracle and is not implemented here. Lemma 3 is why the deferral cost carries the alpha times D_TV term. Equation, Section, Algorithm and Lemma numbers are otherwise unchanged. Comment and docstring text only; no functional change. mtplx/sampling.py is deliberately NOT touched: its cascade_defer_decision docstring carries the same stale citation, but that function's full source is published as sha256 3ba053b7097fa91e and asserted byte-identical to the measured 2eac2fee, so editing it would invalidate a provenance line already live on the pull request.

cascade_decode_vs_alpha, cascade_accept_vs_alpha and cascade_decode_by_context now include the two added grid points (six cascade alphas: 0.0, 0.25, 0.5, 0.75, 1.0, 2.0). Fastest-of-seeds with min-max bands, same generators. Charts only; no code or measurement change.

cascade_decode_vs_alpha and cascade_accept_vs_alpha now plot both deferral rules: OPT (Equation 10, alpha 0/0.25/0.5/0.75/1/2) and TokenV3 (Equation 15, alpha 0.25/0.5/0.75/0.9/0.95), with distinct markers and dash, the same exact-law and typical-0.09 reference lines, and the legend outside the axes. Captions name both rules and state that equal alphas are NOT comparable across them, since TokenV3's alpha is a fraction of the target peak probability. Charts and manifest only; nothing under mtplx/.

…ade-rule r_OPT (arXiv:2405.19261 v2, Eq. 10) decides between q and p by comparing only their peaks, so a drafted token x_t~q that does not maximise q can be accepted because q is more peaked than p even when the token is poor (Sec. 4.4). This is the cause of the HumanEval loss at every alpha. Implement the token-specific rules that judge the drafted token: tokenv1 (Eq. 13): defer v iff q(v) < max_v' p(v') - alpha tokenv2 (Eq. 14): defer v iff p(v) < max_v' p(v') - alpha tokenv3 (Eq. 15): defer v iff p(v) < max_v' p(v') * (1 - alpha) On a deferred token the exact coin/residual runs with the token-specific target pi_Token (Eq. 11) instead of p -- Algorithm 6 (Appendix D) is GenSpecSample(q, p, pi_Token). A token in Top_alpha has pi(v)=q(v)+p(v)*eta, so the coin accepts it with probability 1 (accept, no coin); a deferred token has pi(v)=p(v)*eta. Rule selector: env MTPLX_FABLE_CASCADE_RULE / flag --cascade-rule, default opt for backward compatibility, same alpha knob, same mutual exclusion with typical. /health reports the rule; the [cascade-accept] verdict line names it. r_OPT's executed code is unchanged: the token-specific branches are a new elif above the OPT branch at both verify sites, so opt (or unset) is byte-for-byte as before. Also correct the stale citation in cascade_defer_decision's docstring (Mreddy/ICLR 2025 -> Narasimhan, Jitkrittum, Rawat, Kim, Gupta, Menon, Kumar, arXiv:2405.19261 v2 (2024)). Since that touches the sha256-hashed function body, docs/perf restates the proof as an AST code-hash (docstring stripped) showing the OPT rule code identical to 2eac2fee/d8efc3f5 while the docstring text changed. Tests: TokenV3 defers a confidently-wrong draft that OPT accepts, Top_alpha accepted, pi_TokenV3 matches Eq. 11 numerically, TokenV1 rule, served-order arming naming the rule, rule-selector read-at-use + default opt + fail-loud. The block-verify structural guard now counts six exact-coin sites (two shipped exact, two OPT defer, two token-specific defer), one coin per depth.

… guidance Add a measured-results section to docs/perf/qwen38-cascade-acceptance.md: the OPT HumanEval strict grid by alpha (0.9024/0.8537/0.7805/0.7073; alpha 1.0 dnf, 2.0 not run), the TokenV3 decode tok/s grid by alpha (84.09/88.70/95.30/96.52/107.10), the TokenV3 alpha 0.95 quality point (0.9695 strict / 1.000 completed / 3.05% trunc at 107.10 tok/s), the exact and typical-0.09 baselines, and the equal-speed comparison at ~105 tok/s. Add the Sec. 4.4 mechanism paragraph (why the peak rule admits poor tokens as alpha rises and the token-specific rule does not). Extend the Switch section with the --cascade-rule selector (opt default for backward compatibility; tokenv1/2/3), noting TokenV3's alpha is a fraction of the target peak so its useful range is >= 0.9, and update the /health and verdict-line docs for the rule_name/rule fields and the rule= verdict field. A placeholder line marks the pending TokenV3 alpha 0.75 quality point. Docs only.

Measured 16K HumanEval cell for TokenV3 at alpha 0.75: strict 0.9695 (159/164), completed-task 0.9876, truncation 1.83% (mean 2,652 tokens), cascade acceptance 0.820, at 95.30 tok/s (+15.1% vs exact 82.85), wall 1 h 19 m. Replaces the placeholder, and adds the closing numbers line (TokenV3 0.9695 at both alpha 0.75 and 0.95; OPT 0.9024 -> 0.7073). Docs only.

cascade_decode_by_context now carries eight arms: the exact pairing, the OPT rule at alpha 0.0/0.25/0.5/0.75/1.0/2.0, and the TokenV3 rule at alpha 0.95 across 1K-128K (fastest of three seeds, min-max bands). 261,120 is absent because every arm exceeds the memory knob on the youssofal#475 base without youssofal#482. Chart and manifest only; nothing under mtplx/.

David's ruling (2026-09-09): TokenV3 is the only cascade rule with decent accuracy (HumanEval 0.9695 strict at alpha 0.95, equal to exact, while OPT loses at every alpha: 0.9024 at 0.0 down to 0.7073 at 0.75), so it becomes the default rule. Flip the default of MTPLX_FABLE_CASCADE_RULE / --cascade-rule from opt to tokenv3 everywhere the default is defined: the reader _cascade_accept_rule() (unset -> tokenv3), the /health cascade_acceptance.rule_name default, and the --cascade-rule help text. The argparse default stays the None sentinel so an unset flag does not override a shell-set env; the effective default resolves in the reader. opt, tokenv1 and tokenv2 stay selectable (env or flag). The mode itself is unchanged and still OFF by default: with the alpha knob unset the exact speculative law runs, so this changes only which rule engages once --cascade-threshold is set. The OPT rule body is untouched: the AST code hash (docstring stripped) of cascade_defer_decision is byte-identical to 2eac2fee/d8efc3f5. Tests: the default-selector test flips to tokenv3, a new test asserts env opt still selects OPT (backward compatibility), and the exact-off and block-verify guards still pass. Docs switch section updated with the new default, why, and how to select opt.

Two charts comparing this pull request's operating points against youssofal#475 and youssofal#478, both re-derived from the receipt json rather than from any table: cascade_vs_475_478_16k.svg decode at 16,384 tokens for release 2.11.2 (exact), youssofal#475 (exact), youssofal#478 (typical 0.09), OPT alpha 0.25, and TokenV3 alpha 0.75 and 0.95, each bar annotated with its own HumanEval strict pass@1 cascade_vs_475_478_ladder.svg decode against context size, 1,024 to 131,072, for the same four arms that have a full ladder Bar height and line point are the fastest seed, every band is min-max, and acceptance mode is in each label: a tok/s figure is not readable without it. The alphas of the two rules are different quantities and the captions say so. No arm carries a 261,120 point, because on the youssofal#475 base without youssofal#482 every arm on this pack exceeds the memory knob.

David: accept rate is never defined for speculative cascade. The served verdict's accept_rate = cascade_accepted / cascade_positions is a KEPT-DRAFT rate over cascade-decided positions (no-defer accepts plus coin-accepted deferred tokens), not the paper's deferral rate r. Add a cascade_deferred counter, incremented on every deferral (the _defer / OPT defer branch, before the coin) at all four verify sites (OPT and TokenV3, batched and lazy). Expose defer_rate = cascade_deferred / cascade_positions in VerifyStats (cascade_deferred, cascade_defer_rate), the [cascade-accept] verdict line (deferred=, defer_rate=), and the /health cascade_acceptance payload (documented rates block). accept_rate is unchanged, now documented as the kept-draft rate; accept_rate + resample_rate == 1 over decided positions, while defer_rate is independent. Rule bodies untouched: the docstring-stripped AST hashes of cascade_defer_decision, total_variation and _peak_probability are byte-identical to 2eac2fee/d8efc3f5. Tests: defer/accept/resample consistency (accepted + resamples == positions; resamples <= deferred <= positions; defer_rate == deferred/positions; deferred>0 for a divergent draft under TokenV3), served-order emission of defer_rate, and exact-off leaves the cascade counters zero. docs/perf gains a Definitions block giving the three quantities, their formulas, and which artifact each comes from (verdict line / VerifyStats vs the receipt's mtp_accept_rate).

The TokenV3 series in cascade_decode_by_context.svg had a hole exactly at 16,384. The by-context renderer merged the 16K rung from battery475/g only, but the two rules were swept separately: the arm-T (TokenV3) 16,384 windows live in battery475/t, and the context ladder appended only the non-16K rungs for those arms. With no row to merge, the point was silently dropped. render_cascade_ladder.py now stages t/manifest.tsv alongside g/, reads past that manifest's leading comment line, and admits rc=65 rows, which are the arm-T gate-regex false negative on valid measured windows. The rc gate itself stays: ladder-cascade carries an rc=1 duplicate row whose receipt_dir points at the 65,536 runroot under a 131,072 ctx, and dropping the gate entirely would let that mis-aliased row through the ctx cross-check. The TokenV3 line now carries six points, 1K through 128K, with 16,384 at 107.10 tok/s (96.70-107.10, n=3), the same fastest-of-seeds and min-max rule as every other point. No other series or value changes.

cascade_accept_vs_alpha.svg plotted the kept-draft rate on a left y axis and tokens per cycle on a right y axis, and its legend named the series '(left)' and '(right)' without saying what those axes were; the right axis itself was clipped out of the figure. It is now two stacked panels on one alpha axis: top = cascade kept-draft rate (Definitions (c), cascade_accepted / cascade_positions), bottom = tokens per verify cycle, one line per rule (OPT Eq 10, TokenV3 Eq 15). Same data, same sidecar.

cascade_accept_vs_alpha.svg is replaced by cascade_kept_draft_vs_alpha.svg (kept-draft rate, Definitions (c)) and cascade_tokens_per_cycle_vs_alpha.svg (tokens per verify cycle), one quantity per chart, one line per rule. Same data, same sidecar.

… as the reference David's rulings applied to the youssofal#485 chart set: RULING 1 ("youssofal#475 is NOT exact"): no chart, legend, reference line, caption or manifest labels youssofal#475 or release 2.11.2 as "exact". The acceptance-off state is now named "acceptance mode off" / "cascade off"; the youssofal#475 arm is "youssofal#475 (base)". * cascade_decode_vs_alpha: reference lines relabeled "cascade off, youssofal#475 base (82.80)" and (see below) "typical 0.2 (99.16)". * cascade_decode_by_context: arm-C-caspair legend "exact (no cascade)" -> "cascade off (youssofal#475 base)". * cascade_vs_475_478_16k / _ladder: "youssofal#475 (base)", "acceptance mode off" / "cascade off"; the exact-acceptance disclaimer now names the ordinary speculative-decoding acceptance law, not any arm. RULING 2 (typical 0.2, not 0.09, is the youssofal#478 reference): every youssofal#485-vs-youssofal#478 comparison now references youssofal#478 at typical threshold 0.2 (pooled 16,384 window, 99.16 tok/s fastest of n=9, HumanEval strict pass@1 0.9695), replacing typical 0.09. Applied to cascade_decode_vs_alpha (dotted reference line), the 16K bars and the context ladder. The §2.4 ABAB (typical 0.09 vs TokenV3 0.95) is a separate measurement and is unchanged. Charts re-rendered from receipts (fastest-of-seeds, min-max band); manifest bytes/sha256 refreshed. Docs only; no receipts touched.

… as the reference David's rulings applied to the youssofal#485 chart set: RULING 1 ("youssofal#475 is NOT exact"): no chart, legend, reference line, caption or manifest labels youssofal#475 or release 2.11.2 as "exact". The acceptance-off state is now named "acceptance mode off" / "cascade off"; the youssofal#475 arm is "youssofal#475 (base)". * cascade_decode_vs_alpha: reference lines relabeled "cascade off, youssofal#475 base (82.80)" and (see below) "typical 0.2 (99.16)". * cascade_decode_by_context: arm-C-caspair legend "exact (no cascade)" -> "cascade off (youssofal#475 base)". * cascade_vs_475_478_16k / _ladder: "youssofal#475 (base)", "acceptance mode off" / "cascade off"; the exact-acceptance disclaimer now names the ordinary speculative-decoding acceptance law, not any arm. RULING 2 (typical 0.2, not 0.09, is the youssofal#478 reference): every youssofal#485-vs-youssofal#478 comparison now references youssofal#478 at typical threshold 0.2 (pooled 16,384 window, 99.16 tok/s fastest of n=9, HumanEval strict pass@1 0.9695), replacing typical 0.09. Applied to cascade_decode_vs_alpha (dotted reference line), the 16K bars and the context ladder. The §2.4 ABAB (typical 0.09 vs TokenV3 0.95) is a separate measurement and is unchanged. Charts re-rendered from receipts (fastest-of-seeds, min-max band); manifest bytes/sha256 refreshed. Docs only; no receipts touched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…al off" David's RULING 1 ("youssofal#475 is NOT exact"): the typical sweep's acceptance-off arm must not be labeled "exact". It is the ordinary acceptance law with typical acceptance turned off, so it is now named "typical off". * acc_vs_length, acc_vs_speed, acc_vs_length_mbpp, acc_vs_speed_mbpp: series label "exact (off)" -> "typical off"; the MBPP single-series title/caption "exact arm" -> "typical-off arm". * tok_s_vs_passk: the base point "exact (off)" -> "typical off". * pr391-charts/pr391-typical-acceptance-decode: legend "exact (lane off)" -> "typical off"; footnote "+/-1 task of exact (153/164)" -> "of typical off (153/164)". * manifest.json alt/caption "exact arm" -> "typical-off arm"; bytes refreshed. RULING 2 does not change this PR's charts: the typical sweep shows every threshold (off / 0.09 / 0.2 / 0.4) as its own series, not a youssofal#485-vs-youssofal#478 comparison, so there is no single youssofal#478 reference to switch here. Charts re-rendered from the sweep receipts. Docs only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…al off" David's RULING 1 ("youssofal#475 is NOT exact"): the typical sweep's acceptance-off arm must not be labeled "exact". It is the ordinary acceptance law with typical acceptance turned off, so it is now named "typical off". * acc_vs_length, acc_vs_speed, acc_vs_length_mbpp, acc_vs_speed_mbpp: series label "exact (off)" -> "typical off"; the MBPP single-series title/caption "exact arm" -> "typical-off arm". * tok_s_vs_passk: the base point "exact (off)" -> "typical off". * pr391-charts/pr391-typical-acceptance-decode: legend "exact (lane off)" -> "typical off"; footnote "+/-1 task of exact (153/164)" -> "of typical off (153/164)". * manifest.json alt/caption "exact arm" -> "typical-off arm"; bytes refreshed. RULING 2 does not change this PR's charts: the typical sweep shows every threshold (off / 0.09 / 0.2 / 0.4) as its own series, not a youssofal#485-vs-youssofal#478 comparison, so there is no single youssofal#478 reference to switch here. Charts re-rendered from the sweep receipts. Docs only.

…senses of "exact" Against pr-bodies/GLOSSARY.md, the same pass the pull request bodies had. The arm table gave an acceptance mode for D and E only, so A and C read as though they had none. Both run with typical off, and their rows now say so. "Exact" appears once here, in the optimization-class sense: an exact optimization is byte-for-byte the stock path, against a rounding-class one that differs only by floating-point rounding. Nothing in the file bound that, so it was indistinguishable from the exact acceptance LAW that youssofal#478 and youssofal#485 use the same word for. The paragraph now binds it and states that it never labels an arm, since arms A and C are rounding-class builds overall. The prose mixed short context labels with exact counts; prose now gives the counts and the short form is left to table row labels.

cascade_humaneval_by_alpha.svg: strict and completed-task pass@1 as grouped bars for cascade off (youssofal#475 base), the OPT rule at alpha 0.0/0.25/0.5/0.75 and the TokenV3 rule at 0.75/0.95, with the cascade-off strict level as a reference line, so the quality cost of the OPT rule is visible next to TokenV3. Data: evalsweep478/armG_summary.json and the youssofal#478 sweep's typical-off cell; nothing re-measured.

…ating-point chart cascade_humaneval_by_alpha.svg gains a youssofal#478 typical 0.2 bar pair (strict 0.9695, completed-task 1.0000, the youssofal#478 sweep's own cell on the same base, seed and sampler) beside cascade off, so the quality-neutral reference the speed comparisons use is on the quality chart too. Both sweep cells are now read from evalsweep478/results.csv instead of being pasted constants.

… queue

bpmforge · 2026-09-18T13:05:11Z

Bottom line: this PR gave us about 25% more speed on short prompts and about 40% more on long ones, on a different Flash-Next model than the one you tested. It still works after merging it onto 2.11.3. The catch is the same one you list: the output is no longer exactly the model's own distribution.

What we ran


Mac	M5 Max, 128 GB, macOS 27.0
Model	Qwen3.8-Flash-Next-Uncensored (orcarouter), built with `forge` at 4-bit
Runtime	MTPLX 2.11.2 and 2.11.3, this PR applied on top of each
Sampler	temperature 1.0, top-p 0.95, top-k 20 (the pack's default)
Prompts	3 coding prompts, 512 tokens each, 2 runs, median shown
Long prompt	32,000 tokens, cold (no cache)
Fans	default, no fan control, so expect about 5% noise
Dates	2026-09-16 and 2026-09-18

Results (tokens per second, higher is better)

Runtime	Setting	Short prompt	32K prompt	Exact output?
2.11.2	stock	86	64	yes
2.11.2 + this PR	cascade 0.95	107	89	no
2.11.2 + this PR	cascade 0.95 + KV cache q8	109	91	no
2.11.3	stock	87	70	yes
2.11.3 + this PR	stock acceptance	93	74	yes
2.11.3 + this PR	cascade 0.95 + KV cache q8	108	96	no

What "exact output" means

Flash-Next has a small draft head that guesses the next 3 tokens. The big model checks the guesses in one pass, so right guesses come almost free. That is where the speed comes from.

Exact (stock): the check uses the standard accept/reject rule. Over many runs the tokens come out exactly as if the big model had picked every one itself. The draft head can only make things faster, never change the answers.
Cascade (this PR): the check is looser. A guess that is "close enough" to the big model's choice is accepted without the coin flip. More guesses pass per step (3.0 to 3.5), so it is faster, but the output is now tilted a little toward what the draft head likes. It is no longer guaranteed to match the big model's own distribution.

In practice the difference is small (your HumanEval numbers are inside noise, and we saw nothing odd in our runs), but it is a real change in what the model emits, which is why it belongs behind an opt-in flag. Rule of thumb: exact for evals or anything you will compare against, cascade for everyday chat and coding when speed matters more.

What it means for this PR

The speed-up is real and repeats on a second model. It comes from accepting more draft tokens per step (3.0 to 3.5), not from a faster step.
2.11.3 on its own did not change our numbers. Its fixes target a different workload (9K prompts with thinking off).
Merging this PR onto 2.11.3 took some hand work: both sides fixed the same flag bug in different ways. I kept upstream's fix and put your new lanes on top. Full test suite passes (7386 tests). The merged branch is at bpmforge/MTPLX, branch pr485-on-2113, if it saves you a rebase.

davidtai added 11 commits September 7, 2026 07:36

docs(cascade): decode-vs-alpha and acceptance-vs-alpha charts for the…

9436965

… arm-G sweep

davidtai requested a review from youssofal as a code owner September 8, 2026 17:25

davidtai added 9 commits September 8, 2026 20:59

davidtai changed the title ~~Speculative-cascade acceptance: a second opt-in lossy decode mode for Qwen3.8 Flash-Next (alternative to #478 typical)~~ Speculative-cascade acceptance (opt-in, off by default): token-specific rule holds HumanEval at +29% decode; peak rule does not Sep 9, 2026

davidtai added 2 commits September 9, 2026 15:05

davidtai added 5 commits September 9, 2026 16:03

davidtai mentioned this pull request Sep 9, 2026

[FEAT] Bare-Speed kernel adaptations with arm FR-Spec and the M=4 stage-3 combine on the Qwen3.8 Flash-Next Q4/g64 pack #488

Open

davidtai force-pushed the perf/qwen38-cascade-acceptance branch from 8c467b4 to d41a68e Compare September 9, 2026 22:30

davidtai added 2 commits September 9, 2026 18:57

davidtai added a commit to davidtai/MTPLX-STREAMING that referenced this pull request Sep 10, 2026

Merge PR youssofal#485 (perf/qwen38-cascade-acceptance) from upstream…

c0102fc

… queue

davidtai mentioned this pull request Sep 14, 2026

Qwen3.8 Flash Next: port MTPLX acceptance work (typical, cascade, block verify, seeded correction) — engine diff and plan ddalcu/mlx-serve#419

Closed

1 task

Navigation Menu

[FEAT] Speculative-cascade acceptance (opt-in, off by default): token-specific rule holds HumanEval at +29% decode; peak rule does not - #485

[FEAT] Speculative-cascade acceptance (opt-in, off by default): token-specific rule holds HumanEval at +29% decode; peak rule does not#485
davidtai wants to merge 30 commits into
youssofal:mainfrom
davidtai:perf/qwen38-cascade-acceptance

davidtai commented Sep 8, 2026 •

edited

Loading

Uh oh!

bpmforge commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

davidtai commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speculative-cascade acceptance: a second opt-in speed optimization for Qwen3.8 Flash-Next

Definitions

1. Strategy and references

1.1 The OPT rule (Equation (10))

1.2 The TokenV3 rule (Equation (15))

1.3 Selector and defaults

2. Speed results

2.1 OPT rule, decode at 16,384 tokens (higher is better)

2.2 OPT rule, prefill, TTFT and peak memory at 16,384

2.3 TokenV3 rule, decode at 16,384 tokens

2.4 Paired ABAB: typical acceptance 0.09 against TokenV3 alpha 0.95

2.5 Context sweep

Decode tok/s by context

Prefill tok/s by context

TTFT s by context

Wall s by context

Peak memory GB by context

2.6 Comparison with #475 and #478

3. Quality results

3.1 The two rules at equal decode speed

3.2 The OPT alpha curve

3.3 The TokenV3 alpha curve

4. Benchmark method

5. Switch and observability

6. Provenance and file map

Uh oh!

bpmforge commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidtai commented Sep 8, 2026 •

edited

Loading