Cannot compact reliably under pi.dev (v0.84.4)

[roehnan]

Category

Model inference issue

What happened

I am not sure if this is a Nativ/mlx-vlm-server issue but when I try to compact, I often get finish_reason=length when length should not be a problem. My current context is 48k/262k, so there should be plenty of room to do the compaction. It appears that the reqeuest is terminating prematurely.

Diagnostics

Environment

• Nativ: 0.3.6 (202608311942)
• macOS: macOS 26.6.2 (Build 25G83)
• Chip: Apple M3 Max
• Memory: 96 GB total, 39.24 GB in use
• mlx-vlm: 0.6.17

Model & Server

• Server: running
• Selected model: On demand
• Loaded model: mlx-community/Ornith-1.5-35B-A3B-8bit
• Max output tokens: 2048
• Context window: model default
• KV quantization: off
• Speculative decoding: off
• Prefix caching: off
• Thinking: on
• Launch arguments: --host 127.0.0.1 --port 8000 --max-tokens 2048

Inference

• Requests completed: 9
• Requests failed: 0
• In flight: 0
• Prompt tokens: 294K
• Generated tokens: 5.1K
• Total processed tokens: 299K
• Avg decode speed: 47.4 tok/s
• Avg request speed: 10.1 tok/s
• Uptime: 36m 17s
• — Latest request —
• Model: mlx-community/Ornith-1.5-35B-A3B-8bit
• Endpoint: /chat/completions
• Prompt tokens: 12.5K
• Completion tokens: 2K
• Generated tokens: 2K
• Total tokens: 14.6K
• Time to first token: 13.3s
• Prefill speed: 945.8 tok/s
• Decode speed: 48.8 tok/s
• Elapsed time: 55.2s
• Peak memory: 65.68 GB
• — Runtime —
• Loaded model: mlx-community/Ornith-1.5-35B-A3B-8bit
• Queue depth: 0
• Batching: On
• APC: Off
• Context limit: 262K
• Tool parser: qwen3_coder
• — All-time —
• Requests completed: 336
• Requests failed: 0
• Prompt tokens: 12.6M
• Generated tokens: 82.7K
• Total processed tokens: 12.7M
• Avg decode speed: 34.5 tok/s
• Avg request speed: 3.7 tok/s

Server output (last 60 lines)

2026-09-01 05:58:01,866 - INFO - Decode progress: request=111fb6990 generated_tokens=1470 elapsed=30.101s rate=46.5 tok/s
2026-09-01 05:58:02,074 - INFO - Decode progress: request=111fb6990 generated_tokens=1480 elapsed=30.310s rate=47.9 tok/s
2026-09-01 05:58:02,280 - INFO - Decode progress: request=111fb6990 generated_tokens=1490 elapsed=30.515s rate=49.4 tok/s
2026-09-01 05:58:02,488 - INFO - Decode progress: request=111fb6990 generated_tokens=1500 elapsed=30.724s rate=41.6 tok/s
2026-09-01 05:58:02,694 - INFO - Decode progress: request=111fb6990 generated_tokens=1510 elapsed=30.929s rate=49.7 tok/s
2026-09-01 05:58:02,894 - INFO - Decode progress: request=111fb6990 generated_tokens=1520 elapsed=31.129s rate=51.0 tok/s
2026-09-01 05:58:03,091 - INFO - Decode progress: request=111fb6990 generated_tokens=1530 elapsed=31.327s rate=50.4 tok/s
2026-09-01 05:58:03,294 - INFO - Decode progress: request=111fb6990 generated_tokens=1540 elapsed=31.530s rate=49.2 tok/s
2026-09-01 05:58:03,497 - INFO - Decode progress: request=111fb6990 generated_tokens=1550 elapsed=31.732s rate=43.0 tok/s
2026-09-01 05:58:03,701 - INFO - Decode progress: request=111fb6990 generated_tokens=1560 elapsed=31.936s rate=52.0 tok/s
2026-09-01 05:58:03,901 - INFO - Decode progress: request=111fb6990 generated_tokens=1570 elapsed=32.137s rate=49.9 tok/s
2026-09-01 05:58:04,103 - INFO - Decode progress: request=111fb6990 generated_tokens=1580 elapsed=32.338s rate=49.1 tok/s
2026-09-01 05:58:04,306 - INFO - Decode progress: request=111fb6990 generated_tokens=1590 elapsed=32.541s rate=50.2 tok/s
2026-09-01 05:58:04,515 - INFO - Decode progress: request=111fb6990 generated_tokens=1600 elapsed=32.750s rate=40.0 tok/s
2026-09-01 05:58:04,718 - INFO - Decode progress: request=111fb6990 generated_tokens=1610 elapsed=32.953s rate=48.3 tok/s
2026-09-01 05:58:04,919 - INFO - Decode progress: request=111fb6990 generated_tokens=1620 elapsed=33.154s rate=48.9 tok/s
2026-09-01 05:58:05,123 - INFO - Decode progress: request=111fb6990 generated_tokens=1630 elapsed=33.358s rate=47.7 tok/s
2026-09-01 05:58:05,329 - INFO - Decode progress: request=111fb6990 generated_tokens=1640 elapsed=33.565s rate=48.1 tok/s
2026-09-01 05:58:05,543 - INFO - Decode progress: request=111fb6990 generated_tokens=1650 elapsed=33.779s rate=40.0 tok/s
2026-09-01 05:58:05,749 - INFO - Decode progress: request=111fb6990 generated_tokens=1660 elapsed=33.984s rate=48.4 tok/s
2026-09-01 05:58:05,963 - INFO - Decode progress: request=111fb6990 generated_tokens=1670 elapsed=34.199s rate=48.6 tok/s
2026-09-01 05:58:06,172 - INFO - Decode progress: request=111fb6990 generated_tokens=1680 elapsed=34.407s rate=47.3 tok/s
2026-09-01 05:58:06,377 - INFO - Decode progress: request=111fb6990 generated_tokens=1690 elapsed=34.612s rate=49.3 tok/s
2026-09-01 05:58:06,583 - INFO - Decode progress: request=111fb6990 generated_tokens=1700 elapsed=34.818s rate=41.5 tok/s
2026-09-01 05:58:06,782 - INFO - Decode progress: request=111fb6990 generated_tokens=1710 elapsed=35.017s rate=50.0 tok/s
2026-09-01 05:58:06,980 - INFO - Decode progress: request=111fb6990 generated_tokens=1720 elapsed=35.215s rate=49.5 tok/s
2026-09-01 05:58:07,186 - INFO - Decode progress: request=111fb6990 generated_tokens=1730 elapsed=35.422s rate=48.5 tok/s
2026-09-01 05:58:07,387 - INFO - Decode progress: request=111fb6990 generated_tokens=1740 elapsed=35.622s rate=50.8 tok/s
2026-09-01 05:58:07,589 - INFO - Decode progress: request=111fb6990 generated_tokens=1750 elapsed=35.825s rate=43.3 tok/s
2026-09-01 05:58:07,789 - INFO - Decode progress: request=111fb6990 generated_tokens=1760 elapsed=36.025s rate=47.6 tok/s
2026-09-01 05:58:07,993 - INFO - Decode progress: request=111fb6990 generated_tokens=1770 elapsed=36.228s rate=50.2 tok/s
2026-09-01 05:58:08,199 - INFO - Decode progress: request=111fb6990 generated_tokens=1780 elapsed=36.434s rate=47.8 tok/s
2026-09-01 05:58:08,405 - INFO - Decode progress: request=111fb6990 generated_tokens=1790 elapsed=36.640s rate=48.5 tok/s
2026-09-01 05:58:08,612 - INFO - Decode progress: request=111fb6990 generated_tokens=1800 elapsed=36.847s rate=42.3 tok/s
2026-09-01 05:58:08,813 - INFO - Decode progress: request=111fb6990 generated_tokens=1810 elapsed=37.048s rate=50.6 tok/s
2026-09-01 05:58:09,019 - INFO - Decode progress: request=111fb6990 generated_tokens=1820 elapsed=37.254s rate=50.7 

Diagnostics truncated — the full report was copied to your clipboard; paste it here to replace this body.

[linear-code]

NEY-129

[Lazarus-931]

hey @roehnan, thanks for the issue!

we're working on redesigning apc in mlx-vlm, which is why this is broken

we'll do a release soon once work is done :)