[ICS] Eval bug: OpenVINO: low tg

[haarika-madaka]

Name and Version

root@ae0463242527:/app# ./llama-cli --version
version: 0.2.0-dev (build 10615, commit f280b2698)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

OpenVINO

Hardware

B580 + 7 5700X3D + 48GB at 3133MHz

Models

Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf, Qwen3-14B-Q4_K_M.gguf, Qwen3.5-0.8B-Q4_K_M.gguf, Qwen3.5-9B-Q4_K_M.gguf

Problem description & steps to reproduce

The tg on llama.cpp OpenVINO is lower than ovms and OpenArc

First Bad Commit

No response

Relevant log output

llama.cpp

root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | OPENVINO   |  -1 |           pp512 |      2491.79 ± 43.30 |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | OPENVINO   |  -1 |           tg128 |         22.91 ± 0.39 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3-14B-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3 14B Q4_K - Medium        |   8.38 GiB |    14.77 B | OPENVINO   |  -1 |           pp512 |      2546.33 ± 46.69 |
| qwen3 14B Q4_K - Medium        |   8.38 GiB |    14.77 B | OPENVINO   |  -1 |           tg128 |         24.63 ± 0.47 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3.5-0.8B-Q4_K_M.gguf 
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 0.8B Q4_K - Medium      | 497.39 MiB |   752.39 M | OPENVINO   |  -1 |           pp512 |     8825.12 ± 138.33 |
| qwen35 0.8B Q4_K - Medium      | 497.39 MiB |   752.39 M | OPENVINO   |  -1 |           tg128 |         36.80 ± 0.22 |

build: f280b2698 (10615)
root@ae0463242527:/app# GGML_OPENVINO_DEVICE=GPU ./llama-bench --model /models/Qwen3.5-9B-Q4_K_M.gguf   
OpenVINO: using device GPU
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | OPENVINO   |  -1 |           pp512 |      2996.08 ± 17.29 |
| qwen35 9B Q4_K - Medium        |   5.28 GiB |     8.95 B | OPENVINO   |  -1 |           tg128 |         18.40 ± 0.67 |

build: f280b2698 (10615)

ovms

podman run --interactive --tty --publish=1235:8080 --rm --device=/dev/dri/renderD128 --user root docker.io/openvino/model_server:2026.3-gpu --rest_port 8080 --model_repository_path /models/ --source_model <model> --target_device GPU --task text_generation

podman run --interactive --tty --rm localhost/llama-benchy:latest --base-url http://host.docker.internal:1235/v3 --pp 512 --tg 128 --runs 5

| model                                       |   test |              t/s |     peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:--------------------------------------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov |  pp512 | 2486.48 ± 215.33 |              | 210.07 ± 18.32 | 207.50 ± 18.32 |  223.74 ± 18.92 |
| OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov |  tg128 |     47.03 ± 5.39 | 48.49 ± 7.51 |                |                |                 |
| model                      |   test |            t/s |     peak t/s |     ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:---------------------------|-------:|---------------:|-------------:|--------------:|---------------:|----------------:|
| OpenVINO/Qwen3-14B-int4-ov |  pp512 | 2690.01 ± 6.41 |              | 191.19 ± 0.45 |  190.33 ± 0.45 |   197.48 ± 4.81 |
| OpenVINO/Qwen3-14B-int4-ov |  tg128 |   41.63 ± 0.12 | 42.00 ± 0.00 |               |                |                 |
| model                         |   test |              t/s |      peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:------------------------------|-------:|-----------------:|--------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen3.5-0.8B-int4-ov |  pp512 | 3613.92 ± 487.60 |               | 144.97 ± 18.60 | 144.35 ± 18.60 |  144.97 ± 18.60 |
| OpenVINO/Qwen3.5-0.8B-int4-ov |  tg128 |    133.51 ± 1.79 | 134.56 ± 1.80 |                |                |                 |
| model                       |   test |              t/s |     peak t/s |      ttfr (ms) |   est_ppt (ms) |   e2e_ttft (ms) |
|:----------------------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:|
| OpenVINO/Qwen3.5-9B-int4-ov |  pp512 | 2098.39 ± 253.55 |              | 249.45 ± 27.33 | 247.76 ± 27.33 |  256.70 ± 20.08 |
| OpenVINO/Qwen3.5-9B-int4-ov |  tg128 |     61.06 ± 0.34 | 61.80 ± 0.40 |                |                |                 |

[haarika-madaka]

Need to replace one-by-one:

OpenVINO/Qwen2.5-Coder-14B-Instruct-int4-ov
OpenVINO/Qwen3-14B-int4-ov
OpenVINO/Qwen3.5-0.8B-int4-ov
OpenVINO/Qwen3.5-9B-int4-ov

First terminal:
podman run --interactive --tty --publish=1235:8080 --rm --device=/dev/dri --user root [docker.io/openvino/model_server:2026.3-gpu](https://docker.io/openvino/model_server:2026.3-gpu) --rest_port 8080 --model_repository_path /models/ --source_model <model> --target_device GPU --task text_generation

Second terminal:
llama-benchy --base-url http://localhost:1235/v3 --pp 512 --tg 128 --runs 5

We ran all the tests (OVMS and llama.cpp) on Lunar Lake machine. Even though the GPU device is different from the customer (B580), but the tg obtained from OVMS and llama.cpp is obvious.

llama.cpp

OVMS

Things to highlight:

1. Customer using llama-benchy (https://github.com/eugr/llama-benchy) for the OVMS test
2. OpenVINO IR is used in OVMS test while GGUF model is used in llama.cpp test but the models should be the same just different in format