diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f09a2913..c72d4dce0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ This cycle expands **model preparation and evaluation** across the CLI: precisio - **Quantization** — `--precision` selects FP16 conversion, RTN INT4, static QDQ, or calibration-free dynamic INT8; `winml quantize` can compose multiple precision passes such as INT4 followed by FP16 (#872, #985, #1047). - `winml build` — `--export-type optimized` produces a complete Qwen3 onnxruntime-genai NPU/QNN bundle, including prefill/decode, embeddings, LM head, tokenizer, and manifest files (#836, #996, #1008, #1081, #1104). -- `winml perf --runtime winml-genai` — benchmarks prebuilt or automatically cached GenAI bundles with TTFT, token throughput, prompt-template controls, EP overrides, and isolated pre-compilation (#1015, #1042, #1046, #1054, #1109). +- `winml perf --runtime ort-genai` — benchmarks prebuilt or automatically cached GenAI bundles with TTFT, token throughput, prompt-template controls, EP overrides, and isolated pre-compilation (#1015, #1042, #1046, #1054, #1109). - **Composite models** — `export` and `build` automatically fan out pipeline components; `export` / `build` / `perf` support `--submodel`, and explicit composite tasks such as summarization and translation are accepted (#1031, #1037, #1058, #1071, #1089). - **Export controls** — dynamic axes and symbolic input dimensions are supported while static TorchScript export remains the default; `build`, `config`, `perf`, and `eval` expose matching shape/input/export overrides (#1074, #1083, #1106, #1141, #1156, #1188). - `winml perf` — real `.npz` inputs, time-budgeted `--duration` runs, cached per-module builds, actual dynamic dimensions, and QNN profiler ONNX metrics (#1004, #1055, #1066, #1102, #1168). diff --git a/docs/commands/eval.md b/docs/commands/eval.md index d47d32523..accda82df 100644 --- a/docs/commands/eval.md +++ b/docs/commands/eval.md @@ -26,7 +26,7 @@ $ winml eval [options] | `--input-specs` | | `PATH` | — | JSON input tensor specs to merge into the Hugging Face export config. Symbolic string dimensions infer dynamic axes. **Ignored for pre-built `.onnx` inputs**. | | `--export-config` | | `PATH` | — | JSON ONNX export config overrides (opset version, constant folding, etc.) to merge into the Hugging Face export config. **Ignored for pre-built `.onnx` inputs**. | | `--dynamic-axes` | | `PATH` | — | JSON dynamic axes mapping for Hugging Face ONNX export, for example `{"input_ids": {"0": "batch", "1": "sequence"}}`. **Ignored for pre-built `.onnx` inputs**. | -| `--runtime` | | `winml\|pytorch` | `winml` | Evaluation runtime. `winml` exports Hugging Face checkpoints to ONNX; `pytorch` evaluates the original checkpoint and supports `auto`, `cpu`, or CUDA-backed `gpu` devices. | +| `--runtime` | | `winml-ort\|pytorch` | `winml-ort` | Evaluation runtime. `winml-ort` exports Hugging Face checkpoints to ONNX; `pytorch` evaluates the original checkpoint and supports `auto`, `cpu`, or CUDA-backed `gpu` devices. | | `--dataset` | | `TEXT` | task default | HuggingFace dataset path (e.g., `imagenet-1k`, `nyu-mll/glue`). If omitted, a default dataset is selected based on the task. | | `--dataset-name` | | `TEXT` | — | Dataset configuration name for multi-config datasets. | | `--dataset-revision` | | `TEXT` | — | Git revision (branch, tag, or commit) of the dataset to load. Use `refs/convert/parquet` for HF datasets that are only served via the parquet mirror. | @@ -48,7 +48,7 @@ $ winml eval [options] ## How it works -`winml eval` loads the model and runs the evaluation pipeline via the internal `evaluate` function, then pulls the requested number of samples from a HuggingFace dataset. By default, Hugging Face model IDs and local checkpoints use the `winml` runtime: they are exported to ONNX and evaluated through WinML. With `--runtime pytorch`, the task-resolved PyTorch model and stored dtype are preserved and the same dataset preprocessing, evaluator, and metrics run directly against that model. PyTorch `auto` selects CUDA when available and otherwise CPU; `gpu` requires CUDA. The JSON report identifies the effective runtime as `winml` or `pytorch`. +`winml eval` loads the model and runs the evaluation pipeline via the internal `evaluate` function, then pulls the requested number of samples from a HuggingFace dataset. By default, Hugging Face model IDs and local checkpoints use the `winml-ort` runtime: they are exported to ONNX and evaluated through WinML. With `--runtime pytorch`, the task-resolved PyTorch model and stored dtype are preserved and the same dataset preprocessing, evaluator, and metrics run directly against that model. PyTorch `auto` selects CUDA when available and otherwise CPU; `gpu` requires CUDA. The JSON report identifies the effective runtime as `winml-ort` or `pytorch`. Python callers can pass an existing model directly with `evaluate(config, pytorch_model=model)`. An explicit `config.model_id` selects the tokenizer or processor; otherwise evaluation infers it from `model.config._name_or_path` and reports an error if neither source is available. diff --git a/docs/commands/perf.md b/docs/commands/perf.md index 840d8b6b0..bc67b4c7b 100644 --- a/docs/commands/perf.md +++ b/docs/commands/perf.md @@ -16,8 +16,8 @@ $ winml perf [options] | Flag | Short | Type | Default | Description | |---|---|---|---|---| -| `--model` | `-m` | `TEXT` | — | HuggingFace model ID or path to a local `.onnx` file. Required. With `--runtime winml-genai`, also accepts a prebuilt genai **bundle directory**, or a HuggingFace model ID that is auto-built into a bundle on demand. | -| `--runtime` | | `winml\|winml-genai` | `winml` | Inference runtime. `winml` benchmarks single-shot ONNX inference; `winml-genai` benchmarks an onnxruntime-genai bundle (LLM generation: time-to-first-token + decode tokens/sec). With `winml-genai`, a model ID that is not a bundle directory is auto-built into one (cached under `~/.cache/winml/`, targeting the NPU HTP via QNN) before benchmarking. GenAI cache controls are tracked in issue #1275. | +| `--model` | `-m` | `TEXT` | — | HuggingFace model ID or path to a local `.onnx` file. Required. With `--runtime ort-genai`, also accepts a prebuilt genai **bundle directory**, or a HuggingFace model ID that is auto-built into a bundle on demand. | +| `--runtime` | | `winml-ort\|ort-genai` | `winml-ort` | Inference runtime. `winml-ort` benchmarks single-shot ONNX inference; `ort-genai` benchmarks an onnxruntime-genai bundle (LLM generation: time-to-first-token + decode tokens/sec). With `ort-genai`, a model ID that is not a bundle directory is auto-built into one (cached under `~/.cache/winml/`, targeting the NPU HTP via QNN) before benchmarking. GenAI cache controls are tracked in issue #1275. | | `--task` | | `TEXT` | auto-detected | Explicit task override (e.g., `image-classification`). Inferred from the model if omitted. | | `--iterations` | | `INTEGER` | `100` | Number of timed inference iterations used to compute statistics. | | `--warmup` | | `INTEGER` | `10` | Number of warm-up iterations run before timing begins; excluded from statistics. | @@ -27,7 +27,7 @@ $ winml perf [options] | `--ep-options` | | `KEY=VALUE` (multiple) | — | Runtime EP provider option forwarded to the inference session (e.g., `--ep-options htp_performance_mode=burst`). Repeatable. Applies to both HuggingFace model IDs and ONNX file inputs. When detail op-tracing automatically compiles a raw ONNX model, these options are also applied to that compilation. | | `--output` | `-o` | `PATH` | `~/.cache/winml/perf//.json` | Output JSON file path for the benchmark report. | | `--batch-size` | | `INTEGER` | `1` | Batch size used when generating synthetic input tensors. Ignored when `--input-data` is set. | -| `--input-data` | | `PATH` | — | Path to a `.npz` file of real input tensors to benchmark with instead of randomly generated inputs. The archive's keys must match the model's inputs exactly; dtypes are cast to the model's expected dtype (with a warning) to mirror normal inference. Not supported with `--module`, `--runtime winml-genai`, or composite (dual-encoder) models. | +| `--input-data` | | `PATH` | — | Path to a `.npz` file of real input tensors to benchmark with instead of randomly generated inputs. The archive's keys must match the model's inputs exactly; dtypes are cast to the model's expected dtype (with a warning) to mirror normal inference. Not supported with `--module`, `--runtime ort-genai`, or composite (dual-encoder) models. | | `--shape-config` | | `PATH` | — | Path to a JSON file containing shape overrides (e.g., `{"height": 480, "width": 480}`). Used for Hugging Face export and random input generation; ignored in `--module` mode and when `--input-data` is set. | | `--input-specs` | | `PATH` | — | JSON input tensor specs to merge into the Hugging Face export config before benchmarking. Symbolic string dimensions infer dynamic axes. Ignored for pre-exported ONNX files and in `--module` mode. | | `--export-config` | | `PATH` | — | JSON ONNX export config overrides to apply when `perf` builds a Hugging Face model before benchmarking. Ignored for pre-exported ONNX files and in `--module` mode. | @@ -36,23 +36,23 @@ $ winml perf [options] | `--use-cache/--no-use-cache` | | flag | `true` | Reuse persistent model build artifacts. `--no-use-cache` performs a fresh build in a temporary folder and discards it after benchmarking. | | `--rebuild/--no-rebuild` | | flag | `false` | Force model rebuild even if a cached artifact already exists. | | `--module` | | `TEXT` | — | PyTorch module class name for per-module benchmarking (e.g., `BertAttention`). Builds and times each matching instance separately. See [Load and export](../concepts/load-and-export.md). | -| `--monitor/--no-monitor` | | flag | `false` | Show a live NPU/CPU utilization chart while the benchmark runs and include hardware metrics in the JSON report. With `--runtime winml-genai`, the monitor wraps the genai load + generation benchmark. | +| `--monitor/--no-monitor` | | flag | `false` | Show a live NPU/CPU utilization chart while the benchmark runs and include hardware metrics in the JSON report. With `--runtime ort-genai`, the monitor wraps the genai load + generation benchmark. | | `--op-tracing` | | `basic\|detail` | — | Enable operator-level profiling. QNN detail tracing requires an EPContext model; a raw ONNX input is detected and compiled automatically with the required profiling options. | -| `--compile` / `--no-compile` | | flag | `false` | Compile the model to EPContext binaries during build. QNN detail op-tracing enables this automatically for a raw ONNX input unless `--no-compile` or `--skip-build` was explicitly specified. For `--runtime winml-genai` on the NPU, `--compile` pre-compiles each QNN stage (in an isolated subprocess) before generation. | -| `--compile-timeout` | | `INTEGER` | `300` | *(winml-genai)* Max seconds to compile each EPContext stage before falling back to the original ONNX. Requires `--compile`. | -| `--prompt` | | `TEXT` | `Explain the theory of relativity in simple terms.` | *(winml-genai)* Prompt text to generate from. Wrapped in the bundle's chat template unless `--no-apply-template`. | -| `--apply-template/--no-apply-template` | | flag | `true` | *(winml-genai)* Wrap `--prompt` in the bundle's chat template before timing. | -| `--max-new-tokens` | | `INTEGER` | `128` | *(winml-genai)* Number of new tokens to generate per iteration. | +| `--compile` / `--no-compile` | | flag | `false` | Compile the model to EPContext binaries during build. QNN detail op-tracing enables this automatically for a raw ONNX input unless `--no-compile` or `--skip-build` was explicitly specified. For `--runtime ort-genai` on the NPU, `--compile` pre-compiles each QNN stage (in an isolated subprocess) before generation. | +| `--compile-timeout` | | `INTEGER` | `300` | *(ort-genai)* Max seconds to compile each EPContext stage before falling back to the original ONNX. Requires `--compile`. | +| `--prompt` | | `TEXT` | `Explain the theory of relativity in simple terms.` | *(ort-genai)* Prompt text to generate from. Wrapped in the bundle's chat template unless `--no-apply-template`. | +| `--apply-template/--no-apply-template` | | flag | `true` | *(ort-genai)* Wrap `--prompt` in the bundle's chat template before timing. | +| `--max-new-tokens` | | `INTEGER` | `128` | *(ort-genai)* Number of new tokens to generate per iteration. | ## How it works `winml perf` loads the model through `WinMLAutoModel` — accepting both HuggingFace IDs and local ONNX files — then generates random input tensors from the model's I/O configuration. It runs the specified number of warm-up iterations (excluded from statistics) followed by the timed iterations, collecting per-sample latency. The final report includes mean, min, max, P50, P90, P95, P99, standard deviation, and throughput in samples per second. When `--monitor` is active, a hardware polling loop runs in parallel and records NPU / GPU utilization, CPU usage, and device memory alongside the timing data. -Both runtime reports include `schema_version: 2` and a `benchmark_info.runtime` discriminator (`winml` or `winml-genai`). Shared metadata such as `model_id`, `running_model_path`, `device`, `ep`, `iterations`, `warmup`, and `timestamp` uses the same field names where the concepts overlap; GenAI also keeps `bundle_dir` because the runnable artifact is a bundle directory. +Both runtime reports include `schema_version: 2` and a `benchmark_info.runtime` discriminator (`winml-ort` or `ort-genai`). Shared metadata such as `model_id`, `running_model_path`, `device`, `ep`, `iterations`, `warmup`, and `timestamp` uses the same field names where the concepts overlap; GenAI also keeps `bundle_dir` because the runnable artifact is a bundle directory. -When `--memory` is enabled, both `winml` and `winml-genai` reports use the same `memory` field names for shared concepts: RSS baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta; VRAM local/shared baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta. +When `--memory` is enabled, both `winml-ort` and `ort-genai` reports use the same `memory` field names for shared concepts: RSS baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta; VRAM local/shared baseline, after-compile/load, after-inference, peak, model-load delta, inference/generation delta, and total delta. -With `--runtime winml-genai`, `winml perf` benchmarks the onnxruntime-genai decoder pipeline rather than a single `session.run()`. The JSON report uses a phase-based schema: `load` contains startup spans, `requests` contains one warmup or timed generation sample per request, `aggregate` summarizes timed requests only, `memory` contains optional RAM/VRAM deltas, and `hw_monitor` contains optional monitor output. The optional `memory` and `hw_monitor` top-level names match the classic `winml` perf report; GenAI keeps `load`/`requests`/`aggregate` instead of classic `latency_ms`/`throughput` because generation has distinct prompt, first-token, and decode phases. +With `--runtime ort-genai`, `winml perf` benchmarks the onnxruntime-genai decoder pipeline rather than a single `session.run()`. The JSON report uses a phase-based schema: `load` contains startup spans, `requests` contains one warmup or timed generation sample per request, `aggregate` summarizes timed requests only, `memory` contains optional RAM/VRAM deltas, and `hw_monitor` contains optional monitor output. The optional `memory` and `hw_monitor` top-level names match the classic `winml-ort` perf report; GenAI keeps `load`/`requests`/`aggregate` instead of classic `latency_ms`/`throughput` because generation has distinct prompt, first-token, and decode phases. ### GenAI metric definitions @@ -157,7 +157,7 @@ and logs a warning. - **Warm-up too low on NPU.** The first several inferences on an NPU EP can be significantly slower due to kernel compilation and caching. The default of 10 warm-up iterations is usually enough for vision models, but transformer models with many operators may need `--warmup 30` or higher to reach steady-state latency. - **Hidden third-party diagnostics.** Normal `winml perf` output suppresses noisy native warning-level diagnostics and Hugging Face download/progress chatter so benchmark results stay readable. Use `-v`/`-vv` or set `WINMLCLI_SHOW_ALL_WARNINGS=1` to show those warnings when debugging provider or Hub issues. -- **`--input-data` keys must match; dtypes are cast.** The `.npz` keys must equal the model's input names — a missing or unexpected key is a hard error (typo protection). Array dtypes are cast to the model's expected dtype with a warning (matching normal inference), so you don't have to hand-match widths. `.npy` files are not supported — save named arrays as `.npz`. When `--input-data` is set, `--batch-size` and `--shape-config` are ignored (the tensors define their own shapes). It is also rejected for `--module` mode, `--runtime winml-genai`, and composite (dual-encoder) models such as CLIP/SigLIP, where each sub-model has its own inputs that a single `.npz` cannot address. +- **`--input-data` keys must match; dtypes are cast.** The `.npz` keys must equal the model's input names — a missing or unexpected key is a hard error (typo protection). Array dtypes are cast to the model's expected dtype with a warning (matching normal inference), so you don't have to hand-match widths. `.npy` files are not supported — save named arrays as `.npz`. When `--input-data` is set, `--batch-size` and `--shape-config` are ignored (the tensors define their own shapes). It is also rejected for `--module` mode, `--runtime ort-genai`, and composite (dual-encoder) models such as CLIP/SigLIP, where each sub-model has its own inputs that a single `.npz` cannot address. - **Real data only binds if the export kept axes dynamic.** When `-m` is a HuggingFace model ID, `perf` exports it with default shapes (because `--shape-config`/`--batch-size` are ignored under `--input-data`). If that export baked in static shapes, ORT will reject differently-shaped `--input-data`. Use `--dynamic-axes`/symbolic `--input-specs` for the Hugging Face build, or point `-m` at an ONNX file that already has dynamic axes. - **`--shape-config` is ignored when real or module inputs own the shape.** It is ignored in `--module` mode and when `--input-data` is set. The command prints a warning in both situations. - **Random inputs do not represent real data distributions.** Latency numbers are accurate, but memory access patterns may differ from production because the generated tensors are uniform random values. For memory-bandwidth-sensitive models this can understate real-world latency. diff --git a/docs/samples/qwen3-genai-bundle.md b/docs/samples/qwen3-genai-bundle.md index f4e56f19a..d873e030c 100644 --- a/docs/samples/qwen3-genai-bundle.md +++ b/docs/samples/qwen3-genai-bundle.md @@ -118,11 +118,11 @@ and token generation on the NPU with `winml perf`: ```bash # Qualcomm Snapdragon NPU -winml perf -m out/qwen3-bundle --runtime winml-genai --device npu --compile \ +winml perf -m out/qwen3-bundle --runtime ort-genai --device npu --compile \ --compile-timeout 600 --max-new-tokens 20 --prompt "What is the capital of France?" # AMD Ryzen AI NPU -winml perf -m out/qwen3-bundle --runtime winml-genai --device npu --ep vitisai --compile \ +winml perf -m out/qwen3-bundle --runtime ort-genai --device npu --ep vitisai --compile \ --compile-timeout 600 --max-new-tokens 20 --prompt "What is the capital of France?" ``` @@ -136,13 +136,13 @@ request latency, optional RAM/VRAM deltas, and a results JSON under onnxruntime-genai does not expose it; the estimate is labeled in JSON. !!! tip "One command from a model id (auto-build)" - `winml perf --runtime winml-genai` also accepts a HuggingFace **model id** directly. + `winml perf --runtime ort-genai` also accepts a HuggingFace **model id** directly. When `-m` is not a prebuilt bundle directory, it builds the genai bundle on demand (into `~/.cache/winml/`, reused on later runs) and then benchmarks it — no separate `winml build` step: ```bash - winml perf -m Qwen/Qwen3-0.6B --runtime winml-genai --compile \ + winml perf -m Qwen/Qwen3-0.6B --runtime ort-genai --compile \ --compile-timeout 600 --max-new-tokens 20 --prompt "What is the capital of France?" ``` diff --git a/scripts/e2e_eval/README.md b/scripts/e2e_eval/README.md index bcef345f6..a2433a983 100644 --- a/scripts/e2e_eval/README.md +++ b/scripts/e2e_eval/README.md @@ -131,7 +131,7 @@ uv run python scripts/e2e_eval/run_eval.py --update-baseline --eval-type accurac ### `run_llm_eval.py` — Run GenAI Context Sweep Runs an existing ONNX Runtime GenAI bundle through `winml perf --runtime -winml-genai` at one or more prompt lengths. The runner enables the EPContext +ort-genai` at one or more prompt lengths. The runner enables the EPContext pre-compilation required by accelerator-backed GenAI stages and generation-window hardware monitoring, preserves each raw perf report as `perf_ctx.json`, and writes a schema-validated `llm_eval_result.json` containing TTFT, decode and prefill diff --git a/scripts/e2e_eval/run_llm_eval.py b/scripts/e2e_eval/run_llm_eval.py index 49eb0876a..c69435fe3 100644 --- a/scripts/e2e_eval/run_llm_eval.py +++ b/scripts/e2e_eval/run_llm_eval.py @@ -6,7 +6,7 @@ """Run a schema-normalized ONNX Runtime GenAI context sweep. The runner consumes an existing GenAI bundle and invokes ``winml perf ---runtime winml-genai`` for each requested prompt length. Generation timing +--runtime ort-genai`` for each requested prompt length. Generation timing and generation-window resource metrics are normalized into one ``llm_eval_result.json`` document. """ @@ -361,7 +361,7 @@ def _perf_args( "-m", str(bundle_dir), "--runtime", - "winml-genai", + "ort-genai", "--device", device, "--prompt-file", @@ -436,7 +436,7 @@ def _context_point( f"got {reported_bundle_dir!r}" ) expected_info = { - "runtime": "winml-genai", + "runtime": "ort-genai", "device": expected_device, "effective_device": expected_device, "compile": True, diff --git a/src/winml/modelkit/commands/_perf_genai.py b/src/winml/modelkit/commands/_perf_genai.py index 8f720d79d..c1e17b4b9 100644 --- a/src/winml/modelkit/commands/_perf_genai.py +++ b/src/winml/modelkit/commands/_perf_genai.py @@ -2,7 +2,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. # -------------------------------------------------------------------------- -"""GenAI generation benchmarking for ``winml perf --runtime winml-genai``. +"""GenAI generation benchmarking for ``winml perf --runtime ort-genai``. Benchmarks a prebuilt ``onnxruntime-genai`` bundle folder through :class:`GenaiSession`. Unlike the single-shot WinML path (which times each @@ -66,7 +66,7 @@ # Constants # ============================================================================= -RUNTIME_TYPE: RuntimeName = "winml-genai" +RUNTIME_TYPE: RuntimeName = "ort-genai" _HW_POLL_INTERVAL_MS = 200 # Built-in benchmark prompt. Mirrored by the ``--prompt`` CLI default and the @@ -74,7 +74,7 @@ _DEFAULT_PROMPT = "Explain the theory of relativity in simple terms." # Sentinel ``--device`` value meaning "respect the bundle's genai_config.json -# routing" (no EP override). It is the winml-genai default: a genai bundle is +# routing" (no EP override). It is the ort-genai default: a genai bundle is # mixed by design (e.g. ctx/iter on the NPU, embeddings/lm_head on CPU) and its # config already encodes that per-stage routing, so the common case leaves it # untouched. A concrete ``--device`` (or ``--ep``) is an explicit override that diff --git a/src/winml/modelkit/commands/eval.py b/src/winml/modelkit/commands/eval.py index ee13bbaed..adae4657d 100644 --- a/src/winml/modelkit/commands/eval.py +++ b/src/winml/modelkit/commands/eval.py @@ -118,10 +118,10 @@ ) @click.option( "--runtime", - type=click.Choice(["winml", "pytorch"]), - default="winml", + type=click.Choice(["winml-ort", "pytorch"]), + default="winml-ort", show_default=True, - help="Evaluation runtime. 'winml' exports Hugging Face checkpoints to ONNX; " + help="Evaluation runtime. 'winml-ort' exports Hugging Face checkpoints to ONNX; " "'pytorch' evaluates the original checkpoint.", ) @click.option( @@ -332,9 +332,9 @@ def eval( # ── 1. Build config: defaults ← config file ← CLI ── cfg, config_fields = _build_eval_config(ctx, config_file, column, label_mapping_path) - if cfg.runtime not in ("winml", "pytorch"): + if cfg.runtime not in ("winml-ort", "pytorch"): raise click.UsageError( - f"Invalid eval runtime {cfg.runtime!r}; expected 'winml' or 'pytorch'." + f"Invalid eval runtime {cfg.runtime!r}; expected 'winml-ort' or 'pytorch'." ) if cfg.runtime == "pytorch": _validate_pytorch_runtime_options(ctx, cfg, config_fields) @@ -782,7 +782,7 @@ def _resolve_genai_ep(ctx: click.Context, cfg: WinMLEvaluationConfig) -> None: an EP override is present and otherwise leaves the bundle untouched. Passing the device straight through would make ``--device`` a no-op: the whole point of an explicit device is to force the pipeline onto it. Mirroring the - ``winml-genai`` perf precedence, an explicitly supplied ``--device`` is + ``ort-genai`` perf precedence, an explicitly supplied ``--device`` is resolved to a concrete EP (via the same device→EP path the ONNX runtime uses), while the default (``auto``) respects the bundle's own routing. diff --git a/src/winml/modelkit/commands/perf.py b/src/winml/modelkit/commands/perf.py index 64151fb81..2012ca4a5 100644 --- a/src/winml/modelkit/commands/perf.py +++ b/src/winml/modelkit/commands/perf.py @@ -72,7 +72,7 @@ # Hardware monitor polling interval (milliseconds) _HW_POLL_INTERVAL_MS = 200 -_RUNTIME_TYPE: RuntimeName = "winml" +_RUNTIME_TYPE: RuntimeName = "winml-ort" def _resolve_runtime(runtime: RuntimeName, model: str) -> RuntimeName: @@ -82,8 +82,8 @@ def _resolve_runtime(runtime: RuntimeName, model: str) -> RuntimeName: model_path = Path(model) if model_path.is_dir() and (model_path / "genai_config.json").is_file(): - return "winml-genai" - return "winml" + return "ort-genai" + return "winml-ort" def _detail_fallback_guidance(reason: TraceFallbackReason | None) -> str: @@ -2302,7 +2302,7 @@ def _run_simple_loop( # perf() param names for WinML-only options that a prebuilt genai bundle # ignores. Mapped to the user-facing flag for the warning message. -# NB: ``--ep`` is intentionally absent — it is honored for winml-genai as an EP +# NB: ``--ep`` is intentionally absent — it is honored for ort-genai as an EP # override (explicit --ep > concrete --device > respect config). _GENAI_IGNORED_FLAGS: dict[str, str] = { "task": "--task", @@ -2364,7 +2364,7 @@ def _warn_ignored_genai_flags( if ignored: console.print( "[yellow]Warning:[/yellow] the following options are ignored with " - f"--runtime winml-genai: {', '.join(sorted(ignored))}" + f"--runtime ort-genai: {', '.join(sorted(ignored))}" ) @@ -2423,7 +2423,7 @@ def _autobuild_genai_bundle( recipe = resolve_genai_bundle(model_type) if recipe is None: raise click.UsageError( - f"--runtime winml-genai cannot auto-build '{model}': no genai bundle recipe " + f"--runtime ort-genai cannot auto-build '{model}': no genai bundle recipe " f"is registered for model type '{model_type or 'unknown'}'. Pass a prebuilt " "genai bundle *directory* (e.g. from " f"'winml build -m {model} -o --device npu --ep qnn')." @@ -2452,7 +2452,7 @@ def _autobuild_genai_bundle( def _run_genai_runtime( ctx: click.Context, *, model: str, console: Console, json_mode: bool ) -> None: - """Validate folder input and dispatch to the winml-genai benchmark path. + """Validate folder input and dispatch to the ort-genai benchmark path. The genai imports are function-local so ``winml perf --help`` does not pay their import cost (see tests/cli/test_import_time.py). @@ -2469,14 +2469,14 @@ def _run_genai_runtime( p = ctx.params # --module walks a live nn.Module graph; meaningless for a prebuilt bundle. if p.get("module_class"): - raise click.UsageError("--module is not supported with --runtime winml-genai.") + raise click.UsageError("--module is not supported with --runtime ort-genai.") # --submodel narrows a composite into a single sub-component benchmarked as a # standalone session; a genai bundle is already the full composite generation # pipeline, so selecting one sub-component is meaningless. Reject rather than # silently ignore (this return runs before the winml-path --submodel handling). if p.get("submodel"): - raise click.UsageError("--submodel is not supported with --runtime winml-genai.") + raise click.UsageError("--submodel is not supported with --runtime ort-genai.") # Keep any bundle-lifetime resources alive across the benchmark. with contextlib.ExitStack() as stack: @@ -2495,7 +2495,7 @@ def _run_genai_runtime( ) elif bundle_dir.suffix.lower() == ".onnx": raise click.UsageError( - f"--runtime winml-genai requires a genai bundle *directory*, got '{model}'." + f"--runtime ort-genai requires a genai bundle *directory*, got '{model}'." ) else: bundle_dir, built_fresh = _autobuild_genai_bundle( @@ -2619,9 +2619,9 @@ def _validate_duration( type=click.Choice(list(RUNTIME_NAMES)), default="auto", show_default=True, - help="'auto' selects winml-genai for folders containing genai_config.json, " - "otherwise winml. 'winml' benchmarks single-shot ONNX inference; " - "'winml-genai' benchmarks an onnxruntime-genai bundle folder " + help="'auto' selects ort-genai for folders containing genai_config.json, " + "otherwise winml-ort. 'winml-ort' benchmarks single-shot ONNX inference; " + "'ort-genai' benchmarks an onnxruntime-genai bundle folder " "(LLM generation: TTFT + decode tokens/sec).", ) @click.option( @@ -2629,21 +2629,21 @@ def _validate_duration( type=str, default="Explain the theory of relativity in simple terms.", show_default=True, - help="[winml-genai] Prompt text to generate from. By default it is wrapped in " + help="[ort-genai] Prompt text to generate from. By default it is wrapped in " "the bundle's chat template; pass --no-apply-template to benchmark it verbatim.", ) @click.option( "--prompt-file", type=click.Path(exists=True, dir_okay=False, path_type=Path), default=None, - help="[winml-genai] Read the prompt from a UTF-8 text file. Mutually exclusive " + help="[ort-genai] Read the prompt from a UTF-8 text file. Mutually exclusive " "with an explicit --prompt; avoids command-line length limits for long contexts.", ) @click.option( "--apply-template/--no-apply-template", default=True, show_default=True, - help="[winml-genai] Wrap --prompt in the bundle's chat template before timing. " + help="[ort-genai] Wrap --prompt in the bundle's chat template before timing. " "Use --no-apply-template to benchmark a prompt that is already formatted.", ) @click.option( @@ -2651,14 +2651,14 @@ def _validate_duration( type=click.IntRange(min=1), default=128, show_default=True, - help="[winml-genai] Number of new tokens to generate per iteration.", + help="[ort-genai] Number of new tokens to generate per iteration.", ) @click.option( "--compile-timeout", type=int, default=300, show_default=True, - help="[winml-genai] Max seconds to compile each EPContext stage before falling back " + help="[ort-genai] Max seconds to compile each EPContext stage before falling back " "to the original ONNX (requires --compile).", ) @click.option( @@ -2711,7 +2711,7 @@ def _validate_duration( default="auto", include_auto=True, include_config=True, - optional_message="'config' (winml-genai only) respects the bundle's genai_config.json routing.", + optional_message="'config' (ort-genai only) respects the bundle's genai_config.json routing.", ) @cli_utils.precision_option() @click.option( @@ -2746,7 +2746,7 @@ def _validate_duration( default=None, help="Path to a .npz file of real input tensors to benchmark with instead " "of randomly generated inputs. Keys must match the model's input names and " - "dtypes exactly. Not supported with --module or --runtime winml-genai.", + "dtypes exactly. Not supported with --module or --runtime ort-genai.", ) @cli_utils.shape_config_option(param_name="shape_config_path") @cli_utils.input_specs_option() @@ -2982,10 +2982,10 @@ def perf( # ========================================================================= # GENAI RUNTIME: benchmark an onnxruntime-genai bundle folder # ========================================================================= - if runtime == "winml-genai": + if runtime == "ort-genai": if input_data is not None: raise click.UsageError( - "--input-data is not supported with --runtime winml-genai; " + "--input-data is not supported with --runtime ort-genai; " "genai benchmarking is driven by --prompt." ) _run_genai_runtime(ctx, model=model, console=console, json_mode=json_mode) @@ -2993,7 +2993,7 @@ def perf( # --duration replaces the fixed iteration count with a wall-clock budget. # Op-tracing runs its own fixed, small iteration count, so the two are - # mutually exclusive. This is a WinML-path constraint only: for winml-genai + # mutually exclusive. This is a WinML-path constraint only: for ort-genai # both flags are ignored (see _GENAI_IGNORED_FLAGS), so the check lives # after the genai early return to keep those options consistently non-fatal. if duration is not None and op_tracing: @@ -3002,13 +3002,13 @@ def perf( "(op-tracing runs a fixed, small iteration count)." ) - # ``--device config`` is a winml-genai-only sentinel (respect the bundle's + # ``--device config`` is a ort-genai-only sentinel (respect the bundle's # genai_config.json routing). It is meaningless for the single-shot WinML # path, so reject it explicitly rather than letting resolve_device raise a # generic "unknown device" error. if device.lower() == "config": raise click.UsageError( - "--device config is only valid with --runtime winml-genai " + "--device config is only valid with --runtime ort-genai " "(it means 'respect the bundle's genai_config.json routing')." ) diff --git a/src/winml/modelkit/eval/config.py b/src/winml/modelkit/eval/config.py index 2fc1562e6..4c6b62e6d 100644 --- a/src/winml/modelkit/eval/config.py +++ b/src/winml/modelkit/eval/config.py @@ -15,7 +15,7 @@ from ..utils.eval_utils import EvalMode -EvalRuntime = Literal["winml", "pytorch"] +EvalRuntime = Literal["winml-ort", "pytorch"] @dataclass @@ -137,7 +137,7 @@ class WinMLEvaluationConfig: output_path: Path to write JSON results. runtime: Evaluation runtime. - - ``"winml"`` (default): export Hugging Face checkpoints to ONNX + - ``"winml-ort"`` (default): export Hugging Face checkpoints to ONNX and evaluate with WinML. - ``"pytorch"``: evaluate the original Hugging Face checkpoint. mode: Evaluation mode (see :data:`EvalMode`). @@ -186,7 +186,7 @@ class WinMLEvaluationConfig: skip_build: bool = True use_cache: bool = True rebuild: bool = False - runtime: EvalRuntime = "winml" + runtime: EvalRuntime = "winml-ort" trust_remote_code: bool = False _auto_device_selected: bool = field(default=False, repr=False, compare=False, kw_only=True) _pipeline_device_override: str | None = field( @@ -247,7 +247,7 @@ def to_dict(self) -> dict: result["output_path"] = str(self.output_path) if self.mode != "onnx": result["mode"] = self.mode - if self.runtime == "winml": + if self.runtime == "winml-ort": result["skip_build"] = self.skip_build result["use_cache"] = self.use_cache result["rebuild"] = self.rebuild @@ -296,6 +296,6 @@ def from_dict(cls, data: dict) -> WinMLEvaluationConfig: skip_build=data.get("skip_build", True), use_cache=data.get("use_cache", True), rebuild=data.get("rebuild", False), - runtime=data.get("runtime", "winml"), + runtime=data.get("runtime", "winml-ort"), trust_remote_code=data.get("trust_remote_code", False), ) diff --git a/src/winml/modelkit/eval/evaluate.py b/src/winml/modelkit/eval/evaluate.py index e02ac79a0..42a9ab8a5 100644 --- a/src/winml/modelkit/eval/evaluate.py +++ b/src/winml/modelkit/eval/evaluate.py @@ -120,7 +120,7 @@ def get_evaluator_class(config: WinMLEvaluationConfig) -> type[WinMLEvaluator]: def _validate_pytorch_runtime_config(config: WinMLEvaluationConfig) -> None: """Validate state that cannot apply to the PyTorch runtime.""" - if config.runtime == "winml": + if config.runtime == "winml-ort": return incompatible: list[str] = [] @@ -649,8 +649,8 @@ def evaluate( """ from ..utils.eval_utils import EVAL_MODES - if config.runtime not in ("winml", "pytorch"): - raise ValueError(f"Invalid runtime {config.runtime!r}; expected 'winml' or 'pytorch'.") + if config.runtime not in ("winml-ort", "pytorch"): + raise ValueError(f"Invalid runtime {config.runtime!r}; expected 'winml-ort' or 'pytorch'.") if pytorch_model is not None: config = _prepare_supplied_pytorch_model(config, pytorch_model) mode = config.mode if config.mode is not None else "onnx" @@ -759,7 +759,7 @@ def print_config(config: WinMLEvaluationConfig) -> None: output_console.print(f"[bold blue]Device:[/bold blue] {config.device}") if config.ep is not None: output_console.print(f"[bold blue]EP:[/bold blue] {config.ep}") - if config.runtime == "winml": + if config.runtime == "winml-ort": output_console.print(f"[bold blue]Precision:[/bold blue] {config.precision}") if config.mode != "compare": output_console.print(f"[bold blue]Dataset:[/bold blue] {ds.path}") diff --git a/src/winml/modelkit/eval/tensor_similarity_evaluator.py b/src/winml/modelkit/eval/tensor_similarity_evaluator.py index f6d47e773..7660d3dd9 100644 --- a/src/winml/modelkit/eval/tensor_similarity_evaluator.py +++ b/src/winml/modelkit/eval/tensor_similarity_evaluator.py @@ -44,7 +44,7 @@ def _make_reference_config(config: WinMLEvaluationConfig) -> WinMLEvaluationConf model_id=None, model_path=config.reference_path, reference_path=None, - runtime="winml", + runtime="winml-ort", device=config.reference_device, ep=config.reference_ep, precision="auto", diff --git a/src/winml/modelkit/models/winml/genai_causal_lm.py b/src/winml/modelkit/models/winml/genai_causal_lm.py index 53deeeb16..101d468c3 100644 --- a/src/winml/modelkit/models/winml/genai_causal_lm.py +++ b/src/winml/modelkit/models/winml/genai_causal_lm.py @@ -31,7 +31,7 @@ class WinMLGenaiCausalLM: Constructed with an already-resolved ``ep`` / ``device`` that pass straight to the session, so inference runs on the same runtime and EP as - ``winml perf --runtime winml-genai``. + ``winml perf --runtime ort-genai``. Args: bundle_dir: Path to the genai bundle directory. diff --git a/src/winml/modelkit/utils/cli.py b/src/winml/modelkit/utils/cli.py index f1ebdfea4..550623b93 100644 --- a/src/winml/modelkit/utils/cli.py +++ b/src/winml/modelkit/utils/cli.py @@ -556,7 +556,7 @@ def device_option( include_all: Whether to include "all" as a valid choice (default: False). include_config: Whether to include "config" as a valid choice - (default: False). Used by ``perf`` for the winml-genai sentinel + (default: False). Used by ``perf`` for the ort-genai sentinel meaning "respect the bundle's genai_config.json routing". Returns: diff --git a/src/winml/modelkit/utils/constants.py b/src/winml/modelkit/utils/constants.py index 95e41cbe9..b536c31dd 100644 --- a/src/winml/modelkit/utils/constants.py +++ b/src/winml/modelkit/utils/constants.py @@ -82,7 +82,7 @@ # Inference runtimes selectable via ``winml perf --runtime``. -RuntimeName = Literal["auto", "winml", "winml-genai"] +RuntimeName = Literal["auto", "winml-ort", "ort-genai"] RUNTIME_NAMES: tuple[RuntimeName, ...] = get_args(RuntimeName) diff --git a/tests/e2e/test_perf_e2e.py b/tests/e2e/test_perf_e2e.py index 83a8067b5..1525ed575 100644 --- a/tests/e2e/test_perf_e2e.py +++ b/tests/e2e/test_perf_e2e.py @@ -1173,7 +1173,7 @@ def test_unknown_submodel_fails(self, tmp_path: Path): # =========================================================================== -# GenAI runtime (winml-genai): --device / --ep override +# GenAI runtime (ort-genai): --device / --ep override # =========================================================================== @@ -1184,7 +1184,7 @@ def _genai_perf_args( device: str | None = None, ep: str | None = None, ) -> list[str]: - """Build argv for a fast winml-genai perf run against a tiny bundle. + """Build argv for a fast ort-genai perf run against a tiny bundle. Kept deliberately small (2 iterations, 1 warmup, 4 new tokens) so the generation loop stays quick while still producing real timing samples. @@ -1193,7 +1193,7 @@ def _genai_perf_args( "-m", str(bundle_dir), "--runtime", - "winml-genai", + "ort-genai", "--iterations", "2", "--warmup", @@ -1211,9 +1211,9 @@ def _genai_perf_args( class TestPerfGenaiContract: - """Contract for the winml-genai ``config`` sentinel — no bundle required. + """Contract for the ort-genai ``config`` sentinel — no bundle required. - These lock the CLI surface that ``config`` is a winml-genai-only + These lock the CLI surface that ``config`` is a ort-genai-only ``--device`` value: it is advertised in ``--help`` and rejected with a helpful message on the single-shot ONNX path. They run on any host under ``-m e2e`` (no genai stack, model download, or accelerator needed). @@ -1224,7 +1224,7 @@ def test_help_lists_config_device(self): result = CliRunner().invoke(perf, ["--help"], obj={}, catch_exceptions=False) assert result.exit_code == 0 assert "[config|auto|cpu|gpu|npu]" in result.output - assert "winml-genai only" in result.output + assert "ort-genai only" in result.output def test_onnx_rejects_device_config(self, tmp_path: Path, onnx_model_path: Path): """``--device config`` is rejected on the ONNX runtime (genai-only sentinel).""" @@ -1240,7 +1240,7 @@ def test_onnx_rejects_device_config(self, tmp_path: Path, onnx_model_path: Path) ) assert result.exit_code == 2, f"expected UsageError exit 2, got {result.exit_code}" - assert "--device config is only valid with --runtime winml-genai" in result.output + assert "--device config is only valid with --runtime ort-genai" in result.output assert not output_file.exists(), "no report should be written on rejection" @@ -1329,7 +1329,7 @@ def _run( ) assert output_file.exists(), f"report not written: {output_file}" data = json.loads(output_file.read_text()) - assert data["benchmark_info"]["runtime"] == "winml-genai" + assert data["benchmark_info"]["runtime"] == "ort-genai" assert data["benchmark_info"]["generated_tokens"] > 0 return data diff --git a/tests/unit/commands/test_eval_pytorch.py b/tests/unit/commands/test_eval_pytorch.py index d7682b9e1..1e0ee55ba 100644 --- a/tests/unit/commands/test_eval_pytorch.py +++ b/tests/unit/commands/test_eval_pytorch.py @@ -24,7 +24,7 @@ def test_help_shows_runtime_choices(self) -> None: result = CliRunner().invoke(eval, ["--help"]) assert result.exit_code == 0 - assert "--runtime [winml|pytorch]" in result.output + assert "--runtime [winml-ort|pytorch]" in result.output def test_pytorch_runtime_dispatches_pytorch(self, tmp_path) -> None: captured: dict[str, WinMLEvaluationConfig] = {} @@ -89,7 +89,7 @@ def fake_evaluate(config: WinMLEvaluationConfig) -> SimpleNamespace: ) assert result.exit_code == 0, result.output - assert captured["config"].runtime == "winml" + assert captured["config"].runtime == "winml-ort" def test_pytorch_runtime_loads_from_config_file(self, tmp_path) -> None: config_path = tmp_path / "eval.json" @@ -355,7 +355,7 @@ def compute(self) -> dict[str, float]: assert result.config.runtime == "pytorch" assert result.config.model_id == "inferred/model" assert result.config.device == "cpu" - assert config.runtime == "winml" + assert config.runtime == "winml-ort" assert config.model_id is None def test_public_evaluate_sets_real_supplied_module_to_eval_mode(self) -> None: diff --git a/tests/unit/commands/test_perf_cli.py b/tests/unit/commands/test_perf_cli.py index c829a5850..fd8d95692 100644 --- a/tests/unit/commands/test_perf_cli.py +++ b/tests/unit/commands/test_perf_cli.py @@ -1749,7 +1749,7 @@ def test_to_dict_includes_schema_version_and_runtime(self) -> None: d = result.to_dict() assert d["schema_version"] == 2 - assert d["benchmark_info"]["runtime"] == "winml" + assert d["benchmark_info"]["runtime"] == "winml-ort" def test_iterations_reports_configured_count_without_duration(self) -> None: """Without --duration, benchmark_info.iterations is the configured value.""" diff --git a/tests/unit/commands/test_perf_genai.py b/tests/unit/commands/test_perf_genai.py index 7edca5508..62ba5564a 100644 --- a/tests/unit/commands/test_perf_genai.py +++ b/tests/unit/commands/test_perf_genai.py @@ -2,7 +2,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. # -------------------------------------------------------------------------- -"""Tests for the winml-genai perf runtime -- mock-based, no model, no genai. +"""Tests for the ort-genai perf runtime -- mock-based, no model, no genai. A fake GenaiSession whose ``generate_timed`` returns canned :class:`GenerationTiming` objects makes the aggregation deterministic, so these @@ -581,7 +581,7 @@ def test_to_dict_shape(self) -> None: } assert d["schema_version"] == 2 info = d["benchmark_info"] - assert info["runtime"] == "winml-genai" + assert info["runtime"] == "ort-genai" assert info["model_id"] == "Qwen/Qwen3-0.6B" assert info["running_model_path"] == "bundle" assert info["bundle_dir"] == "bundle" @@ -860,7 +860,7 @@ def test_write_genai_report_writes_json(self, tmp_path: Path) -> None: assert out.exists() data = json.loads(out.read_text(encoding="utf-8")) - assert data["benchmark_info"]["runtime"] == "winml-genai" + assert data["benchmark_info"]["runtime"] == "ort-genai" def test_display_genai_report_does_not_crash(self) -> None: display_genai_report(self._result(), Console()) @@ -934,7 +934,7 @@ def test_json_mode_emits_json(self, tmp_path: Path, monkeypatch, capsys) -> None run_genai_perf(cfg, console=Console(stderr=True), json_mode=True) out = capsys.readouterr().out - assert json.loads(out)["benchmark_info"]["runtime"] == "winml-genai" + assert json.loads(out)["benchmark_info"]["runtime"] == "ort-genai" def test_not_installed_becomes_click_error(self, monkeypatch) -> None: import click @@ -956,7 +956,7 @@ def run(self) -> None: # --------------------------------------------------------------------------- -# CLI dispatch (winml perf --runtime winml-genai) +# CLI dispatch (winml perf --runtime ort-genai) # --------------------------------------------------------------------------- @@ -994,7 +994,7 @@ def test_dispatches_and_maps_device_to_ep( ) bundle = _make_bundle(tmp_path) result = runner.invoke( - perf, ["-m", str(bundle), "--runtime", "winml-genai", "--device", "npu"] + perf, ["-m", str(bundle), "--runtime", "ort-genai", "--device", "npu"] ) assert result.exit_code == 0, result.output cfg = capture_run["config"] @@ -1019,7 +1019,7 @@ def test_device_auto_resolves_best_ep( ) bundle = _make_bundle(tmp_path) result = runner.invoke( - perf, ["-m", str(bundle), "--runtime", "winml-genai", "--device", "auto"] + perf, ["-m", str(bundle), "--runtime", "ort-genai", "--device", "auto"] ) assert result.exit_code == 0, result.output cfg = capture_run["config"] @@ -1034,7 +1034,7 @@ def test_explicit_ep_overrides_device( bundle = _make_bundle(tmp_path) result = runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--device", "npu", "--ep", "cpu"], + ["-m", str(bundle), "--runtime", "ort-genai", "--device", "npu", "--ep", "cpu"], ) assert result.exit_code == 0, result.output cfg = capture_run["config"] @@ -1047,7 +1047,7 @@ def test_explicit_ep_without_device( # --ep alone forces that EP even though --device is omitted (its # effective default for genai is "config" = respect the bundle). bundle = _make_bundle(tmp_path) - result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai", "--ep", "dml"]) + result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai", "--ep", "dml"]) assert result.exit_code == 0, result.output cfg = capture_run["config"] assert cfg.ep == "dml" @@ -1087,7 +1087,7 @@ def test_explicit_device_config_respects_bundle( # override (and it must not trigger device resolution). bundle = _make_bundle(tmp_path) result = runner.invoke( - perf, ["-m", str(bundle), "--runtime", "winml-genai", "--device", "config"] + perf, ["-m", str(bundle), "--runtime", "ort-genai", "--device", "config"] ) assert result.exit_code == 0, result.output cfg = capture_run["config"] @@ -1095,19 +1095,19 @@ def test_explicit_device_config_respects_bundle( assert cfg.ep is None def test_onnx_runtime_rejects_device_config(self, runner: CliRunner, tmp_path: Path) -> None: - # "config" is a winml-genai-only sentinel; auto resolves an ONNX path to + # "config" is a ort-genai-only sentinel; auto resolves an ONNX path to # winml and rejects it with a clear message rather than a generic error. result = runner.invoke(perf, ["-m", str(tmp_path / "model.onnx"), "--device", "config"]) assert result.exit_code != 0 - assert "winml-genai" in result.output + assert "ort-genai" in result.output def test_ep_flag_not_warned_as_ignored( self, runner: CliRunner, tmp_path: Path, capture_run: dict ) -> None: - # --ep is honored for winml-genai, so it must not appear in the + # --ep is honored for ort-genai, so it must not appear in the # "options are ignored" warning. bundle = _make_bundle(tmp_path) - result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai", "--ep", "qnn"]) + result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai", "--ep", "qnn"]) assert result.exit_code == 0, result.output assert "--ep" not in result.output @@ -1115,7 +1115,7 @@ def test_duration_op_tracing_conflict_not_fatal_for_genai( self, runner: CliRunner, tmp_path: Path, capture_run: dict ) -> None: # The --duration + --op-tracing conflict is a WinML-path constraint. For - # winml-genai both flags are ignored (non-fatal), so passing both must + # ort-genai both flags are ignored (non-fatal), so passing both must # warn and continue rather than aborting with the conflict UsageError. bundle = _make_bundle(tmp_path) result = runner.invoke( @@ -1124,7 +1124,7 @@ def test_duration_op_tracing_conflict_not_fatal_for_genai( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--duration", "5", "--op-tracing", @@ -1141,7 +1141,7 @@ def test_genai_iteration_defaults( self, runner: CliRunner, tmp_path: Path, capture_run: dict ) -> None: bundle = _make_bundle(tmp_path) - runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai"]) + runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai"]) cfg = capture_run["config"] assert cfg.iterations == 10 assert cfg.warmup == 2 @@ -1161,7 +1161,7 @@ def test_explicit_iterations_honored( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--iterations", "50", "--warmup", @@ -1182,7 +1182,7 @@ def test_prompt_and_max_new_tokens_forwarded( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--prompt", "hello there", "--max-new-tokens", @@ -1206,7 +1206,7 @@ def test_prompt_file_is_read_as_utf8( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--prompt-file", str(prompt_file), ], @@ -1228,7 +1228,7 @@ def test_prompt_and_prompt_file_are_mutually_exclusive( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--prompt", "argv prompt", "--prompt-file", @@ -1253,7 +1253,7 @@ def test_invalid_utf8_prompt_file_is_a_click_error( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--prompt-file", str(prompt_file), ], @@ -1268,7 +1268,7 @@ def test_default_prompt_used_when_omitted( self, runner: CliRunner, tmp_path: Path, capture_run: dict ) -> None: bundle = _make_bundle(tmp_path) - runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai"]) + runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai"]) # The CLI --prompt default must match the GenaiPerfConfig field default. assert capture_run["config"].prompt == perf_genai._DEFAULT_PROMPT assert capture_run["config"].prompt == GenaiPerfConfig(bundle_dir=Path("x")).prompt @@ -1277,7 +1277,7 @@ def test_apply_template_defaults_true( self, runner: CliRunner, tmp_path: Path, capture_run: dict ) -> None: bundle = _make_bundle(tmp_path) - runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai"]) + runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai"]) assert capture_run["config"].apply_template is True def test_no_apply_template_forwarded( @@ -1286,7 +1286,7 @@ def test_no_apply_template_forwarded( bundle = _make_bundle(tmp_path) runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--no-apply-template"], + ["-m", str(bundle), "--runtime", "ort-genai", "--no-apply-template"], ) assert capture_run["config"].apply_template is False @@ -1300,7 +1300,7 @@ def test_compile_flag_forwarded( "-m", str(bundle), "--runtime", - "winml-genai", + "ort-genai", "--compile", "--compile-timeout", "120", @@ -1316,7 +1316,7 @@ def test_memory_and_monitor_flags_forwarded( bundle = _make_bundle(tmp_path) result = runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--memory", "--monitor"], + ["-m", str(bundle), "--runtime", "ort-genai", "--memory", "--monitor"], ) assert result.exit_code == 0, result.output assert "--memory" not in result.output @@ -1331,7 +1331,7 @@ def test_warns_and_ignores_winml_only_flags( bundle = _make_bundle(tmp_path) result = runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--batch-size", "4"], + ["-m", str(bundle), "--runtime", "ort-genai", "--batch-size", "4"], ) assert result.exit_code == 0, result.output assert "ignored" in result.output.lower() @@ -1343,7 +1343,7 @@ def test_module_rejected(self, runner: CliRunner, tmp_path: Path, capture_run: d bundle = _make_bundle(tmp_path) result = runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--module", "Foo"], + ["-m", str(bundle), "--runtime", "ort-genai", "--module", "Foo"], ) assert result.exit_code != 0 assert "--module" in result.output @@ -1352,12 +1352,12 @@ def test_module_rejected(self, runner: CliRunner, tmp_path: Path, capture_run: d def test_submodel_rejected(self, runner: CliRunner, tmp_path: Path, capture_run: dict) -> None: # --submodel narrows a composite to one standalone sub-session; a genai # bundle is already the full composite generation pipeline, so it must be - # rejected rather than silently ignored (the winml-genai return runs before + # rejected rather than silently ignored (the ort-genai return runs before # the winml-path --submodel handling). bundle = _make_bundle(tmp_path) result = runner.invoke( perf, - ["-m", str(bundle), "--runtime", "winml-genai", "--submodel", "decoder"], + ["-m", str(bundle), "--runtime", "ort-genai", "--submodel", "decoder"], ) assert result.exit_code != 0 assert "--submodel" in result.output @@ -1366,7 +1366,7 @@ def test_submodel_rejected(self, runner: CliRunner, tmp_path: Path, capture_run: def test_onnx_file_rejected(self, runner: CliRunner, tmp_path: Path, capture_run: dict) -> None: onnx = tmp_path / "model.onnx" onnx.write_bytes(b"fake") - result = runner.invoke(perf, ["-m", str(onnx), "--runtime", "winml-genai"]) + result = runner.invoke(perf, ["-m", str(onnx), "--runtime", "ort-genai"]) assert result.exit_code != 0 assert "directory" in result.output.lower() assert "config" not in capture_run @@ -1384,7 +1384,7 @@ def _boom(*_a: object, **_k: object): raise ValueError("nope") monkeypatch.setattr(loader_mod, "resolve_loader_config", _boom) - result = runner.invoke(perf, ["-m", str(tmp_path / "nope"), "--runtime", "winml-genai"]) + result = runner.invoke(perf, ["-m", str(tmp_path / "nope"), "--runtime", "ort-genai"]) assert result.exit_code != 0 assert "config" not in capture_run @@ -1393,7 +1393,7 @@ def test_missing_genai_config_rejected( ) -> None: empty = tmp_path / "empty" empty.mkdir() - result = runner.invoke(perf, ["-m", str(empty), "--runtime", "winml-genai"]) + result = runner.invoke(perf, ["-m", str(empty), "--runtime", "ort-genai"]) assert result.exit_code != 0 assert "genai_config.json" in result.output assert "config" not in capture_run @@ -1416,7 +1416,7 @@ def test_hf_model_id_autobuilds_and_dispatches( winml_models, "build_genai_bundle", _fake_build_genai_bundle(build_calls) ) - result = runner.invoke(perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "winml-genai"]) + result = runner.invoke(perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "ort-genai"]) assert result.exit_code == 0, result.output # Built once, pinned to the NPU HTP via QNN regardless of --device. @@ -1448,7 +1448,7 @@ def test_autobuild_reuses_cached_bundle( winml_models, "build_genai_bundle", _fake_build_genai_bundle(build_calls) ) - result = runner.invoke(perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "winml-genai"]) + result = runner.invoke(perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "ort-genai"]) assert result.exit_code == 0, result.output assert "build" not in build_calls # cache hit: never rebuilt @@ -1476,7 +1476,7 @@ def test_rebuild_forces_autobuild( ) result = runner.invoke( - perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "winml-genai", "--rebuild"] + perf, ["-m", "Qwen/Qwen3-0.6B", "--runtime", "ort-genai", "--rebuild"] ) assert result.exit_code == 0, result.output @@ -1505,7 +1505,7 @@ def test_autobuild_honored_flags_not_warned_as_ignored( "-m", "Qwen/Qwen3-0.6B", "--runtime", - "winml-genai", + "ort-genai", "--rebuild", "--task", "text-generation", @@ -1524,7 +1524,7 @@ def test_prebuilt_bundle_still_warns_build_flags( # A prebuilt bundle dir ignores the build-driving flags, so --rebuild is # reported as ignored (no auto-build happened). bundle = _make_bundle(tmp_path) - result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "winml-genai", "--rebuild"]) + result = runner.invoke(perf, ["-m", str(bundle), "--runtime", "ort-genai", "--rebuild"]) assert result.exit_code == 0, result.output assert "--rebuild" in result.output @@ -1554,7 +1554,7 @@ def test_cache_hit_warns_dropped_build_input_flags( "-m", "Qwen/Qwen3-0.6B", "--runtime", - "winml-genai", + "ort-genai", "--precision", "w8a16", "--task", @@ -1584,7 +1584,7 @@ def test_autobuild_without_recipe_rejected( monkeypatch.setattr(winml_models, "resolve_genai_bundle", lambda _mt: None) result = runner.invoke( - perf, ["-m", "google-bert/bert-base-uncased", "--runtime", "winml-genai"] + perf, ["-m", "google-bert/bert-base-uncased", "--runtime", "ort-genai"] ) assert result.exit_code != 0 @@ -1594,7 +1594,7 @@ def test_autobuild_without_recipe_rejected( def test_runtime_help_shows_auto_default(self, runner: CliRunner, capture_run: dict) -> None: result = runner.invoke(perf, ["--help"]) assert result.exit_code == 0 - assert "[auto|winml|winml-genai]" in result.output + assert "[auto|winml-ort|ort-genai]" in result.output assert "default: auto" in result.output assert "config" not in capture_run @@ -1602,7 +1602,7 @@ def test_runtime_help_shows_auto_default(self, runner: CliRunner, capture_run: d class TestAutoRuntime: def test_selects_genai_for_bundle_folder(self, tmp_path: Path) -> None: bundle = _make_bundle(tmp_path) - assert _resolve_runtime("auto", str(bundle)) == "winml-genai" + assert _resolve_runtime("auto", str(bundle)) == "ort-genai" @pytest.mark.parametrize("model_kind", ["plain-folder", "onnx-file", "model-id"]) def test_selects_winml_without_bundle_marker(self, tmp_path: Path, model_kind: str) -> None: @@ -1617,8 +1617,8 @@ def test_selects_winml_without_bundle_marker(self, tmp_path: Path, model_kind: s else: value = "organization/model" - assert _resolve_runtime("auto", value) == "winml" + assert _resolve_runtime("auto", value) == "winml-ort" - @pytest.mark.parametrize("runtime", ["winml", "winml-genai"]) + @pytest.mark.parametrize("runtime", ["winml-ort", "ort-genai"]) def test_preserves_explicit_runtime(self, runtime: str) -> None: assert _resolve_runtime(runtime, "organization/model") == runtime diff --git a/tests/unit/eval/test_eval.py b/tests/unit/eval/test_eval.py index 19ed0f772..c02379189 100644 --- a/tests/unit/eval/test_eval.py +++ b/tests/unit/eval/test_eval.py @@ -1618,7 +1618,7 @@ def test_make_onnx_reference_config_uses_independent_environment(self): assert reference.model_path == "reference.onnx" assert reference.model_id is None assert reference.reference_path is None - assert reference.runtime == "winml" + assert reference.runtime == "winml-ort" assert reference.device == "gpu" assert reference.ep == "dml" assert reference.mode == "onnx" diff --git a/tests/unit/eval/test_run_eval_script.py b/tests/unit/eval/test_run_eval_script.py index e2a363d2c..b01eb64ad 100644 --- a/tests/unit/eval/test_run_eval_script.py +++ b/tests/unit/eval/test_run_eval_script.py @@ -1209,7 +1209,7 @@ def _entry(hf_id="microsoft/resnet-50", task="image-classification"): def _perf_result(mean=1.25): return { "schema_version": 2, - "benchmark_info": {"runtime": "winml", "model_id": "model.onnx"}, + "benchmark_info": {"runtime": "winml-ort", "model_id": "model.onnx"}, "model_info": {"input_names": ["input"], "output_names": ["output"]}, "latency_ms": { "mean": mean, diff --git a/tests/unit/eval/test_run_llm_eval_script.py b/tests/unit/eval/test_run_llm_eval_script.py index 7a5779d42..fae34e1d4 100644 --- a/tests/unit/eval/test_run_llm_eval_script.py +++ b/tests/unit/eval/test_run_llm_eval_script.py @@ -47,7 +47,7 @@ def _perf_report( ) -> dict: return { "benchmark_info": { - "runtime": "winml-genai", + "runtime": "ort-genai", "bundle_dir": str(bundle_dir), "ep": ep, "device": device, @@ -96,7 +96,7 @@ def test_perf_args_match_genai_command(self, runner, tmp_path: Path) -> None: ep="qnn", ) - assert args[args.index("--runtime") + 1] == "winml-genai" + assert args[args.index("--runtime") + 1] == "ort-genai" assert args[args.index("--device") + 1] == "npu" assert args[args.index("--ep") + 1] == "qnn" assert args[args.index("--compile-timeout") + 1] == "1800" @@ -275,7 +275,7 @@ def test_result_validates_against_schema(self, runner) -> None: "total_vram_mb": 4000.0, "gpu_memory_gb": 4.0, }, - command="winml perf -m bundle --runtime winml-genai --device npu", + command="winml perf -m bundle --runtime ort-genai --device npu", timed_out=False, ) schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8")) diff --git a/tests/unit/utils/test_runtime_constants.py b/tests/unit/utils/test_runtime_constants.py index 13be57e8c..2962220e2 100644 --- a/tests/unit/utils/test_runtime_constants.py +++ b/tests/unit/utils/test_runtime_constants.py @@ -13,4 +13,4 @@ def test_runtime_names_match_runtime_name_literal() -> None: assert get_args(RuntimeName) == RUNTIME_NAMES - assert RUNTIME_NAMES == ("auto", "winml", "winml-genai") + assert RUNTIME_NAMES == ("auto", "winml-ort", "ort-genai")