docs(examples): add end-to-end GPU inference codelab

[Li Chen (AaronLiChen)]

What this adds

Adds the first example under examples/end-to-end-AI-codelabs: a guided AKS lab that starts with zero GPU nodes and follows a three-pod inference workload through:

1. Kueue workload admission
2. ProvisioningRequest creation with count: 3
3. AKS cluster autoscaler GPU scale-up from zero to three
4. vLLM startup with Qwen2.5-0.5B on each node
5. Three validated OpenAI-compatible inference requests
6. Cleanup and GPU scale-down

The lab includes eight guided modules, Kubernetes manifests, preflight/deployment/watch/cleanup scripts, checkpoints, and focused troubleshooting.

Scope note

The autoscaling path uses a conventional GPU pool with the AKS-installed driver and the upstream NVIDIA device plugin. Managed GPU node pools don't support cluster autoscaler during preview, so the lab documents that limitation rather than presenting the two features as compatible.

The node pool carries nvidia.com/gpu.present=true. This satisfies the upstream device-plugin chart's discovery affinity without requiring Node Feature Discovery.

Live validation

Validated end to end in eastasia:/

• Pool: 3 × Standard_NV6ads_A10_v5 (A10-4Q, 4 GB each), autoscaling 0–3
• Initial state: POOL=0, READY=0, Job suspended
• Kueue created one Workload and a ProvisioningRequest with count: 3
• CAS scaled the pool 0→3
• ProvisioningRequest reached Provisioned=True, reason CapacityIsProvisioned
• Kueue admitted the Job after all three nodes became Ready
• Job completed 3/3; every pod printed INFERENCE_VALIDATED
• Submission-to-completion time: 418 seconds

vLLM was constrained for the 4-GB profile with 70% GPU memory utilization, a 2,048-token context, four sequences, and eager execution.

Additional validation

• Shell syntax validation for every script
• YAML parsing for both manifests
• Embedded inference command syntax validation
• Relative Markdown link validation
• Kueue 0.17.1 chart pull and render from MCR
• NVIDIA device-plugin 0.17.0 chart render and live registration with kubelet

All validation resource groups were deleted after testing.

[Li Chen (AaronLiChen)]

Historical validation attempt — superseded. This attempt is no longer representative of the checked-in codelab. The later East Asia run completed the full zero-to-three GPU path successfully; see the current PR description and final validation comment.

[Li Chen (AaronLiChen)]

Historical validation attempt — superseded. This attempt is no longer representative of the checked-in codelab. The later East Asia run completed the full zero-to-three GPU path successfully; see the current PR description and final validation comment.

[Li Chen (AaronLiChen)]

Update: This successful three-node result is now reflected in the checked-in defaults and documentation.

Successful end-to-end validation:

• Subscription: AKS E2E - GPU SKU Test (internal validation environment)
• Region: eastasia
• Pool: 3 × Standard_NV6ads_A10_v5 (A10-4Q, 4 GB each), autoscaling 0–3
• Start: POOL=0, READY=0, Job suspended
• CAS: pool 0→3; ProvisioningRequest Provisioned=True, reason CapacityIsProvisioned
• Admission: all 3 nodes Ready before Kueue released the Job
• Result: Job Complete 3/3; all three pods printed INFERENCE_VALIDATED
• End-to-end submission-to-completion time: 418 seconds

Validation produced two fixes:

1. Add nvidia.com/gpu.present=true to node-pool labels so the upstream NVIDIA device-plugin chart schedules without Node Feature Discovery.
2. Constrain vLLM for the 4-GB A10-4Q profile (--gpu-memory-utilization 0.70, --max-model-len 2048, --max-num-seqs 4, --enforce-eager).

The validation resource group was deleted after testing.

[Li Chen (AaronLiChen)]

The checked-in codelab now matches the validated run (fe3218ed): East Asia, three Standard_NV6ads_A10_v5 nodes, pool range 0–3, three-pod Job, three-GPU queue quota, 4-GB-safe vLLM settings, node-pool GPU discovery label, and verification that all three pods print INFERENCE_VALIDATED. The PR description and earlier validation comments have been updated accordingly.

[Copilot]

🟡 Changes recommended

Cleanup can delete shared resources, while configuration and preflight gaps can leave workloads stuck or fail unexpectedly.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds an end-to-end AKS GPU inference codelab covering Kueue admission, autoscaling, vLLM inference, monitoring, and cleanup.

Changes:

• Adds eight guided codelab modules and documentation.
• Adds infrastructure, validation, monitoring, and cleanup scripts.
• Adds Kueue configuration and a three-pod vLLM inference workload.

File summaries

File	Description
examples/end-to-end-AI-codelabs/README.md	Adds the codelab index.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/README.md	Documents the complete lab.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/01-check-prerequisites.md	Documents prerequisites.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/02-create-cluster.md	Guides cluster creation.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/03-create-gpu-pool.md	Guides GPU pool setup.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/04-install-controllers.md	Documents controller installation.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/05-configure-provisioning.md	Configures Kueue provisioning.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/06-run-inference.md	Runs and validates inference.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/07-observe-and-troubleshoot.md	Provides diagnostics.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/08-cleanup.md	Documents cleanup.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/lib.sh	Defines shared configuration.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh	Validates prerequisites and quota.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/10-create-cluster.sh	Creates the AKS cluster.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/20-create-gpu-pool.sh	Creates the autoscaling GPU pool.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/30-install-controllers.sh	Installs Kueue and the NVIDIA plugin.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/40-configure-queue.sh	Applies the Kueue queue.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/50-submit-inference.sh	Submits the inference job.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/60-watch-flow.sh	Watches execution progress.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/90-cleanup.sh	Removes lab resources.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/manifests/kueue-gpu-queue.yaml	Defines Kueue resources and provisioning.
examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/manifests/inference-job.yaml	Defines the vLLM workload.

Review details

Suppressed comments (2)

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/manifests/kueue-gpu-queue.yaml:14

• LAB_GPU_POOL is treated as configurable by the scripts, but this flavor hard-codes agentpool: gpupool. If a user changes the variable, the script creates and watches one pool while Kueue provisions against another name, so the workload remains pending even when the custom pool is Ready. Template the manifest from the shared configuration or make the pool name non-configurable.

  nodeLabels:
    agentpool: gpupool

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh:32

• This preflight only checks that $LAB_GPU_SKU is listed and unrestricted. It does not verify that the SKU advertises an NVIDIA GPU or that the two $LAB_SYSTEM_SKU nodes and their quota are available, so a run can pass preflight and then either fail during cluster creation or remain pending without nvidia.com/gpu capacity. Validate both required SKU families and the GPU capability before reporting success.

step "Checking $LAB_GPU_SKU in $LAB_LOCATION"
SKU_JSON=$(az vm list-skus --location "$LAB_LOCATION" --size "$LAB_GPU_SKU" --all -o json)
SKU_COUNT=$(python3 -c 'import json,sys; print(len(json.load(sys.stdin)))' <<<"$SKU_JSON")
[[ "$SKU_COUNT" -gt 0 ]] || fail "$LAB_GPU_SKU isn't listed in $LAB_LOCATION. Set LAB_LOCATION or LAB_GPU_SKU."

• Files reviewed: 21/21 changed files
• Comments generated: 6
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Unresolved critical scheduling and cluster-scoped resource ownership and cleanup issues remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh:25

• This preflight claims to check the required resource providers, but it checks only Compute and ContainerService while the next script also creates AKS networking and enables a managed identity. A subscription with Microsoft.Network or Microsoft.ManagedIdentity unregistered can pass this check and then fail during az aks create; check every provider required by that command so the failure is reported before resource creation.

for provider in Microsoft.Compute Microsoft.ContainerService; do
  state=$(az provider show --namespace "$provider" --query registrationState -o tsv 2>/dev/null || true)
  if [[ "$state" == "Registered" ]]; then
    pass "$provider is registered"
  else
    fail "$provider is $state. Run: az provider register --namespace $provider"

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh:77

• These checks compare each SKU only with its own family quota. Azure also enforces the subscription's Total Regional vCPUs limit across both the two system nodes and three GPU nodes; a subscription can have family headroom but fewer than the 26 total vCPUs required, so preflight passes and cluster or pool creation then fails. Check the aggregate requirement against the total-regional usage entry as well.

quota = next((q for q in json.load(open(usage_file))
              if norm(q.get("name", {}).get("value", "")) == norm(family)), None)
if quota is None:
    raise SystemExit(f"FAIL|quota family {family!r} isn't present in the regional usage list")
used, limit = int(quota["currentValue"]), int(quota["limit"])
required = vcpus * count
if limit - used < required:
    raise SystemExit(
        f"FAIL|{family} has {limit-used} free vCPUs; {count} {expected_sku} nodes need {required}")

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/20-create-gpu-pool.sh:24

• The existing-pool validation does not check the node operating system. A same-SKU Windows gpupool can therefore be accepted, but the workload uses a Linux vLLM image and cannot run on Windows nodes. Validate that an existing pool is Linux before treating it as compatible.

if pool.get("mode", "").lower() != "user":
    errors.append(f"mode is {pool.get('mode')}, expected User")
if not pool.get("enableAutoScaling"):
    errors.append("cluster autoscaler is disabled")

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/90-cleanup.sh:21

• Deleting the Job is not enough synchronization for Kueue: the Job can disappear while its Workload and ProvisioningRequest are still being reconciled. This immediately removes the queue, admission, and provisioning resources, racing the controller and potentially leaving provisioning state or GPU capacity behind. Wait until the namespace has no Workloads or ProvisioningRequests, with a timeout/error path, before deleting the queue manifest.

  kubectl delete -f "$ROOT/manifests/inference-job.yaml" --ignore-not-found
  kubectl delete -f "$ROOT/manifests/kueue-gpu-queue.yaml" --ignore-not-found

• Files reviewed: 21/21 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Four unresolved findings remain, including a critical cleanup context-safety issue and three moderate configuration or ownership issues.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh:65

• Module 1 requires an override SKU to expose one GPU per node because the workload and documented zero-to-three-node flow assume three GPUs map to three nodes, but this check accepts any gpus >= 1. A multi-GPU SKU can therefore pass preflight and satisfy all three pods on fewer nodes, invalidating the codelab's advertised topology. Reject non-1 GPU SKUs here, or update the workload and expectations to support them.

if require_gpu and gpus != 1:
    raise SystemExit(

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/00-preflight.sh:20

• The preflight labels these as the required resource providers but checks only Compute and ContainerService. A subscription with the Network provider (or another provider required by the managed-identity AKS create path) unregistered passes preflight and then fails in Module 2. Include every provider used by the exact az aks create invocation, at minimum Microsoft.Network, in this check.

for provider in Microsoft.Compute Microsoft.ContainerService; do

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/90-cleanup.sh:21

• This deletes the cluster-scoped ResourceFlavor, ProvisioningRequestConfig, AdmissionCheck, and ClusterQueue from the queue manifest, but the guard only verifies a label on the namespace. Because 40-configure-queue.sh applies these fixed names without checking ownership, running the codelab against an existing cluster can mutate or delete another Kueue setup. Add ownership markers and refuse unowned cluster-scoped objects before applying/deleting them, or use unique names.

    "admissioncheck/gpu-inference-provisioning"

• Files reviewed: 21/21 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Unresolved moderate watcher and cleanup issues can cause stale monitoring, ongoing GPU billing, or incomplete cleanup.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/modules/07-observe-and-troubleshoot.md:47

• The backslash before the dot is consumed by the shell because this -o argument isn't quoted, so kubectl receives ...nvidia.com/gpu instead of the escaped extended-resource key. The custom-columns path then splits the resource name at the dot and won't show the GPU capacity; quote the argument or pass a doubled backslash so the escape reaches kubectl.

  -o custom-columns=NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/60-watch-flow.sh:80

• When this 45-minute timeout fires, only the watcher exits; the Job and any ProvisioningRequest remain in the cluster. A suspended or running Job can therefore keep the GPU pool at three nodes and continue billing after the script reports failure. Bound the Job with an active deadline or clean up the workload on timeout and interruption.

  if (( elapsed >= 2700 )); then
    print_diagnostics
    fail "Timed out after 45 minutes."

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/60-watch-flow.sh:36

• These queries always select .items[0] and aren't scoped to the Workload or ProvisioningRequest created for the current Job. After 50-submit-inference.sh deletes and recreates the fixed-name Job, the previous Kueue objects can still exist briefly, so the watcher may display an old admission or provisioning state while monitoring the new Job. Wait for old objects to disappear before reapplying, or select resources through the current Job's owner reference.

  workload=$(kubectl -n "$LAB_NAMESPACE" get workload \
    -o jsonpath='{.items[0].status.conditions[?(@.type=="Admitted")].status}' \
    2>/dev/null || true)
  [[ -n "$workload" ]] || workload="Pending"
  provisioning=$(kubectl -n "$LAB_NAMESPACE" get provisioningrequest \

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/60-watch-flow.sh:31

• Because lib.sh enables set -euo pipefail, a transient nonzero from kubectl get nodes makes this assignment fail and exits the watcher instead of retrying during its 45-minute loop. Handle this pipeline failure (for example, default ready_nodes to 0) so temporary API errors don't abort the observation flow.

  ready_nodes=$(kubectl get nodes -l "agentpool=$LAB_GPU_POOL" --no-headers 2>/dev/null \
    | awk '$2 == "Ready" {count++} END {print count+0}')

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/90-cleanup.sh:40

• Deleting the Job does not synchronously remove Kueue's generated Workload and ProvisioningRequest. Immediately deleting the queue and namespace can race those controllers, leaving namespace termination stuck or cleanup incomplete; the repository's CAS example waits until these objects disappear before removing queue resources. Poll for an empty workloads,provisioningrequests set, with a bounded timeout, before deleting this manifest.

  kubectl delete -f "$ROOT/manifests/kueue-gpu-queue.yaml" --ignore-not-found

• Files reviewed: 21/21 changed files
• Comments generated: 1
• Review effort level: Lite

[Li Chen (AaronLiChen)]

Also addressed the latest review's suppressed findings in f0cfb8a9:

• Preflight checks Microsoft.Network and Microsoft.ManagedIdentity in addition to Compute and ContainerService.
• Preflight checks aggregate Total Regional vCPUs for two system nodes plus three GPU nodes (26 vCPUs with defaults).
• Existing pool validation now requires Linux.
• The Job has a 40-minute active deadline; watcher timeout/interruption deletes it to release GPU capacity.
• Resubmission waits for the previous Workload and ProvisioningRequest to disappear.
• Transient kubectl get nodes failures no longer terminate the watcher.
• The GPU custom-column argument is quoted so the extended-resource escape reaches kubectl.
• Cleanup waits for generated Kueue resources before removing queue configuration.

[Copilot]

🔵 Needs a closer look

Two moderate setup issues and one diagnostic nit remain unresolved.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/30-install-controllers.sh:6

• Unlike the other scripts that mutate the cluster, this script never calls require_lab_context before the Helm installs. If the user switches the current kubeconfig context between modules, both releases are installed or upgraded in whichever cluster is active while the Azure variables still refer to the lab cluster, so this can modify an unrelated cluster. Call require_lab_context before the first Helm command (or pass an explicit context).

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/30-install-controllers.sh:7

• helm repo add fails when the nvdp repository already exists, and set -e exits before either chart is installed. That makes Module 4 fail on reruns or on machines that already have this repo configured; use --force-update (or explicitly handle an existing repo) so the setup remains repeatable.

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin >/dev/null

examples/end-to-end-AI-codelabs/kueue-gpu-inference-autoscaling/scripts/60-watch-flow.sh:24

• On failure or timeout, the watcher calls print_diagnostics and then deletes the Job (lines 88/93), but this function only describes pods and never prints their container logs. That loses the vLLM/model-load error while Modules 6 and 7 tell users to inspect kubectl logs job/vllm-inference-check; emit the pod logs before stop_workload (or retain the Job) so failures remain diagnosable.

  echo "--- Pending pod ---"
  kubectl -n "$LAB_NAMESPACE" describe pod -l job-name="$LAB_JOB" 2>/dev/null || true

• Files reviewed: 21/21 changed files
• Comments generated: 0 new
• Review effort level: Lite