[Kubernetes] In a pod, context=None means this cluster, not the kubeconfig's current-context - #6
Open
tigist-far wants to merge 16 commits into
Open
[Kubernetes] In a pod, context=None means this cluster, not the kubeconfig's current-context#6tigist-far wants to merge 16 commits into
tigist-far wants to merge 16 commits into
Conversation
…062) Every Azure launch was failing at network interface creation with "TypeError: Object of type SubResource is not JSON serializable". _create_network_interface built the subnet reference with compute.SubResource (an azure.mgmt.compute.models type) but embedded it in a network.IPConfiguration sent to the network client. azure-mgmt-network 31.0.0 (2026-07-01) regenerated its models with the TypeSpec toolchain and a strict SdkJSONEncoder that only serializes network-SDK models, so passing a foreign compute model now raises. The previous serializer (<=30.2.0) accepted any object exposing `.id`, which had masked this bug since the SDK refactor in PR #4139. Use network.Subnet(id=...), the declared type of IPConfiguration.subnet, which serializes correctly on both 30.x and 31.x. Also drop the now-unused compute models alias in that function. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: GitHub Action <action@github.com>
SKYPILOT_SSH_CONTROL_PATH and SKYPILOT_SSH_CONTROL_PERSIST point the SSH command runner at a ControlMaster the user opened by hand. SkyPilot multiplexes under a private %C-keyed path it can never share, so a login node that demands a password and a one-time code per connection prompts on every remote command. Unset, nothing changes. Signed-off-by: Taufeeque <taufeeque@far.ai>
`bash --login -i -c` makes the shell interactive, and RHEL's /etc/profile then sources /etc/profile.d with output visible, so a login banner lands in the captured output SkyPilot parses (`Failed to set up SkyPilot runtime on cluster` with a banner for stdout). --login already provides /etc/profile, where PATH and module setup live on an HPC login node. Signed-off-by: Taufeeque <taufeeque@far.ai>
slurmctld intermittently fails a query (`slurm_load_job_state: Unable to query jobs state`) and squeue exits non-zero. Raising out of _wait_for_job_nodes declares the provision failed and cancels a healthy PENDING job, losing its accumulated queue priority. Retry the poll. Signed-off-by: Taufeeque <taufeeque@far.ai>
Immediately after an allocation the compound squeue/scontrol node query can exit 0 with empty stdout during a transient slurmctld failure, and get_job_nodes then raises and tears down the allocation. Retry that narrow query for a bounded interval; persistent failures still raise. Signed-off-by: Taufeeque <taufeeque@far.ai>
…k nodes Backport of skypilot-org/skypilot#9924, merged after 0.13.0. Without a node IP, Ray on a hostNetwork node with several NICs can report an interface other than status.podIP, which is what SkyPilot recorded for the node. Signed-off-by: Taufeeque <taufeeque@far.ai>
Backport of skypilot-org/skypilot#9924. The rank sorter looked the reported IP up in the recorded map and crashed (`'<' not supported between 'int' and 'NoneType'`) before the run block started. Signed-off-by: Taufeeque <taufeeque@far.ai>
…ward API Backport of skypilot-org/skypilot#9924: expose status.podIP to the pod so the Ray bootstrap can bind to it. Signed-off-by: Taufeeque <taufeeque@far.ai>
…RBAC An ordinary Deployment GET already carries .status. Waiting for an HA jobs controller via the dedicated /deployments/<name>/status subresource needs a separate RBAC grant that namespaced users often lack, so the Deployment is created and provisioning then fails 403. Signed-off-by: Taufeeque <taufeeque@far.ai>
Clusters with namespace quotas require CPU and memory requests on every container. The HA-only init container specified neither, so the Deployment controller could not create a Pod. Equal requests and limits keep the Guaranteed QoS; init scheduling uses max(init, app), so the controller Pod's effective request is unchanged. Signed-off-by: Taufeeque <taufeeque@far.ai>
kubernetes.provision_timeout applied to ordinary Pods, but an HA Deployment was gated by a separate hard-coded 300 s wait. The init container copies the controller home onto durable storage and can legitimately exceed that on a fresh PVC. Pass the configured timeout through, including the documented negative value for an indefinite wait. Signed-off-by: Taufeeque <taufeeque@far.ai>
The wheel is built from this branch without npm, and MANIFEST.in ships sky/dashboard/out when present. These files are byte-identical to the PyPI 0.13.0 wheel's. Signed-off-by: Taufeeque <taufeeque@far.ai>
First FAR.AI build of SkyPilot 0.13.0; FARAI.md describes the branch and how to cut the next one. Signed-off-by: Taufeeque <taufeeque@far.ai>
Carried over from the launcher repository, which used to test this logic against its patch script. Signed-off-by: Taufeeque <taufeeque@far.ai>
…econfig's current-context Utilities that accept `context=None` substituted the kubeconfig's current-context name and handed it to the API client explicitly. Inside a pod that is the wrong answer: the client adaptor authenticates an explicit name through the kubeconfig, and the kubeconfig a managed-jobs controller holds is the *client's* (the API server rebuilds it from the client's `KUBECONFIG` at every launch), whose current-context may authenticate through an exec plugin the pod does not have. The provisioner passes `context=None` on the controller (`get_context_from_config` maps the in-cluster context to None so pods are created with in-cluster auth). A launch pinned with `allowed_nodes` names or ips lists nodes per pod through `get_kubernetes_nodes(context=None)`, which substituted the client's current-context and failed every pod with `unknown command "oidc-login" for "kubectl"` -- once per pod, on every retry -- while an unpinned launch of the same task provisioned fine. `resolve_current_context()` returns the in-cluster context name when in-cluster auth is available and the kubeconfig's current-context otherwise; the five substitutions in this module use it. The adaptor takes the in-cluster name through in-cluster auth, and per-context config lookups now key on the cluster the pod is in rather than on whichever context the client had selected. Outside a pod nothing changes. Tests cover both environments and an explicit context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
|
Checks run for this PR (the repository runs no CI on PRs): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A managed job pinned to nodes (
kubernetes.allowed_nodeswithnamesorips) failed to provision on the jobs controller, every pod, every retry, withunknown command "oidc-login" for "kubectl", while the same task unpinned provisioned fine.The controller provisions with
context=None:get_context_from_configmaps the in-cluster context toNoneso pods are created with in-cluster auth. Pinning is the only path that lists nodes per pod (inject_allowed_nodes_affinity→get_kubernetes_nodes), andget_kubernetes_nodes(context=None)replacedNonewith the kubeconfig's current-context name before building its client. The adaptor authenticates an explicit name through the kubeconfig, and the kubeconfig on the controller is the client's, rebuilt by the API server from the client'sKUBECONFIGat every launch: its current-context was the developer's other cluster, authenticated with an exec plugin the pod does not have.What
resolve_current_context()returns the in-cluster context name when in-cluster auth is available and the kubeconfig's current-context otherwise. The five places insky/provision/kubernetes/utils.pythat substituted the current-context name forNoneuse it (get_kubernetes_nodes,get_allocated_resources_by_node,get_kubernetes_node_infotwice,get_skypilot_pods). The adaptor takes the in-cluster name through in-cluster auth; per-context config lookups key on the cluster the pod is in instead of whichever context the client had selected. Outside a pod nothing changes.Tests
tests/unit_tests/kubernetes/test_kubernetes_utils.py: in a pod,context=Nonereaches the client as the in-cluster context and the allowed-nodes filter sees the same name; outside a pod it is still the kubeconfig's current-context; an explicit context is used as given. Whole file: 212 passed.Upstream
The same code is on upstream
master(get_kubernetes_nodesandget_context_from_configare identical), so this is an upstream bug. The commit is written upstream-first; a copy cherry-picked onto upstream master is ontigist/upstream-in-cluster-current-contextfor an upstream PR (this repository is not a GitHub fork of skypilot-org/skypilot, so that PR must come from one that is).The workaround in nemotron/nemo-rl (forwarding a kubeconfig whose current-context is the pinned cluster) stays; it is harmless and covers today's pinned build.