Skip to content

fix(kubestellar): stop KubeFlex ControlPlane status hot loop - #648

Open
castrojo wants to merge 4 commits into
mainfrom
fix/kubeflex-reconcile-loop
Open

fix(kubestellar): stop KubeFlex ControlPlane status hot loop#648
castrojo wants to merge 4 commits into
mainfrom
fix/kubeflex-reconcile-loop

Conversation

@castrojo

@castrojo castrojo commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Upgrade the GitOps-managed KubeStellar core chart from 0.29.0 (KubeFlex v0.9.1) to 0.30.0 (KubeFlex v0.9.3) and document the live failure mode and retired nginx endpoint.

Verified causal chain

  • kubectl get ingressclass returned no classes; wds1-system/wds1 is an owner-referenced generated Ingress with ingressClassName: nginx, empty status.loadBalancer, and an unchanged resourceVersion during a 10-second loop sample.
  • The canonical upstream detector, kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx, returns No resources found; there is no controller to claim this object.
  • The live wds1 ControlPlane is nevertheless Ready=True, and its PostCreateHook templates contain only RBAC, ConfigMap, Service, Deployment, and Job resources — no Ingress. KubeFlex creates the endpoint in its type-specific reconciler, outside PostCreateHooks.
  • KubeFlex v0.9.1 logs repeatedly ended with failed to update final status: ... object has been modified; this is the observed requeue signal. The sustained ~6.5 Mbps RX is controller status/reconcile traffic, not external ingress traffic.

Upstream networking policy

ADR-0004 says external reachability is unnecessary for this lab. The Kubernetes Steering and Security Response Committees' Ingress NGINX retirement statement says: "There will be no more releases for bug fixes, security patches, or any updates of any kind after the project is retired" and warns that remaining on retired Ingress NGINX leaves users vulnerable. Kubernetes' Ingress Controllers documentation says: "The Kubernetes project recommends using Gateway instead of Ingress. The Ingress API has been frozen." We therefore do not install ingress-nginx, point the object at another IngressClass, or install Gateway API/controller components for an unused endpoint. If external reachability is needed later, use the Gateway API getting-started guidance.

Fix / follow-up

Use the newer core chart/KubeFlex operator release through ArgoCD. The v0.9.3 chart values have no ingress-disable switch; the generated endpoint is upstream KubeFlex behavior. Upstream opt-out request: kubestellar/kubeflex#709. If status conflicts persist after rollout, the remaining code fix belongs upstream rather than in an ingress manifest. External reachability remains off by default.

Validation

  • just lint passes.
  • helm template ... core-chart --version 0.30.0 passes.
  • Live ArgoCD sync and post-rollout log/Prometheus verification remain pending: the live Application is still on 0.29.0 because the local ArgoCD proxy is unavailable.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

The live loop was driven by ControlPlane status-update conflicts, not the inert nginx Ingress. Move core-chart from 0.29.0/KubeFlex 0.9.1 to 0.30.0/KubeFlex 0.9.3 and document the diagnostic distinction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Collaborator Author

Live verification update

Before the PR can be merged, the live app is still core-chart 0.29.0 (kubectl -n argocd get application kubestellar: Synced/Healthy, targetRevision 0.29.0). just argocd-sync could not connect to the local ArgoCD proxy (localhost:8443).

The baseline Prometheus query returned 816587.7448 bytes/s for kubeflex-controller-manager over [10m] (about 6.53 Mbps), matching the reported 6.5 Mbps. Logs still show the failed to update final status ... object has been modified loop. Post-merge ArgoCD rollout must re-run the log and Prometheus checks.

pi and others added 3 commits August 13, 2026 18:11
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff, CI (lint/validate green), and the causal-chain writeup.

This is a real root-cause fix, not a symptom patch. The PR traces the hot loop to KubeFlex v0.9.1's status-write race across infra/PostCreateHook/final-readiness phases (object has been modified conflicts), independently rules out ingress-nginx as the cause (no controller pods, no ingressclasses, generated Ingress has no status), and correctly declines to work around KubeFlex's hardcoded Ingress generation by installing an unused controller — consistent with ADR-0004 and upstream's Ingress-frozen/Gateway-preferred guidance. The runbook and skill docs are updated with the verified failure signature, which will help whoever triages this next if it recurs.

One caveat worth tracking post-merge: live ArgoCD sync and post-rollout log/Prometheus verification are explicitly called out as pending (the local ArgoCD proxy was unavailable during validation), so the fix is unconfirmed against KubeFlex v0.9.3's actual runtime behavior. Worth a follow-up comment or issue once someone can watch the rollout and confirm the status-conflict loop is actually gone, per the checklist item added here.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants