Define how Kubernetes edge devices can become targetable via Margo conformant applications

[ajcraig]

Feature description

Goal: Define the requirements and definitions that a Helm/Kubernetes deployment specification needs to enable applications to target Kubernetes environments within the Margo ecosystem. Rather than defining "leader" and "worker" as separate certified roles, this feature centers on schema-agnostic capability discovery and matching: a cluster reports its topology, node roles, and node/GPU capabilities, and Margo matches deployments against those reported capabilities.

Boundary: the device's edge Kubernetes orchestrator (kube-apiserver, scheduler, controller-manager, kubelet, or an equivalent distribution) remains solely responsible for scheduling, placement, node health, and cluster lifecycle..

• This functionality will be tackled via the Device Management contributor working group.

Note: This feature is limited to vendor-homogeneous clusters. Multi-vendor Kubernetes cluster support may come in a future TWG feature.

Provide adequate technical acceptance criteria(s) associated with this feature below:

• Define the requirements and definitions needed in a Helm/Kubernetes deployment specification to enable Kubernetes device and/or platform providers to become devices within the Margo ecosystem.
• Ensure cluster capabilities are standardized and can be reported to the Workload fleet manager
  • Report topology and node capabilities so that deployment eligibility can be based on matching rather than role assumptions.
• Ensure a Helm/Kubernetes deployment specification can describe the capabilities required for a deployment to function within enrolled clusters.
• Coordinate operator and CRD deployment and management requirements with Issue 105.
• Clarify requirements for Kubernetes enabled device(s) to be Margo conformant, current list is shown below:
  • Current leader requirements (Clarify with SUP solution)
    • Margo workload management client
    • OTEL Collector functioning for all nodes in the cluster
    • Kubernetes core components, or an equivalent service, that enable cluster leader functionality: API server, scheduler, controller manager
    • Helm client to enable local rendering and deployments
  • Cluster worker requirements (if applicable):
    • Kubernetes kubelet service
    • OCI container runtime
• Clarify how device capabilities and the Helm/Kubernetes deployment specification can enable workloads to be assigned to the appropriate nodes required by the application.
  • Ensure this is not stretching the local scheduler's responsibilities to the WFM, but enabling the WFM to understand which resources are already assigned vs available.
  • Rely on native Kubernetes scheduling primitives, such as topologySpreadConstraints, for replica placement and topology spread rather than duplicating that scheduling semantics in Margo created descriptors.
    • Ensure we can enable an application to describe and be paired with a cluster that can enable it's workloads to be spread across multiple nodes.
• Define the Kubernetes API version matching requirements and restrictions between the cluster leader and cluster worker(s).
  • Or whether this is irrelevant in the current specification construction. Could be pushed to device management scope.
• Provide best-practice guidance for building Margo-friendly Helm charts, including topology-spread recommendations and capability declaration patterns, alongside any new specification fields.

Out of scope for this feature:

Margo reports and matches; it does not manage or schedule cluster components.

• Node-specific policies, such as node-affinity rules that require the Helm chart to know node labels.
• Duplicating Kubernetes scheduling semantics, such as replica spread, affinity, and skew, into Margo descriptors.
• Expanding Margo into platform-management tooling territory already covered by tools such as OpenShift or RHEL; that scope belongs to a separate "Device/Platform Management" effort.

Although not required, it is highly encouraged to provide feature use-cases below:

• Enables user applications to take advantage of multi-node cluster features:
  • Scalable applications
  • Resilient/redundant applications
• Enables end users to scale their cluster compute capacity while operations continue.
• Enables redundant storage capabilities within the cluster.
• Achieves high availability today: two replicas spread across separate nodes via topologySpreadConstraints in the Helm chart, so a multi-node device spreads replicas while a single-node device keeps them scaled down, all without the chart knowing about specific nodes.

Additional information

The Helm/Kubernetes deployment specification should use schema-agnostic capability matching:

1. A cluster reports its capabilities, including topology and node/GPU capabilities, to the workload fleet manager (WFM).
2. A deployment declares the capabilities it requires.
3. If a required capability is missing, the deployment is incompatible and the device is not eligible.
4. When the cluster re-reports matching capabilities, the device becomes eligible and the WFM can place the workload.