Skip to content

Add Clabernetes runtime support - #3217

Open
FloSch62 wants to merge 46 commits into
mainfrom
c9s
Open

Add Clabernetes runtime support#3217
FloSch62 wants to merge 46 commits into
mainfrom
c9s

Conversation

@FloSch62

@FloSch62 FloSch62 commented Jun 5, 2026

Copy link
Copy Markdown
Member

Summary

  • Add the Clabernetes/c9s runtime path for running containerlab topologies through Kubernetes
  • Support core runtime operations including deploy, inspect, exec, lifecycle actions, destroy, and events
  • Add c9s-compatible Robot coverage for the supported Linux lifecycle flow

@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 7cea67f:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9616259261/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9616180216/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

…etes runtime

Adapt the clabernetes lab runtime to the direct (native kubernetes)
clabernetes architecture and make the Topology resource the primary
deployment path: containerlab persists the rendered topology and the
manager compiles and owns the Node, Link, and NodeProfile resources.

Deploy waits until the controller has observed the Topology's current
generation (status.observedGeneration) and the lab reports ready;
controllers that predate observedGeneration fall back to the readiness
wait alone. The created Topology sets statusProbes.enabled explicitly
because CRD structural defaulting cannot materialize the absent
statusProbes object.

--no-topology-cr on deploy and redeploy keeps the previous behavior:
the topology is compiled client-side and the Node, Link, and
NodeProfile resources are managed directly, with no Topology object
persisted. A Topology-owned lab is rejected when the flag is set, and
deploying a primitive-only lab without the flag hands ownership to the
controller, which adopts the label-matched resources in place.
@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 3e8ce69:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9648400953/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9648295761/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

The rollback of an interrupted deployment ran from the signal handler
goroutine, concurrently with the deploy that was still unwinding, and both
it and main raced to os.Exit(1). On the clabernetes runtime main always won
because the deploy returns from its wait loop the instant the context is
canceled, so the process died before the destroy had issued a single
request and the lab was left behind in full: namespace, Topology, Nodes,
pods, services and staged ConfigMaps.

Give the rollback to the command that created the lab instead. deployFn now
tears down what it managed to create before surfacing the cancellation, so
nothing races the process exit and the destroy no longer runs against a
deploy that is still creating resources. The signal handler keeps exiting
immediately when there is nothing to roll back, which preserves the
behaviour for reconciliations and every non-deploy command, and a second
signal aborts a cleanup that hangs or that the user does not want.

The rollback deadline is now runtime aware: controller-driven runtimes wait
for the cluster to converge on the deletion, which legitimately outlasts
the bound that is generous for a local teardown.

CleanOnCancel becomes an atomic.Bool: it is written by deploy and read from
the signal handler goroutine, and it now decides whether the process waits
for a cleanup.

As a side effect a signal arriving after a deploy has already succeeded no
longer destroys the finished lab.
@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit e04e9ce:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9649038778/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9649010166/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 6a70892:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9712890496/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9712889260/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 1ab005c:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9713006343/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9713002934/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@hellt

hellt commented Aug 29, 2026

Copy link
Copy Markdown
Member

TODO:

  • required direct helper is not ready unclear message

@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 2caa997:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9737082449/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9737077308/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@FloSch62

Copy link
Copy Markdown
Member Author

Field report from a Talos cluster (public images only, no pull secret needed): every clab deploy -r c9s of a plain public-image lab fails out of the box because of the implicit regcred pull secret.

What happens:

  • labruntime/runtime.go sets DefaultImagePullSecret = "regcred" and topologyWithImagePullSecret (labruntime/clabernetes/topology.go) falls back to it even for an explicitly empty value, so every Topology is stamped with spec.imagePull.pullSecrets: [regcred].
  • The clabernetes direct-mode manager fails closed when a named pull Secret is absent (resolving image pull Secret <ns>/regcred: secrets "regcred" not found), and today that failure sets no Node condition and no event (Missing image pull Secret stalls Node reconciliation with a completely empty status (no condition, no event) clabernetes/clabernetes#323), so the deploy just shows pending nodes: n1 (unknown) until the timeout.
  • Pre-creating the Secret is not possible in the default flow: the managed c9s-<lab> namespace is created by the same deploy. Only the --namespace path (pre-created namespace) allows it.

Verified on a live cluster with this branch @2caa997: 1-node lab with ghcr.io/srl-labs/network-multitool, no regcred Secret, stalls silently until the deploy timeout.

Proposal:

  • Only set spec.imagePull on the Topology when --image-pull-secret was explicitly passed; unset/empty means no pull secret at all (public images, or clusters whose CRI already has credentials/mirrors).

  • Drop the DefaultImagePullSecret fallback in topologyWithImagePullSecret; keep regcred at most as a documented convention in docs/manual/clabernetes/runtime.md.

  • no implicit regcred: omit imagePull unless --image-pull-secret is explicitly given

Related clabernetes-side issue from the same debugging session: clabernetes/clabernetes#322 (OCI metadata resolution cannot follow CRI registry mirrors on airgapped clusters).

…sted

An unset --image-pull-secret no longer falls back to an implicit "regcred":
the Topology (and the --no-topology-cr NodeProfile) carries no pull secret
reference at all, and a reconcile clears a previously stamped default. The
deploy wait now also aborts on the manager's ImagePullSecret* plan failures
with the same debounce as registry metadata failures instead of hanging
until the lab timeout.
@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 20fcb05:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9768460597/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9768451010/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 963aa4d:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9863324260/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9863262389/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit e38f8fb:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9886240718/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9886229109/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

Local files a lab mounts into its nodes (startup configs, binds, env files,
licenses) were always staged as ConfigMap binaryData, which Kubernetes
serializes as base64. That kept every staged file unreadable in the cluster
and in any exported manifest.

Text files (valid UTF-8 without NUL bytes) now land in data as plain strings;
only genuinely binary content stays in binaryData. The c9s node controller
reads either field, and reconciling an existing lab rewrites its ConfigMaps
in place while keeping their owner references.
`deploy --emit-crs` renders the resources a c9s deployment would create and
writes them to stdout as a multi-document YAML stream (or a v1 List with
--format json) without contacting the cluster, so they can be saved, edited
by hand, and applied with kubectl.

The bundle follows the runtime's apply order: the managed lab Namespace
(omitted when a namespace override is in effect), the staged ConfigMaps, and
then the Topology or, with --no-topology-cr, the compiled NodeProfile, Link,
and Node resources. The other c9s deploy flags shape the emitted objects
exactly as they shape a real deployment, and every object carries the labels
containerlab uses to discover a lab, so an applied bundle stays manageable
with inspect, exec, and destroy.

The emitter is a ManifestEmitter implemented by the c9s runtime on top of the
same preparation step deploy uses; the Namespace and ConfigMap constructors
are shared with the apply path so emitted and applied objects are identical.
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit de83aab:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9892339308/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/9892325197/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 55ff224:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10110862205/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10110757623/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

Comment thread labruntime/clabernetes/lifecycle.go Fixed
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 208cd5b:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10111400587/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10111392665/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit cca4dd9:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10111671154/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10111659057/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit e0830fb:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10113644220/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10113629035/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@github-actions

Copy link
Copy Markdown
Contributor
Download Containerlab binaries for commit 8c9679a:

containerlab (linux/amd64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10149367389/zip" \
  -o containerlab.zip && unzip -p containerlab.zip containerlab > containerlab.tmp && \
  mv containerlab.tmp containerlab && rm containerlab.zip && chmod +x containerlab

containerlab-arm64 (linux/arm64)

curl -sL \
  -H "Authorization: Bearer $(gh auth token)" \
  "https://api.github.com/repos/srl-labs/containerlab/actions/artifacts/10149358527/zip" \
  -o containerlab-arm.zip && unzip -p containerlab-arm.zip containerlab > containerlab-arm.tmp && \
  mv containerlab-arm.tmp containerlab && rm containerlab-arm.zip && chmod +x containerlab

@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 47.20000% with 2442 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.06%. Comparing base (45ad882) to head (8c9679a).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
core/labruntime.go 25.11% 287 Missing and 26 partials ⚠️
labruntime/clabernetes/save.go 14.05% 213 Missing and 1 partial ⚠️
labruntime/clabernetes/nodes.go 15.51% 199 Missing and 8 partials ⚠️
labruntime/clabernetes/files.go 62.94% 158 Missing and 48 partials ⚠️
labruntime/clabernetes/lifecycle.go 59.16% 147 Missing and 49 partials ⚠️
labruntime/clabernetes/events.go 20.28% 163 Missing and 2 partials ⚠️
labruntime/clabernetes/exec.go 19.33% 144 Missing and 2 partials ⚠️
labruntime/clabernetes/iface_stats.go 14.54% 136 Missing and 5 partials ⚠️
labruntime/clabernetes/resources.go 39.53% 93 Missing and 11 partials ⚠️
labruntime/clabernetes/reconcile.go 69.52% 74 Missing and 22 partials ⚠️
... and 36 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3217      +/-   ##
==========================================
- Coverage   59.41%   58.06%   -1.35%     
==========================================
  Files         252      277      +25     
  Lines       27126    31967    +4841     
==========================================
+ Hits        16116    18562    +2446     
- Misses       9249    11339    +2090     
- Partials     1761     2066     +305     
Files with missing lines Coverage Δ
cmd/options.go 90.68% <100.00%> (+0.09%) ⬆️
core/file.go 50.51% <100.00%> (+0.51%) ⬆️
labruntime/names.go 100.00% <100.00%> (ø)
runtime/runtime.go 72.58% <ø> (+1.89%) ⬆️
cmd/destroy.go 92.30% <0.00%> (-2.50%) ⬇️
cmd/events.go 96.36% <0.00%> (-3.64%) ⬇️
cmd/exec.go 83.60% <77.77%> (-0.27%) ⬇️
cmd/restart.go 76.92% <0.00%> (-7.08%) ⬇️
cmd/start.go 76.92% <0.00%> (-7.08%) ⬇️
cmd/stop.go 76.92% <0.00%> (-7.08%) ⬇️
... and 40 more

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants