fix(docker): reconcile duplicate sandbox containers

[ericksoa]

Summary

Enforce an explicit Docker sandbox-instance ownership contract while preserving
NemoClaw's GPU compatibility replacement flow. Matching identity labels are
discovery metadata only; they no longer authorize arbitrary containers to drive
sandbox status.

Related Issue

Related to NemoClaw #6110.

Changes

• Record the authoritative Docker container ID from create or persisted sandbox
  status and restore it before reconciliation starts.
• Add a driver-issued instance generation and owner-only, atomically persisted
  ownership journal for exact replacement intent, restart recovery, and rollback.
• Require two consistent observations before adopting a generation-matching
  canonical replacement. This supports both the overlapping NemoClaw backup
  handoff and a poll-observed/missed gap without deleting durable sandbox state.
• Fail closed for generic copied labels, missing or incorrect generations,
  ambiguous duplicates, interrupted intents, and terminal replacement states.
• Emit deduplicated warning logs and driver platform events for ownership
  conflicts and transitions.
• Stop or delete every exact namespace/ID/name match, complete persisted deletes
  safely at startup, and close create/delete cancellation races.
• Preserve sandbox spec, metadata, policy version, settings, SSH sessions, logs,
  and watch state across missing/conflicting replacement observations.
• Prevent a connected supervisor session from promoting an ownership-pending
  sandbox to Ready before the replacement is authorized.
• Document the ownership, trust, migration, and recovery contract.

Containers created by older drivers without a journaled generation fail closed
for external handoff and must be recreated through OpenShell first.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Focused validation:

• cargo test -p openshell-driver-docker --lib (122 passed)
• cargo test -p openshell-server --lib (887 passed)
• cargo clippy -p openshell-driver-docker -p openshell-server --all-targets -- -D warnings
• Generation, overlap, observed-gap, restart, rollback, cancellation, warning-event,
  startup restoration, and durable-state preservation regressions
• Exact-head Branch Checks
  and Branch E2E Checks passed

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[github-actions]

Label test:e2e-gpu applied for 7ffe7d5. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute GPU E2E after building the required supervisor image once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[github-actions]

Label test:e2e applied for 7ffe7d5. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[elezar]

Blocking: out-of-band container adoption needs an explicit ownership contract

The deterministic duplicate reconciliation fixes the observed NemoClaw overlap, but this change also formalizes a broader ownership rule: any container with matching managed-by, namespace, sandbox ID, and sandbox name labels can become the authoritative sandbox instance. Those labels are copyable discovery metadata, not proof that the driver created or authorized the container.

In the current local, single-player Docker threat model, this is probably not a privilege-escalation vulnerability because Docker daemon access is already effectively host-level authority. It is still a control-plane integrity problem and establishes a problematic contract for future shared deployments.

There is also a timing-dependent failure mode that the duplicate tests do not cover. If an external compatibility flow deletes the original container before launching its replacement:

1. If the replacement appears before the driver's two-second poll, OpenShell treats it as an ordinary update and preserves the existing sandbox spec.
2. If a poll observes the gap, the driver emits Deleted. The gateway deletes the durable sandbox record, SSH sessions, sandbox settings, logs, and watch state.
3. When the replacement later appears with copied identity labels, apply_sandbox_update_locked creates a skeletal record with spec: None, empty metadata labels, and reset policy-version state.

The same logical replacement therefore either succeeds transparently or destructively removes and partially resurrects control-plane state depending only on poll timing.

I am requesting changes until the ownership and replacement behavior is made explicit and tested. I think this needs:

• A clear decision on whether out-of-band containers may be adopted as managed instances. If not, duplicate or unknown matching containers should be reported as ownership conflicts rather than selected by status.
• If external replacement is supported, an explicit driver-coordinated replacement/adoption mechanism with an instance generation or replacement intent. Matching labels alone should not authorize adoption.
• Regression tests for both overlapping replacement and delete-then-recreate, including the case where the polling loop observes the absence. The tests should demonstrate that the durable sandbox spec and owned state are not silently lost.
• An operator-visible warning or platform/security event when duplicate identities or an unexpected replacement are detected.

For the overlapping NemoClaw case specifically, preferring the canonical OpenShell container name before merely preferring RUNNING would be a safer interim rule: the replacement takes the canonical name and the stopped original is renamed to *-nemoclaw-gpu-backup-*. However, preference ordering alone does not solve the delete/recreate race or establish provenance.

Stopping or deleting all exact matches can still be reasonable as conflict cleanup, but that should be kept distinct from trusting every match to drive sandbox status.

[ericksoa]

@elezar, the requested ownership boundary is now implemented on 378674280712766e34ff17d64dace59849608adf and ready for re-review.

The revised contract treats matching Docker labels only as discovery metadata. Authority comes from a private, owner-only, atomically persisted driver journal containing the authoritative container ID, driver generation, retired IDs, and an exact A -> B replacement intent. External replacement is adopted only after two consistent observations of the supported NemoClaw handoff (including delete/recreate poll gaps and restart); ambiguity, churn, copied labels, missing lineage, or interrupted intent fails closed.

The watcher now preserves the durable sandbox/server record through overlap and gap states instead of synthesizing Deleted, emits a deduplicated operator warning and platform event for pending/conflict states, and prevents a supervisor session from promoting those states to Ready. Startup restoration, explicit deletion, stop/delete identity checks, rollback restrictions, and provision/delete cancellation were also tightened so only journal-owned instances are acted on and retained authority is cleaned up deliberately.

Regression coverage includes overlap in both observation orders, observed and missed gaps, restart during intent, interrupted/churned handoffs, copied or legacy labels, ambiguous successors, retired rollback/conflicts, durable-state preservation, event delivery/deduplication, startup restore failures, explicit cleanup, and create/delete races. Documentation now spells out the supported external replacement protocol and recovery path.

Validation on the exact head:

• cargo test -p openshell-driver-docker --lib — 122 passed
• cargo test -p openshell-server --lib — 887 passed
• cargo clippy -p openshell-driver-docker -p openshell-server --all-targets -- -D warnings
• mise run pre-commit — full workspace gate passed
• Exact-head Branch Checks
  and Branch E2E Checks passed
• DCO passed

Please take another look when convenient.