Skip to content

fix(podman): harden sandbox callbacks#2111

Open
maxamillion wants to merge 1 commit into
NVIDIA:mainfrom
maxamillion:gateway_callback
Open

fix(podman): harden sandbox callbacks#2111
maxamillion wants to merge 1 commit into
NVIDIA:mainfrom
maxamillion:gateway_callback

Conversation

@maxamillion

Copy link
Copy Markdown
Collaborator

Summary

Fix Podman sandbox callbacks for containerized and rootless pasta deployments. Sandbox JWTs now use Podman secrets instead of host bind-mounted token files, and loopback-bound host gateways can add a protected callback listener for rootless pasta without binding the primary listener to every interface.

Related Issue

Fixes #1909

Changes

  • Deliver per-sandbox gateway JWTs through Podman secrets and clean them up on sandbox deletion/failure.
  • Add bindability-checked callback listeners for direct-host Podman gateways, including rootless pasta loopback binds discovered from local interfaces.
  • Change the RPM default gateway bind address to 127.0.0.1:17670 and document the automatic callback listener behavior.
  • Add enable_auto_callback_listener and OPENSHELL_PODMAN_ENABLE_AUTO_CALLBACK_LISTENER opt-out support.
  • Improve containerized Podman socket diagnostics and suppress misleading host subuid/subgid checks inside gateway containers.

Testing

  • mise run pre-commit passes
  • cargo test -p openshell-driver-podman --lib gateway_bind_address -- --nocapture
  • cargo test -p openshell-server --lib rpm_default_config_parses_and_has_podman_defaults
  • cargo test -p openshell-server podman_auto_callback_listener -- --nocapture
  • cargo test -p openshell-server driver_config -- --nocapture
  • Built Fedora 44 RPMs with packit srpm and mock -r fedora-44-x86_64
  • Validated RPM gateway on Fedora 44 VM with rootless Podman/pasta loopback bind, wildcard bind, and callback-listener opt-out
  • Validated containerized gateway with host rootless Podman socket using ttl.sh/openshell-gateway:dev and ttl.sh/openshell-supervisor:dev

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Deliver per-sandbox JWTs through Podman secrets so containerized gateways do not leak gateway-local paths to the host Podman API.

Add protected, bindability-checked callback listeners for host Podman gateways, including rootless pasta loopback binds discovered from local interfaces. Default RPM gateways to loopback, expose enable_auto_callback_listener for opt-out, and improve containerized socket diagnostics.

Signed-off-by: Adam Miller <admiller@redhat.com>
@maxamillion maxamillion requested review from a team, derekwaynecarr and mrunalp as code owners July 1, 2026 21:17
@copy-pr-bot

copy-pr-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@maxamillion

Copy link
Copy Markdown
Collaborator Author

/ok to test 74eb820

const VOLUME_PREFIX: &str = "openshell-sandbox-";

/// Secret name prefix for per-sandbox gateway JWTs.
const TOKEN_SECRET_PREFIX: &str = "openshell-token-";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do secret names allow "namespacing"? (i.e. would something like token.openshell/ be valid)?

Comment on lines 81 to +101
/// Gateway gRPC endpoint the sandbox connects back to.
///
/// When empty, the driver auto-detects the endpoint using
/// `gateway_port` and `host.containers.internal`.
pub grpc_endpoint: String,
/// Port the gateway server is actually listening on.
///
/// Used by the driver's auto-detection fallback when `grpc_endpoint`
/// is empty. The server must set this to `config.bind_address.port()`
/// so the correct port is used even when `--port` differs from the
/// default. Defaults to [`openshell_core::config::DEFAULT_SERVER_PORT`].
pub gateway_port: u16,
/// Full gateway bind address supplied by the gateway process at runtime.
#[serde(skip)]
pub gateway_bind_address: Option<SocketAddr>,
/// Whether the gateway has TLS enabled. Supplied by the gateway process.
#[serde(skip)]
pub gateway_tls_enabled: bool,
/// Whether gateway callbacks are protected by an auth mechanism.
#[serde(skip)]
pub gateway_callback_auth_enabled: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would assume that these gateway-specific properties are shared across other drivers. Does it make sense to embed them in a struct?

Comment on lines +990 to +991
uid: 0,
gid: 0,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UID=0 and GID=0 are the defaults. Also, does this need to be readable by any other users? (we only specified ro when this was handled as a mount).

@elezar

elezar commented Jul 3, 2026

Copy link
Copy Markdown
Member

@maxamillion, is the use of Podman secrets separate from the callback changes? If so, could we pull the secret changes into a separate PR? They independently fix the containerized gateway token-path problem, while the callback changes affect broader behavior—for example, automatically selecting a host interface and exposing the full gateway service on the additional listener. Separating them would make both changes easier to review and test.

@maxamillion

Copy link
Copy Markdown
Collaborator Author

@elezar yes, but that change is also related to fixes for issue ticket I referenced so I wasn't sure if I should break it out or leave them combined. I'm happy to split them if preferred.

@elezar

elezar commented Jul 3, 2026

Copy link
Copy Markdown
Member

@elezar yes, but that change is also related to fixes for issue ticket I referenced so I wasn't sure if I should break it out or leave them combined. I'm happy to split them if preferred.

I definitely have some thoughts on the callback changes, but don't want to block the secret changes on those, so splitting them out may be cleaner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: containerized Podman gateway broken on Fedora 44 (SELinux, UID mapping, port binding, state path)

2 participants