Skip to content

docker-socket-proxy: extend to tes/celery-worker/control-center/launcher/workflow-bundles, and close workflow_runner's nested-DinD gap #54

Description

@man4ish

Follow-up from the #265 docker-socket-proxy fix (see docker-socket-proxy/ in this repo, wired into docker-compose.yml for the workbench service only).

Not yet covered: 5 other services with raw docker.sock RW mounts

Grepped docker-compose.yml for every service mounting /var/run/docker.sock (RW, not counting deploy-verify's existing :ro mount): tes, workbench (now proxied), celery-worker, control-center, launcher, workflow-bundles. Only workbench was migrated to the proxy in the initial PR -- it's the dominant real consumer (omnibioai/plugin_executor/ml_utils.py's ~50 ML plugin executors run there), but the other 5 still get unrestricted raw socket access. Migrating each is mechanically the same pattern (swap the raw bind mount for the docker-proxy-socket named volume + set DOCKER_HOST), but each service needs its own real docker.sock usage audited first (same way workbench's ml_utils.py/three-bind-mount pattern was audited) so PROXY_ALLOWED_BIND_PREFIXES doesn't silently break something real -- not safe to copy-paste without that check per service.

Not yet covered: workflow_runner's nested sibling-container dispatch

plugins/workflow_runner/views.py's _run_in_docker_runner() spawns its own sibling container for Nextflow's Docker-in-Docker executor, and that spawned container itself gets -v /var/run/docker.sock:/var/run/docker.sock -- the RAW host socket, not the proxy's. This happens even when workflow_runner's own outer docker run call is issued through the proxy (from a proxied service): the bind-mount SOURCE path is resolved by the real daemon against ITS OWN filesystem, not the calling container's, so -v /var/run/docker.sock:/var/run/docker.sock in that command always resolves to the real host socket regardless of what socket the outer caller used to submit the request.

Note the proxy's own bind-mount validation (policy.py's check_create_body) already rejects a request that tries to bind-mount /var/run/docker.sock as a source from an arbitrary caller (see test_docker_sock_bind_mount_blocked) -- but workflow_runner's own code constructs exactly that bind-mount deliberately, for its own legitimate Nextflow DinD need, so simply keeping the current allowlist as-is would just break workflow_runner rather than closing the gap.

Real fix needs one of:

  • Route workflow_runner's own bind-mount source through the SAME proxy socket instead of the raw one -- requires the proxy to expose its listening socket at a real, stable HOST filesystem path (not just a Docker named volume, which doesn't have an application-predictable host-side path) so the literal string in views.py's docker_cmd construction can reference it, plus updating that one hardcoded bind-mount source.
  • Or: accept and clearly document that workflow_runner's own dispatched containers are a deliberate, scoped exception to the #265 fix (Nextflow's DinD need is real and legitimate), with the exposure narrowed some other way (e.g. a workflow_runner-specific proxy instance with a much tighter policy than the general one, if that's even sufficient given the same fundamental host-filesystem-resolution issue would apply to IT too).

This needs real design work, not a copy-paste of the current proxy -- filed here rather than left implicit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions