Add Docker workflows for GPU training, serving and offline deployment - #25
Open
KraHsu wants to merge 14 commits into
Open
Add Docker workflows for GPU training, serving and offline deployment#25KraHsu wants to merge 14 commits into
KraHsu wants to merge 14 commits into
Conversation
Provide a locked CUDA 12.8 image, runtime UID mapping, persistent caches and outputs, GPU smoke checks, and serving/training/development/worktree configurations. Preserve training signal delivery and configured inference endpoints.
Resolve Make image selection through the effective Compose configuration and export versioned bundles with image-bound configuration and integrity checks. Document complete online/offline workflows and cover image overrides, mounts, worktrees, runtime identities and bundle delivery.
Build and load the linux/amd64 image in GitHub Actions, validate Compose profiles, run container CPU checks and exercise Docker integration workflows. Include Docker tooling in the standard lint job.
Record the offline H100 inference and training/resume acceptance, subsequent runtime and Compose checks, and the validation limits for each tested image revision.
Docker inspect defaults an untagged repository to latest, while Docker save archives every tag in that repository. Resolve the default tag before saving and record the same explicit reference in the bundle manifest and environment. Preserve explicit tags, digests and image ID inputs, including registry ports. Extend the real offline round-trip test with two distinct images in one local repository and inspect the saved tar to ensure only the selected tag is present. Cover reference forms in the CLI tests and document the default tag behavior. Validation: the new integration regression fails before the fix (2 images instead of 1). The rebuilt image passes make docker-check and make docker-integration-check: 1970 CPU tests and 9 Docker integration tests.
Bind digest and image-ID exports to a content-derived local tag, preserve the source reference, and refuse conflicting delivery tags. Verify a real digest pull, export, offline load, and Compose creation using a temporary loopback registry; cover stable tags and collisions in CLI tests.
Put a private per-container compatibility directory on the image's library search path. The non-root bootstrap atomically selects the opt-in CUDA libraries there, so the entrypoint, docker exec, and restarted processes use the same loader configuration. Leave the directory inactive by default and report missing compatibility libraries before starting the requested command. Document recreation after changing the setting. The real regression checks enabled and disabled modes, arbitrary UID/GID, exec, restart, and missing libraries; it fails against the previous image at the exec comparison. Validation: the rebuilt CUDA image passes make docker-check and make docker-integration-check (1972 CPU tests and 11 integration tests), including the separate digest bundle fix and offline Compose creation.
Group Compose overlays, Make targets, build exclusions and tests under docker/. Consolidate four guides into usage and revision-scoped validation documents. Preserve the root Compose entrypoint and schema 2 offline compatibility while adopting the schema 3 bundle layout. Clarify upgrade and GPU-shell training workflows, reject malformed manifests cleanly, isolate test configuration and keep help available with legacy image settings. Validation includes 49 current tool tests and 12 Docker integration checks; the preceding review image passed 1,984 CPU tests.
Replace the standalone lock script with make docker-lock, preserving dependency versions and selecting the effective Compose image. Update documentation and cover image selection and upgrade arguments in the existing integration test.
Keep the tested GPU revision, key results, resource measurements and limitations alongside the later CPU review results. Remove repeated iteration logs and reproduction instructions already covered by the usage guide.
d-finite
reviewed
Sep 12, 2026
d-finite
left a comment
d-finite
left a comment
Member
There was a problem hiding this comment.
Review summary
I reviewed PR #25 from an isolated worktree and checked the Docker, Compose, offline bundle, CI, runtime-user, and documentation changes.
Validation
- Full non-GPU suite in the isolated environment: 2029 passed, 12 skipped, 22 deselected.
- Docker tool tests: 49 passed.
- Ruff, Python compilation, and Bash syntax checks passed.
- The failed run
34624371754was traced totests/dataloader/test_exclusion_io.py: the lock-contender subprocess imported the full OpenWAM dataloader stack and did not create itsreadymarker within the five-second startup window. The Docker image build itself succeeded and 1,989 other tests passed. - Commit
eb087aafixes that race by loading onlyexclusion_io.pyin the contender and using a 30-second, failure-diagnostic wait. The targeted regression test and Ruff pass locally; the latest standard GitHub tests are green.
Non-blocking findings
-
Offline bundle verification can accept stale extra Compose files.
docker/offline.pyverifies the manifest-listed files but does not reject unexpected files such as an oldcompose.override.yamlor.env. Runningdocker compose upin a reused bundle directory can therefore apply configuration outside the verified bundle. The guide warns users to use a fresh directory, but the importer could enforce this more safely by rejecting unexpected Compose overrides or requiring an explicit compose file. -
The Debian packages in
docker/Dockerfileare not version-pinned. CUDA and Python dependencies are pinned, butapt-get install python3 ...resolves current repository versions. Rebuilding the same source later can therefore produce a different system layer. A Debian snapshot or recorded package versions would improve reproducibility.
Recommendation
I found no blocking execution defect in the PR after the eb087aa test fix. The Docker layout and task-oriented guide are clear and maintainable. The two items above are follow-up improvements rather than merge blockers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenWAM currently requires a manually prepared Python/CUDA environment. This adds a pinned CUDA 12.8, Python 3.12 and PyTorch 2.7.1 image for serving, training and source development, plus offline delivery to GPU hosts without internet access.
Changes
.env, environment variables and Compose overrides.docker execand after container restarts.:latest; digest and image-ID selections receive content-derived delivery tags that survive offline loading. Weights and datasets are transferred separately.Validation
openwam:digest-compat-fixes.make docker-check: dependency, serve CLI and Ruff checks passed; 1,972 CPU tests passed, 12 skipped and 21 deselected.make docker-integration-check PYTHON=python3: 11 tests passed, including digest export/load after shutting down the source registry, Compose creation from the delivered bundle, worktree Git operations, arbitrary runtime IDs and CUDA library selection acrossrun,execand restart.Earlier H100 acceptance covers released-checkpoint inference and four-GPU training/recovery through steps 1–20. That GPU run predates the final runtime/export fixes; current-head GPU training, multi-node/RDMA and optional Cosmos GPU behavior have not been revalidated.