Containerization & Reproducibility

#Why Containers for ML

A container packages: OS userspace, system libraries (CUDA, cuDNN, NCCL), Python interpreter, packages, your code, and your model artifacts into a single immutable image. Anywhere that runs the container, the environment is identical.

For ML this matters more than for typical web apps because:

• GPU drivers + CUDA + cuDNN versions are tightly coupled
• Floating-point determinism depends on hardware + library versions
• Models are large (GB-scale) and need predictable mounts
• Distributed training requires identical environments across nodes

#ML-Optimized Dockerfile

# Multi-stage build for ML training
# Layer 1: CUDA base
FROM nvidia/cuda:12.4.0-cudnn-devel-ubuntu22.04 AS base

# Layer 2: System deps (rarely changes — caches well)
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 python3-pip git curl wget \
    && rm -rf /var/lib/apt/lists/*

# Layer 3: Python deps (changes when requirements.txt changes)
COPY requirements.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements.txt

# Layer 4: Application code (changes every commit)
WORKDIR /app
COPY . /app

# Layer 5: Entry point
CMD ["python", "train.py"]

Key principles

1. Order layers by change frequency — system deps rarely change (top), code changes often (bottom). This caches well.
2. Use --no-cache-dir with pip to avoid bloating the image.
3. Pin everything — torch==2.5.1, cuda:12.4.0, exact versions everywhere.
4. Use .dockerignore — exclude __pycache__, .git, large datasets.
5. Multi-stage builds for inference — separate "training" image (with full toolchain) from "serving" image (minimal runtime).

#Multi-Stage Builds for ML

# ---------- Stage 1: build/train (heavy toolchain) ----------
FROM nvidia/cuda:12.4.0-cudnn-devel-ubuntu22.04 AS build
RUN apt-get update && apt-get install -y gcc g++ make python3.11 python3-pip git
COPY requirements-train.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements-train.txt
COPY . /src
WORKDIR /src
RUN python train.py --output /artifacts/model.pt

# ---------- Stage 2: serve (lean runtime) ----------
FROM nvidia/cuda:12.4.0-cudnn-runtime-ubuntu22.04 AS serve
RUN apt-get update && apt-get install -y python3.11 python3-pip && rm -rf /var/lib/apt/lists/*
COPY requirements-serve.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements-serve.txt
COPY --from=build /artifacts/model.pt /model/model.pt
COPY serve.py /app/serve.py
CMD ["python3", "/app/serve.py"]

The serving image only contains: CUDA runtime (not the dev SDK), Python, the inference-side packages, the trained model artifact, and a serving script. You drop gcc, the training data loaders, Jupyter, profilers, etc. Typical size differences in 2026:

The size difference matters for two reasons: pull latency on every node (each cold pod has to download the image once), and attack surface (every binary in the image is potentially exploitable). A 7-GB image is ~30s to pull on a 2 Gbps link; a 600-MB image is ~3 seconds.

#Choosing requirements management: requirements.txt vs poetry vs uv vs pixi

The 2024-2026 landscape for Python dependency management exploded with new tools. Here's the practical map:

# 2026 idiomatic: uv-based dependency install
RUN pip install --no-cache-dir uv==0.5.0
COPY requirements.txt /tmp/
RUN uv pip install --system --no-cache-dir -r /tmp/requirements.txt

#GPU Container Runtime: nvidia-container-toolkit

# Install nvidia-container-toolkit on the host (one time):
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo systemctl restart docker

# Now run a container with GPU access:
docker run --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
docker run --gpus '"device=0,1"' my-train-image python train.py  # only GPUs 0 and 1

#Kubernetes Operators for ML

Beyond the GPU Operator, several Operators handle ML-specific concerns:

#Deterministic CI/CD for ML Training

Reproducibility extends to the build pipeline. The minimum bar for a 2026 ML CI/CD pipeline:

1. Pin everything in the Dockerfile — base image SHA, system package versions, Python deps via uv.lock or requirements.txt.
2. Tag images by git SHA — myorg/train:abc123def, not myorg/train:latest. latest is poison; it's the tag that means "I don't know which version".
3. Reproducible layers. Use BUILDKIT_INLINE_CACHE=1 and a remote cache so two CI runs of the same git SHA produce byte-identical images.
4. Sign images — cosign sign after build; verify on deploy. Required by SLSA Level 3 (and the EU AI Act for high-risk systems).
5. Don't bake secrets. Use BuildKit --mount=type=secret for pip indices or model registry tokens; never COPY .npmrc-style secret baking.
6. Promote images, don't rebuild them — the image that runs in staging is the same SHA that goes to prod. Rebuilding "the same Dockerfile" produces a different image because base image SHAs, transitive dep versions, and apt mirrors drift hourly.

# Skeleton GitHub Actions workflow
name: build-ml-image
on: { push: { branches: [main] } }
jobs:
  build:
    runs-on: ubuntu-latest
    permissions: { id-token: write, contents: read }
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} }
      - name: Build & push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}/train:${{ github.sha }}
          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}/cache:train
          cache-to:   type=registry,ref=ghcr.io/${{ github.repository }}/cache:train,mode=max
          provenance: true
      - name: Sign
        run: cosign sign --yes ghcr.io/${{ github.repository }}/train:${{ github.sha }}

#Base Image Selection

#Pinning Strategy

Three levels of pinning:

FROM nvidia/cuda@sha256:abc123def... AS base

#Hands-On

#Compute the Dockerfile Size Difference: CUDA-full vs Slim

Let's quantify the layer-by-layer size of a typical ML Docker image and see exactly where the bytes go. This lets you reason about caching, pull latency, and the cost of multi-stage separation.

The 60-70% size reduction between the training and inference image isn't only about disk: it shows up directly in cold-start latency on every new pod, in attack surface (every binary in the image is a potential CVE), and in deploy time. Once you internalize this, multi-stage builds stop being a "best practice" and become the obvious default.

#Key Takeaways

• Containers solve "works on my machine". Package OS, CUDA, Python, deps, code into one immutable image
• Order layers by change frequency. System deps top, code bottom; caches well
• Pin everything. Base image SHA, exact CUDA version, locked Python deps
• Multi-stage builds keep inference images small. Devel toolchain in training, runtime-only in inference
• Modal / Anyscale / Vertex AI abstract container management — start there for new projects

#Quick Check