tao-run-on-docker

Docker for NVIDIA GPU Workloads

This skill documents the generic Docker conventions that GPU container workloads rely on. Model and data skills specify what image and what command to run; this skill covers how to run docker in a way that satisfies GPU + NVIDIA container requirements.

Sources: official Docker CLI reference (https://docs.docker.com/reference/cli/docker/) and NVIDIA Container Toolkit docs.

Prerequisites

1. Host GPU runtime — NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0. Check with the tao-setup-nvidia-gpu-host skill before any GPU workflow starts.
2. Docker — docker --version must return ≥ 20.10. Install: https://docs.docker.com/engine/install/.
3. NGC API key for nvcr.io/* pulls. Get from https://ngc.nvidia.com/.

TAO_SKILL_BANK_ROOT="${TAO_SKILL_BANK_ROOT:-$PWD}"
SETUP_SCRIPT="${TAO_SKILL_BANK_ROOT}/platform/tao-setup-nvidia-gpu-host/scripts/setup-nvidia-gpu-host.sh"

bash "$SETUP_SCRIPT" --backend docker --check-only || {
  echo "MISSING: TAO GPU host runtime is not ready."
  echo "After user approval, run (append --yes for non-interactive agent runs):"
  echo "  bash \"$SETUP_SCRIPT\" --backend docker --install"
  exit 1
}

docker --version
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
[ -n "$NGC_KEY" ] || echo "NGC_KEY unset — cannot pull nvcr.io images"

NGC authentication

echo "$NGC_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin

Persists in ~/.docker/config.json across reboots. Re-run on unauthorized errors.

docker run — canonical flags

docker run \
  --gpus all \                        # all GPUs (requires nvidia-container-toolkit)
  --rm \                              # delete container after exit (image is preserved)
  --shm-size=8g \                     # shared mem for torchrun / DataLoader
  -v /host/data:/data \               # bind-mount input
  -v /host/results:/results \         # bind-mount output
  -e HF_TOKEN -e NGC_KEY \            # env-var passthrough (values from parent shell)
  <image> \
  <command>

Notes:

• --gpus '"device=0,1"' — specific GPUs (double-quote-escaped). Without nvidia-container-toolkit: could not select device driver "" with capabilities: [[gpu]].
• --rm — clean up the container at exit; omit when you want docker logs after exit.
• --shm-size=8g — torchrun + PyTorch DataLoaders exhaust the default 64 MB /dev/shm otherwise; size it for multi-GPU training and raise (e.g. 16g) if you still hit Bus error.
• -v host:container — bind mount; the command references container paths only.
• -e VAR — passthrough from parent shell (no value needed if already set). Use this form for secrets.

Container name collision

docker run --name X fails if a container named X already exists. Defensive pattern before reusing a name:

docker stop my-worker 2>/dev/null; docker rm my-worker 2>/dev/null
docker run --name my-worker ...

Detached + exec pattern

For multi-step workflows on the same container (download → run → post-process), avoid restart cost:

docker run -d --name <worker> \
  --gpus all --shm-size=8g \
  -v <mounts...> -e <envs...> \
  --entrypoint sh \
  <image> -c "tail -f /dev/null"

docker exec <worker> <step_1>
docker exec <worker> <step_2>

docker stop <worker> && docker rm <worker>

Pull-if-missing idiom

docker image inspect <image> >/dev/null 2>&1 || docker pull <image>

Labels for discovery

Tag containers for filtered listing later:

docker run --label tao-toolkit ...
docker ps --filter 'label=tao-toolkit'

Mount patterns

The container expects its data at conventional paths defined by the image (often /data, /results, /workspace/checkpoints). The host side is arbitrary. The command inside docker run references container paths only.

Env-var conventions

Common passthrough vars for TAO-style workloads (the calling skill declares which it needs):

• NGC_KEY — nvcr.io pulls; some runtimes also read at runtime
• HF_TOKEN — gated HuggingFace model downloads
• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL — S3 I/O inside the container
• WANDB_API_KEY — optional W&B logging

Use -e VAR (no =value) when the var is in the parent shell. Avoid placing secrets on the command line.

Alternative GPU selection: -e NVIDIA_VISIBLE_DEVICES=0,1 (or all) and -e NVIDIA_DRIVER_CAPABILITIES=all instead of --gpus. The --gpus flag is preferred on standard x86 hosts; the env-var form is older and is what runtime=nvidia (Tegra/Jetson) requires.

Container inspection

docker ps                                # running containers only
docker ps -a                             # all containers, including exited
docker ps --filter status=running --format '{{.Names}} {{.Image}}'
docker logs <name_or_id>                 # stdout/stderr
docker logs -f <name_or_id>              # follow (tail -f equivalent)
docker logs --tail 100 <name_or_id>      # last N lines
docker inspect <name_or_id>              # full config, mounts, env, network, state (JSON)
docker inspect --format '{{.State.Status}}' <name_or_id>
docker stats                             # live CPU/mem/network/block I/O
docker stats --no-stream                 # one snapshot, non-interactive

docker inspect is the canonical source of truth for a container's mounts, env, cmd, network, and exit code. Use it to debug why a container isn't behaving as expected.

Image management

docker pull <image>
docker image ls
docker system df                # disk usage
docker system prune -a --volumes # reclaim space — destructive, removes unused images + volumes

Pull once per host; docker run reuses cached image. NVIDIA images are typically 5-40GB.

Split-disk data-root relocation

Some cloud GPU providers ship with a small root volume + larger ephemeral. Docker writes to /var/lib/docker on root by default — large images fill it. Check:

df -h /         # root volume size/free
lsblk           # all block devices and mount points

If / is smaller than your total image footprint and there's a larger disk mounted elsewhere, relocate before pulling images:

sudo systemctl stop docker
sudo mkdir -p <large_volume_path>/docker
sudo rsync -aP /var/lib/docker/ <large_volume_path>/docker/
sudo mv /var/lib/docker /var/lib/docker.old

sudo tee /etc/docker/daemon.json <<'EOF'
{ "data-root": "<large_volume_path>/docker" }
EOF

sudo systemctl start docker
docker info | grep 'Docker Root Dir'
sudo rm -rf /var/lib/docker.old

Networks (multi-container patterns)

For microservice containers that talk to each other by name, create a docker network and attach containers:

docker network create tao-net
docker run --network tao-net --name api ...
docker run --network tao-net --name worker ...   # can resolve `api` by name

Most TAO training workloads don't need this — single container per job.

Common error modes

could not select device driver "" with capabilities: [[gpu]] — NVIDIA Container Toolkit missing or Docker is not configured for the NVIDIA runtime. Run tao-setup-nvidia-gpu-host with --backend docker --install after user approval (append --yes for a non-interactive agent run), then restart Docker.

unauthorized: authentication required on docker pull — NGC key invalid/missing. Re-run docker login nvcr.io.

no space left on device — root volume full. docker system df to inspect; relocate data-root (above) or docker system prune -a --volumes.

Bus error / DataLoader worker exited unexpectedly — /dev/shm too small. Increase shared memory with --shm-size (e.g. --shm-size=16g).

permission denied on bind-mounted paths — container UID ≠ host UID. Either -u $(id -u):$(id -g), or pre-create host files owned by the host user, or chmod 777 (dev only).

Error: No such container: <name> after docker run -d — container crashed on startup. docker ps -a shows exited; docker logs <name> for cause. Drop --rm while debugging.

Scope boundary

This skill covers the how of running docker on a GPU host. Platform-specific layering (how to get onto the host, dispatch via a CLI wrapper) lives in:

• tao-skill-bank:tao-run-on-brev — running docker via brev exec on a Brev instance
• tao-skill-bank:tao-run-platform — optional Python layer wrapping docker invocations with Job handles, state persistence, and S3 I/O

Model and data skills specify what image and command; they defer to this skill for the how.