Installing NVIDIA Userspace Drivers in an LXC Container: A Critical Step in GPU Cluster Provisioning

In message [msg 8533] of a sprawling opencode session dedicated to deploying a DFlash training pipeline on a newly provisioned Proxmox host called kpro6, the assistant executes what appears at first glance to be a routine command: copying an NVIDIA driver installer into an LXC container and running it. But this message represents a pivotal moment in the infrastructure setup—the bridge between having raw GPU device nodes visible inside a container and having a fully functional GPU compute environment capable of running PyTorch, flash-attn, and the demanding DFlash training workload. Understanding why this particular approach was chosen, what assumptions it rests on, and what it reveals about the architecture of GPU-accelerated containers is essential to appreciating the craftsmanship of modern ML infrastructure engineering.

The Message in Full

The assistant writes:

Container is up with networking. Now install the NVIDIA userspace inside the container. I'll copy the .run file in and run it with --no-kernel-modules:

>

``bash ssh -o ConnectTimeout=10 root@10.1.2.6 ' # Copy NVIDIA .run into container rootfs cp /root/NVIDIA-Linux-x86_64-595.71.05-no-compat32.run /scratch/containers/subvol-200-disk-0/root/ # Install prerequisites and run NVIDIA installer inside container pct exec 200 -- bash -c " apt-get update -qq && apt-get install -y -qq kmod 2>&1 | tail -3 chmod +x /root/NVIDIA-Linux-x86_64-595.71.05-no-compat32.run /root/NVIDIA-Linux-x86_64-595.71.05-no-compat32.run --no-kernel-modules --silent --no-questions 2>&1 echo exit=\$? nvidia-smi -L 2>&1 " ' 2>&1 ``

>

The output confirms the installation proceeded: Setting up kmod, Verifying archive integrity... OK, and Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 595.71.05 followed by a long ellipsis indicating the decompression was underway.

The message is concise, but every element carries deliberate engineering intent.

The Broader Mission: Why This Message Exists

To understand this message, one must understand what came before it. The session is part of a larger effort to provision kpro6—a Proxmox host equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs—for production DFlash training. The preceding messages (see [msg 8519] through [msg 8532]) document a meticulous process: downloading an Ubuntu 24.04 LXC template, creating container CT 200 with 480 GB RAM and 64 cores, configuring static networking, and passing through all eight GPU devices via LXC mount entries. By message [msg 8533], the container has networking, the GPU device nodes (/dev/nvidia0 through /dev/nvidia7, /dev/nvidia-uvm, etc.) are visible inside the container, and the assistant has reached the critical inflection point: making those devices actually usable.

The GPU device nodes are necessary but not sufficient for GPU compute. The NVIDIA userspace driver stack—libraries like libcuda.so, libnvidia-ml.so, and the CUDA driver API—must be installed inside the container. Without them, PyTorch cannot call cudaSetDevice(), nvidia-smi returns nothing, and the entire training pipeline is dead on arrival. This message is the moment the container becomes a real GPU compute node.

Technical Decisions and Their Rationale

Why --no-kernel-modules?

The single most important flag in this message is --no-kernel-modules. In a standard bare-metal installation, the NVIDIA .run installer would attempt to build and install kernel modules (nvidia.ko, nvidia-uvm.ko, etc.) that interface with the Linux kernel. But inside an LXC container, this is both impossible and unnecessary. The container shares the host's kernel—the kernel modules are already loaded on the Proxmox host (kpro6 runs kernel 6.14.11-9-bpo12-pve with NVIDIA 595.71.05 open drivers compiled from source, as documented in segment 49). The container only needs the userspace libraries that communicate with those host-side kernel modules via the device files.

Using --no-kernel-modules is the correct approach for any containerized GPU deployment. A mistake here—omitting the flag or attempting a full installation—would fail with kernel-headers errors or, worse, attempt to load conflicting modules. The assistant's explicit mention of this flag in the natural language preamble ("I'll copy the .run file in and run it with --no-kernel-modules") shows deliberate awareness of this architectural constraint.

Why Copy the .run File Instead of Bind-Mounting?

The assistant copies the 302 MB installer directly into the container's rootfs at /scratch/containers/subvol-200-disk-0/root/. An alternative approach would be to bind-mount the file from the host filesystem into the container, avoiding the copy. The choice to copy is pragmatic: the installer is needed only once during setup, disk space on the 1 TB rootfs is plentiful, and copying avoids the complexity of managing bind-mount entries in the LXC config for a temporary file. The copy is a one-time cost that simplifies the overall procedure.

Why Install kmod?

The apt-get install -y -qq kmod command installs the kmod package, which provides utilities like lsmod, modprobe, and modinfo. While the container won't load kernel modules, the NVIDIA installer's post-install scripts may call these utilities to check module status or trigger module loading. Pre-installing kmod prevents spurious errors during the installation. The tail -3 piped to the output is a minor aesthetic choice to keep logs concise, though the resulting "Broken pipe" error from tail (visible in the output) is harmless—it occurs because the apt-get output was shorter than the three lines tail tried to read.

The Silent Installation Flags

The combination --silent --no-questions ensures fully unattended installation. In an automated provisioning script—which this effectively is, even though executed interactively—any prompt would block progress. The --silent flag suppresses progress bars and verbose output, while --no-questions accepts all defaults. The echo exit=$? afterward is a defensive check: if the installer fails, the exit code will be visible in the output, allowing the assistant to detect failure and react in a subsequent round.

Assumptions Embedded in This Message

Every engineering decision rests on assumptions, and this message reveals several:

  1. The host kernel modules are already loaded and functional. This is a safe assumption given the preceding segment's work (segment 49) where the assistant built the custom 6.14 kernel and compiled NVIDIA 595.71.05 open drivers from source, verified all 8 GPUs were recognized, and confirmed the system was stable.
  2. The driver version inside the container must match the host kernel module version. The assistant copies the exact same .run file (595.71.05-no-compat32) that was used on the host. Mismatched versions would cause libcuda.so version conflicts, leading to "CUDA driver version is insufficient" errors at runtime.
  3. The container has sufficient disk space for the installation. The NVIDIA userspace installation consumes several hundred megabytes in /usr/lib, /usr/lib32, and /usr/share. The container's 1 TB rootfs on the scratch ZFS pool makes this trivially safe.
  4. The pct exec mechanism provides a reliable execution environment. The assistant chains multiple commands inside a single bash -c invocation, which is more robust than separate pct exec calls because it avoids race conditions and ensures atomicity of the installation sequence.
  5. The nvidia-smi -L command at the end will succeed if the installation is correct. This is the standard litmus test for GPU visibility. If nvidia-smi lists all 8 GPUs, the userspace stack is working.

Potential Issues and Subtle Risks

The output shown in the message is truncated—it ends with "Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 595.71.05" followed by ellipsis dots, without showing the echo exit=$? result or the nvidia-smi -L output. This could mean several things: the SSH session timed out during the lengthy decompression, the output was captured before the command completed, or the output was clipped by the tool's response size limit. The assistant would need to verify completion in a subsequent message (which indeed happens in later messages in the session).

The "Broken pipe" error from tail -3 is benign but worth noting: it indicates that apt-get finished producing output before tail had consumed three lines, causing tail to exit with a broken pipe when apt-get closed its stdout. This is a cosmetic artifact of piping a short output stream through tail -3.

A more subtle risk is that the NVIDIA installer's --no-kernel-modules flag might skip installing certain components that the container actually needs. For example, the NVIDIA Persistence Daemon (nvidia-persistenced) or the fabric manager for NVLink might not be installed. For the DFlash training use case—which primarily needs CUDA compute and NCCL communication—the standard userspace libraries are sufficient, but this assumption should be validated.

What This Message Teaches About Containerized GPU Infrastructure

This message is a microcosm of the broader challenge of GPU-accelerated containerization. The fundamental insight is that GPU acceleration in containers is a split-stack architecture: the kernel modules live on the host, the userspace libraries live in the container, and they communicate through device files passed through via cgroup device allowances and bind mounts. Getting this split right requires understanding which layer owns which responsibility.

The assistant's approach demonstrates several best practices:

Conclusion

Message [msg 8533] is a deceptively simple step in a complex infrastructure journey. It represents the moment a container transitions from having GPU devices to being a GPU compute node. The assistant's decisions—using --no-kernel-modules, copying the installer rather than bind-mounting, pre-installing kmod, and chaining verification commands—reflect a deep understanding of the split-stack GPU architecture in containers. While the output is truncated and a minor cosmetic error appears from the tail pipe, the approach is sound and follows established patterns for containerized GPU provisioning. This message, read in context, is a textbook example of how to correctly install NVIDIA userspace drivers in an LXC container for high-performance ML training workloads.