The Moment of Truth: Verifying GPU Access in an LXC Container

A Single nvidia-smi Command That Held the Weight of an Entire Architecture Decision

In any systems engineering effort, there are moments when a single command serves as a decisive checkpoint — a pass/fail gate that determines whether an entire approach is viable. Message 472 in this opencode session is precisely such a moment. The assistant executes a straightforward nvidia-smi command inside an LXC container on a Proxmox host, and the output appears to show success: all 8 NVIDIA RTX PRO 6000 Blackwell GPUs are detected, the driver version 590.48.01 is loaded, and CUDA 13.1 is reported as available. On the surface, this message looks like a routine verification step. But understanding why this message was written, what it was meant to prove, and what it didn't reveal requires unpacking the complex infrastructure context that led to this point.

Why This Message Was Written: The P2P Bottleneck Crisis

The assistant and user had been engaged in an extended battle against a fundamental hardware limitation. The system in question was a Proxmox VE host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, intended for large-scale ML inference workloads using the GLM-5-NVFP4 model. The original deployment strategy used a KVM virtual machine with VFIO passthrough to give the VM direct access to all 8 GPUs. This worked for basic GPU compute, but a critical problem emerged: the GPU topology inside the VM showed PHB (PCIe Host Bridge) connectivity between all GPUs, rather than the NODE (direct P2P) and SYS (system interconnect) topology seen on bare metal.

This distinction matters enormously for performance. When GPUs communicate via P2P DMA (direct peer-to-peer over PCIe), they bypass the CPU and system memory, achieving dramatically higher bandwidth for operations like tensor parallelism in large model inference. The PHB topology meant every cross-GPU communication had to go through the host bridge, effectively funneling through a single shared bottleneck. For an 8-GPU deployment running a model that requires tight GPU-to-GPU coordination, this was potentially catastrophic for performance.

The assistant had spent significant effort diagnosing this issue, exploring options like ACS (Access Control Service) disable, IOMMU group merging, and kernel parameter tuning — all to no avail. The root cause was that each GPU was on its own PCIe root complex, and VFIO's isolation boundaries prevented P2P across root complexes. The LXC container approach emerged as a potential escape hatch: by running the ML workload in a container that shares the host's kernel and driver stack directly, rather than through VFIO, the GPUs would retain their native PCIe topology.

The Message: A Deceptively Simple Verification

The message itself is terse. The assistant runs:

ssh root@10.1.230.174 "nvidia-smi 2>&1"

And receives the standard nvidia-smi header output showing all GPUs detected. The output is truncated in the conversation (shown with ...), but the critical information is visible: driver version 590.48.01, CUDA Version 13.1, and the implication that all 8 GPUs are present.

This message is the culmination of a long chain of preparatory work:

1. Installing the NVIDIA driver on the Proxmox host (messages 443-451), which required blacklisting the open-source nouveau driver, downloading the 397MB driver package, and installing it with DKMS support.
2. Configuring the LXC container (messages 453-457), which involved converting the container from unprivileged to privileged mode, adding lxc.cgroup2.devices.allow entries for the NVIDIA device major numbers (195 for nvidia, 504 for nvidia-uvm, 507 for nvidia-caps), and bind-mounting all 8 GPU device nodes plus the control and UVM devices.
3. Fixing ownership issues (messages 459-463) caused by the unprivileged-to-privileged conversion, where files in the ZFS subvolume had uid 100000 offsets that needed to be shifted back to root ownership.
4. Installing the NVIDIA userspace driver inside the container (messages 468-471) using the --no-kernel-module flag, since the kernel module is provided by the host. Each of these steps was necessary for this single nvidia-smi command to succeed. The assistant was effectively testing whether the entire LXC approach was viable — whether a container could see and interact with all 8 GPUs with the correct driver stack.

The Thinking Process: What the Assistant Was Trying to Prove

The assistant's reasoning at this point can be reconstructed from the surrounding context. The core hypothesis was: if the LXC container shares the host's kernel and sees the GPUs through bind-mounted device nodes rather than VFIO, then the GPU topology should match bare metal, and P2P DMA should work.

The nvidia-smi command was the first of two critical tests. The second test, which immediately follows in message 473, is nvidia-smi topo -m to verify the topology matrix. The assistant's thinking was structured as a decision tree:

• If nvidia-smi fails → the LXC approach is dead; the driver installation or device mounts are broken.
• If nvidia-smi succeeds but topology shows PHB → the container approach didn't help; the VFIO/IOMMU bottleneck persists even in LXC.
• If nvidia-smi succeeds and topology shows NODE/SYS → the LXC approach works for topology; proceed to install the ML stack and benchmark. The message is therefore a gate. The assistant cannot proceed to the topology check without first confirming basic GPU visibility. And the topology check (message 473) does indeed show the desired NODE and SYS topology — a major victory.

Assumptions Made

Several assumptions underpin this message:

1. That nvidia-smi success implies CUDA runtime success. This is the most consequential assumption, and it turns out to be incorrect. nvidia-smi uses the NVIDIA kernel module interface directly, while CUDA runtime initialization (cuInit) requires additional userspace-kernel interactions including GSP (GPU System Processor) firmware loading. As the chunk summary reveals, cuInit later fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED) despite nvidia-smi working perfectly.
2. That the container's kernel version (6.8.12-9-pve) is compatible with the Blackwell GPU architecture. The Proxmox VE kernel is based on the 6.8 Linux kernel, which may lack proper support for Blackwell's GSP firmware requirements. The driver 590.48.01 ships GSP firmware files only for ga10x (Ampere) and tu10x (Turing) architectures, not for Blackwell.
3. That the --no-kernel-module install flag is sufficient for full CUDA functionality. This flag skips kernel module compilation, relying on the host's pre-installed modules. While this works for basic device access, it may miss architecture-specific kernel module features needed for Blackwell.
4. That the SSH connectivity and container networking are stable. The assistant connects to the container via its IP address (10.1.230.174), which required fixing SSH key ownership issues after the unprivileged-to-privileged conversion.

Input Knowledge Required

To fully understand this message, the reader needs:

• Understanding of GPU virtualization approaches: The difference between VFIO passthrough (KVM), LXC bind-mounts, and bare-metal GPU access.
• Knowledge of nvidia-smi output format: What the driver version, CUDA version, and GPU listing signify.
• Awareness of the P2P DMA problem: Why PHB vs NODE topology matters for multi-GPU ML workloads.
• Familiarity with Proxmox VE concepts: LXC containers, ZFS subvolumes, device cgroup permissions, and the unprivileged-to-privileged conversion process.
• Understanding of the NVIDIA driver stack: The distinction between the kernel module (nvidia.ko) and the userspace driver (libcuda.so, nvidia-smi), and how nvidia-smi can work even when CUDA runtime fails.

Output Knowledge Created

This message creates several important pieces of knowledge:

1. Confirmation that the LXC container can see all 8 GPUs. The device bind-mounts and cgroup permissions are correctly configured.
2. Confirmation that the NVIDIA userspace driver installed correctly inside the container. The --no-kernel-module install succeeded.
3. Evidence that the host's NVIDIA driver installation is functional. The kernel module loaded and is responding to queries.
4. A false positive signal. The message creates the impression that the LXC approach is fully working, which is later contradicted by the CUDA initialization failure. This makes it a critical inflection point: the assistant and user celebrate a victory that is later revealed to be incomplete.

The Hidden Trap: Why nvidia-smi Is Not Enough

The most instructive aspect of this message is what it doesn't say. The nvidia-smi output truncates the GPU list with ..., but even the full output would only show that the kernel module can enumerate devices and report their properties. CUDA runtime initialization is a more complex process that involves:

• Loading GSP firmware onto the GPU's embedded processor
• Initializing the UVM (Unified Virtual Memory) driver
• Setting up memory management contexts
• Negotiating driver-GPU capabilities For Blackwell GPUs, the GSP firmware requirement appears to be stricter than for previous generations. The Proxmox VE kernel (6.8.12-9-pve) may lack the necessary GSP support infrastructure, or the driver 590.48.01 may not include the required Blackwell-specific firmware blobs. This creates a situation where the kernel module can talk to the GPU at a basic level (enough for nvidia-smi), but the CUDA runtime cannot fully initialize the device. This is a classic systems debugging pattern: a low-level verification passes, creating confidence, while a higher-level integration fails for subtle reasons. The assistant's next steps — installing CUDA toolkit, creating a Python venv, and attempting to run ML workloads — will all fail at the CUDA runtime level, requiring yet another round of debugging.

Conclusion

Message 472 is a moment of apparent success that contains the seeds of future failure. The nvidia-smi command works, the GPUs are visible, the topology will show NODE connectivity — but the Blackwell GSP firmware incompatibility lurks beneath the surface, invisible to this particular verification. The message exemplifies a fundamental truth about systems engineering: passing a check doesn't mean the system works; it only means that particular check passed. The assistant's methodical approach — verify device access, then topology, then CUDA runtime, then ML stack — is sound, but the layered complexity of modern GPU virtualization means that each layer can fail independently, and success at one layer provides no guarantee about the next.