The Topology Revelation: How an LXC Container Exposed True GPU P2P Capability on Proxmox

In the course of a complex multi-day effort to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message from an AI assistant marked a pivotal moment of validation. Message [msg 474] reads:

This is the key result — NODE and SYS topology, identical to bare metal! In the KVM VM it was PHB everywhere. The container sees the real PCIe topology with proper NUMA affinity.

This brief exclamation, accompanied by a todowrite update marking several tasks as completed, represents far more than a simple status update. It is the culmination of a sustained debugging effort spanning multiple sessions, a hypothesis proven correct, and a bittersweet victory that would soon be tempered by a new, unexpected blocker. To understand why this message matters, one must trace the thread of reasoning that led to it, the assumptions that guided the work, and the technical context that gave it significance.

The Problem: VFIO/IOMMU and the PHB Topology

The session's broader goal was to achieve high-throughput inference of a 400+ billion parameter model using tensor parallelism across eight GPUs. Tensor parallelism requires frequent all-reduce operations between GPUs, and the performance of these operations depends critically on whether GPUs can communicate via direct Peer-to-Peer (P2P) DMA over the PCIe bus, rather than being forced to route data through system memory (host RAM).

Earlier in the conversation ([msg 453]), the assistant had verified the GPU topology inside a KVM virtual machine on the Proxmox host. The output of nvidia-smi topo -m showed PHB (PCIe Host Bridge) connectivity between every pair of GPUs. In NVIDIA's topology matrix, PHB indicates that two GPUs are behind different PCIe host bridges and cannot perform direct P2P DMA — all cross-GPU communication must go through the CPU's memory controller and system RAM. This is a severe performance bottleneck for tensor-parallel workloads, where every transformer layer requires synchronizing activations across all GPUs.

The root cause was the VFIO (Virtual Function I/O) passthrough mechanism used by KVM. When GPUs are passed through to a VM via VFIO, the VM's NVIDIA driver sees each GPU behind its own emulated PCIe root port, isolated by the IOMMU. The physical PCIe topology — including the NVSwitch or PCIe switch fabric that connects GPUs at the hardware level — is invisible to the guest. From the VM's perspective, every GPU appears to be on its own isolated PCIe segment, hence the PHB topology.

The Hypothesis: LXC Containers Bypass VFIO Isolation

The assistant's reasoning was that an LXC container, unlike a KVM VM, does not virtualize the PCIe bus. LXC containers share the host kernel and simply isolate processes via namespaces and cgroups. Device nodes like /dev/nvidia0 through /dev/nvidia7 are bind-mounted into the container from the host. The NVIDIA userspace driver (libcuda.so) running inside the container communicates with the kernel module (nvidia.ko) running on the host. Since there is no IOMMU emulation or VFIO layer between the container and the hardware, the driver should see the real PCIe topology.

This hypothesis carried several assumptions:

1. The Proxmox host could run the NVIDIA driver. The host runs a custom Proxmox VE kernel (6.8.12-9-pve), which is based on Ubuntu's kernel but with modifications for virtualization. NVIDIA's driver must be compatible with this kernel.
2. Device bind-mounts would be sufficient. The assistant assumed that bind-mounting the NVIDIA device files (/dev/nvidia[0-7], /dev/nvidiactl, /dev/nvidia-uvm, etc.) into the container would give the userspace driver full access to the GPUs.
3. The NVIDIA kernel module on the host would handle all GPU control. The userspace driver inside the container would communicate with the host's kernel module via ioctl calls on the device files, without needing its own kernel module.
4. The topology reported by nvidia-smi topo -m inside the container would match the bare-metal topology. This was the key prediction to test.

Executing the Hypothesis

The assistant executed this plan over a series of messages ([msg 445] through [msg 473]). The steps included:

• Installing the NVIDIA driver (590.48.01) on the Proxmox host ([msg 450]). This required downloading the 397MB driver package and running it with --dkms --silent --no-x-check. The driver loaded successfully, and nvidia-smi on the host showed all 8 GPUs.
• Converting the LXC container from unprivileged to privileged ([msg 455]). Unprivileged containers use user namespace mapping that shifts UIDs, which complicates device access. Switching to privileged mode gave the container root-level access to the host's device nodes.
• Configuring device bind-mounts ([msg 455]). The assistant added lxc.mount.entry lines for each GPU device node, along with lxc.cgroup2.devices.allow rules to grant cgroup access to the NVIDIA device major numbers (195, 504, 507).
• Fixing UID ownership ([msg 460], [msg 462]). The conversion from unprivileged to privileged left files owned by UID 100000 (the id mapping offset). The assistant used a Perl-based batch chown to shift ownership back to root.
• Installing the NVIDIA userspace driver inside the container ([msg 471]). The same driver .run file was copied into the container and installed with --no-kernel-module --silent --no-x-check, which installs only the userspace libraries (libcuda.so, libnvidia-ml.so, etc.) without attempting to load a kernel module.
• Verifying GPU visibility ([msg 472]). nvidia-smi inside the container confirmed all 8 GPUs were visible.
• The critical topology test ([msg 473]). The assistant ran nvidia-smi topo -m inside the container.

The Result: NODE and SYS Topology Confirmed

Message [msg 473] showed the topology matrix. The output revealed:

• GPU0 through GPU3: Connected via NODE to each other. These four GPUs share the same NUMA node (NUMA 0) and are directly connected via the PCIe switch fabric.
• GPU4 through GPU7: Also connected via NODE to each other, on NUMA node 1.
• Cross-NUMA pairs (e.g., GPU0 to GPU4): Connected via SYS (system memory), meaning P2P between sockets requires going through the CPU interconnect. This is the exact topology that would be seen on bare metal. The NODE designation means GPUs within the same NUMA domain can perform direct P2P DMA without involving the CPU. This is a dramatic improvement over the PHB topology seen in the KVM VM, where every GPU pair required system memory routing. Message [msg 474] is the assistant's immediate reaction. The emphasis ("This is the key result") and the contrast ("In the KVM VM it was PHB everywhere") highlight the significance. The hypothesis was validated: LXC containers do expose the true PCIe topology.

The Bittersweet Aftermath

The victory was short-lived. Immediately after this message, the assistant proceeded to install the ML stack — CUDA Toolkit 12.8, PyTorch, and sglang — inside the container ([msg 475] through [msg 490]). But when it tried to run a CUDA program, cuInit returned error code 3 (CUDA_ERROR_NOT_INITIALIZED) ([msg 510]). This failure occurred both inside the container and on the host itself.

The root cause appeared to be a driver compatibility issue with the Proxmox VE kernel. The NVIDIA driver 590.48.01, while detecting the Blackwell GPUs correctly via nvidia-smi, could not initialize the CUDA runtime. The open-source kernel module (nvidia.ko) saw the GPUs but CUDA could not communicate with them. When the proprietary kernel module was loaded instead, the GPUs became completely invisible.

The chunk summary identifies the likely cause: the driver lacks Blackwell GSP (GPU System Processor) firmware files. The Proxmox VE kernel (6.8.12-9-pve) is older than the kernels typically used with Blackwell GPUs, and the GSP firmware requirements for Blackwell may not be met by driver 590.48.01. This is a fundamental difference from the KVM VM scenario, where the guest's own NVIDIA driver stack handled VFIO-passed GPUs without requiring host-level GSP firmware.

What This Message Teaches Us

Message [msg 474] is a study in the nature of debugging complex systems. It represents:

1. A hypothesis validated by experiment. The assistant had a theory about why P2P was broken (VFIO/IOMMU topology virtualization) and a proposed fix (LXC containers). The experiment confirmed the theory.
2. The limits of abstraction. KVM's device virtualization provides strong isolation but at the cost of hiding hardware topology. LXC's lighter-weight isolation preserves hardware visibility. This tradeoff is fundamental to virtualization design.
3. The layered nature of GPU software. Even when the topology is correct and nvidia-smi works, the CUDA runtime can still fail. The NVIDIA driver stack has multiple layers — kernel module, userspace library, GSP firmware — and each can be a point of failure.
4. The importance of kernel compatibility. The Proxmox VE kernel, while excellent for virtualization, may lag behind the kernel versions that NVIDIA targets for new GPU architectures like Blackwell.

Conclusion

Message [msg 474] captures a moment of clarity in a complex debugging session. The assistant correctly identified that VFIO passthrough was obscuring the true GPU topology, correctly predicted that LXC containers would expose the real topology, and confirmed this prediction with nvidia-smi topo -m. The NODE and SYS topology inside the container was identical to bare metal, proving that P2P DMA should work.

Yet the message also foreshadows the next challenge. The topology was right, but CUDA initialization failed. The hardware was visible but not usable. This pattern — progress followed by a new blocker — is characteristic of deep infrastructure debugging. Each layer solved reveals the next layer's problem. The assistant's work would continue into investigating GSP firmware, kernel versions, and driver compatibility, but message [msg 474] stands as the moment when one critical piece of the puzzle fell into place.