The Topology Verdict: Why `NODE` vs `PHB` Made or Broke Multi-GPU ML

"Excellent! This is the critical result."

With these words, the assistant in message [msg 453] of an opencode coding session acknowledged a pivotal moment in a multi-day struggle to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message, deceptively brief at first glance, represents the culmination of a hardware-level debugging saga that had spanned three previous segments of work — and it unknowingly teeters on the edge of a new, unexpected failure.

To understand why this single message matters, one must appreciate the journey that led here. The session had begun with a straightforward goal: deploy a 4-bit quantized language model across eight top-tier GPUs using SGLang, a high-performance inference engine. But the path was anything but straightforward.

The P2P Bottleneck That Started It All

In earlier segments of the conversation, the team had successfully deployed GLM-5-NVFP4 inside a KVM virtual machine on a Proxmox host. The model ran, but performance was severely limited. The root cause was a topology problem visible through nvidia-smi topo -m: inside the VM, all GPU-to-GPU connections showed as PHB (PCIe Host Bridge), meaning every cross-GPU communication had to traverse the host bridge rather than using direct Peer-to-Peer (P2P) DMA transfers. For tensor-parallel inference — where model layers are split across GPUs and intermediate results must be shuffled constantly — this PHB topology is a performance killer. The virtualization layer (VFIO/IOMMU) was inserting itself between every GPU transaction, adding latency and reducing throughput.

The team had spent segment 3 exhaustively trying to fix this within the VM: adjusting Proxmox host kernel parameters, migrating from i440FX to Q35 chipset, fixing BAR allocation, disabling ACS (Access Control Services), and investigating IOMMU groups. But the fundamental hardware topology — each GPU sitting on its own PCIe root complex — made P2P impossible inside any VM, regardless of configuration tweaks.

The LXC Gambit

Segment 4 began with a new strategy: abandon the VM entirely and run the ML workload inside an LXC (Linux Container) directly on the Proxmox host. Unlike KVM virtual machines, LXC containers share the host kernel and have direct access to hardware devices. If the GPUs could be passed through to the container, the container would see the bare-metal topology — the same NODE and SYS relationships that the host sees — rather than the virtualized PHB topology that VMs impose.

The assistant's first task in segment 4 was to install the NVIDIA driver (590.48.01) directly on the Proxmox host, which initially had no GPU driver at all. Messages [msg 435] through [msg 452] document this process: downloading the 397MB driver, installing it with DKMS, blacklisting the open-source nouveau driver, and verifying that all eight GPUs appeared in nvidia-smi. By message [msg 452], the host was fully driver-equipped, and the assistant was ready for the critical validation.

The Critical Topology Validation

Message [msg 453] opens with the assistant's reaction to running nvidia-smi topo -m inside the LXC context. The full message reads:

Excellent! This is the critical result. The bare-metal topology shows: - GPU0-3 (NUMA 0): NODE to each other - GPU4-7 (NUMA 1): NODE to each other - Cross-NUMA: SYS

Device nodes: major numbers are 195 (nvidia), 504 (nvidia-uvm), 507 (nvidia-caps).

Now let me make the modules persistent and configure the LXC container: [bash] ssh root@10.1.2.6 bash <<'EOFHOST' # Enable persistence mode nvidia-smi --persistence-mode=1 2>&1

# Add nvidia modules to /etc/modules for boot persistence cat >> /etc/modules << 'EOF' nvidia nvidia-uvm nvidia-modeset nvidia-drm EOF

echo "Modules added to /etc/modules"

# The container is currently unprivileged. For full GPU access with 8 GPUs # it's cleaner to make it privileged. Let's check current config: cat /etc/pve/nodes/kpro6/lxc/129.conf EOFHOST Enabled Legacy persistence mode for GPU 00000000:01:00.0. [Enabled for all 8 GPUs...] All done. Modules added to /etc/modules...

The topology data is the centerpiece. In NVIDIA's topology notation, NODE means GPUs are connected within the same NUMA (Non-Uniform Memory Access) node — they share a CPU socket and can communicate via direct PCIe links without traversing the host bridge. SYS means communication must go through system-wide interconnect (typically Infinity Fabric or similar), which is slower but unavoidable for cross-socket traffic. Crucially, neither is PHB — the virtualized topology that had plagued the VM.

This result validated the entire LXC strategy: inside the container, GPUs 0-3 (on NUMA node 0) could talk to each other at full P2P speed, and GPUs 4-7 (on NUMA node 1) could do the same. Cross-NUMA traffic would be slower, but that's a hardware limitation, not a virtualization artifact. The eight-GPU tensor-parallel deployment would now have four fast P2P pairs instead of eight GPUs all bottlenecked through the host bridge.

The Reasoning Behind the Actions

The assistant's response reveals several layers of reasoning. First, the exclamation "Excellent! This is the critical result" shows that the assistant recognized this topology check as a make-or-break validation step. Had the container still shown PHB topology, the LXC approach would have been a dead end, and the team would have needed to reconsider the entire deployment strategy.

Second, the assistant immediately pivots to making the configuration persistent. Enabling persistence mode (nvidia-smi --persistence-mode=1) keeps the GPU driver loaded and initialized even when no process is using the GPUs — essential for a server that needs to respond quickly to inference requests. Adding the NVIDIA kernel modules to /etc/modules ensures they load automatically on boot, so the GPUs are always available after a system restart.

Third, the assistant notes that the container is currently "unprivileged" and states that "for full GPU access with 8 GPUs it's cleaner to make it privileged." This is a significant operational decision. Unprivileged LXC containers have stronger security isolation but restricted access to hardware devices. Privileged containers can access any device node on the host, which is necessary for GPU device files (/dev/nvidia*), the NVIDIA UVM (Unified Virtual Memory) device, and the NVIDIA control capabilities device. The assistant is trading security for functionality — a reasonable tradeoff in a dedicated ML server environment.

Assumptions Embedded in This Message

Several assumptions underpin the assistant's actions here, and some of them would prove incorrect.

The primary assumption is that a working nvidia-smi and correct topology automatically imply a working CUDA runtime. The assistant assumes that because the driver loads, the GPUs appear, and the topology is correct, the full CUDA stack will function inside the container. This assumption is reasonable — it's normally true — but in this case, it would be wrong.

A secondary assumption is that the driver installation on the Proxmox host is complete and sufficient. The assistant has installed the NVIDIA driver package, which includes the kernel module, the nvidia-smi tool, and the userspace CUDA driver library (libcuda.so). But the driver also requires GSP (GPU System Processor) firmware files for Blackwell architecture GPUs, and these may not be present or compatible with the Proxmox VE kernel version (6.8.12-9-pve).

The assistant also assumes that making the container privileged is the cleanest path forward. While this is true for device access, it introduces security considerations and operational complexity that an unprivileged container with carefully curated device cgroup entries might avoid.

The Hidden Pitfall: GSP Firmware and CUDA Initialization

The chunk summary for segment 4 reveals that this message's optimism was premature. After the container was configured with GPU bind-mounts and the topology was confirmed, the team hit a major blocker: CUDA runtime initialization (cuInit) failed with error code 3 (CUDA_ERROR_NOT_INITIALIZED). The error occurred both on the host and inside the container, despite nvidia-smi detecting all eight GPUs correctly.

The root cause was a driver compatibility issue with the Proxmox VE kernel. The NVIDIA driver 590.48.01 lacked Blackwell GSP firmware files — only gsp_ga10x.bin and gsp_tu10x.bin existed, covering older GPU architectures. The Blackwell RTX PRO 6000 GPUs require gsp_* firmware for their GSP, and without it, the CUDA runtime cannot initialize even though the kernel module and nvidia-smi work. The older PVE kernel (6.8.12-9-pve) may also lack necessary support for the Blackwell architecture's GSP requirements.

This is a fascinating failure mode: nvidia-smi works because it uses the kernel module interface (IOCTL calls to /dev/nvidia*), which doesn't require GSP firmware. But CUDA initialization requires the userspace driver to communicate with the GSP, which in turn requires the firmware to be loaded. The symptom — nvidia-smi works but CUDA doesn't — is highly confusing and would have been difficult to diagnose without deep knowledge of NVIDIA's driver architecture.

Input Knowledge Required

To fully understand message [msg 453], a reader needs several pieces of background knowledge:

NVIDIA topology notation: Understanding what NODE, SYS, and PHB mean in nvidia-smi topo -m output, and why NODE is desirable for multi-GPU communication.
NUMA architecture: Knowledge that modern multi-socket servers have multiple NUMA domains, and that GPU placement relative to NUMA nodes affects memory access latency.
LXC container security models: Understanding the difference between privileged and unprivileged containers, and why device access requires privileged mode.
NVIDIA driver stack: Familiarity with the kernel module (nvidia), userspace driver (libcuda.so), UVM (nvidia-uvm), and the role of GSP firmware.
Proxmox VE specifics: Knowledge of /etc/pve/nodes/ configuration files, LXC container config format, and the relationship between host kernel and container device access.
The broader context: Awareness that this is a multi-segment debugging effort where P2P performance was the central challenge, and that the LXC approach was a fallback after VM-based solutions failed.

Output Knowledge Created

This message produces several concrete outputs:

Validation data: The confirmed topology (NODE within sockets, SYS across sockets) proves that LXC containers expose bare-metal GPU topology, unlike KVM VMs which show PHB. This is a reusable finding for anyone deploying multi-GPU workloads on Proxmox.
Device node mapping: The major numbers (195 for nvidia, 504 for nvidia-uvm, 507 for nvidia-caps) are documented, which is essential for configuring LXC bind-mounts.
Persistence configuration: The commands to enable persistence mode and add modules to /etc/modules provide a repeatable setup procedure.
Container configuration direction: The decision to convert from unprivileged to privileged mode establishes the operational pattern for GPU-enabled LXC containers.
A cautionary tale: The gap between this message's optimism and the subsequent CUDA failure serves as a valuable lesson about the difference between kernel-level GPU detection and userspace CUDA runtime initialization.

Conclusion

Message [msg 453] captures a moment of triumph that was, unbeknownst to the assistant, about to be undercut by a subtle driver firmware issue. It demonstrates the importance of topology validation in multi-GPU deployments, the power of LXC containers for bypassing virtualization overhead, and the danger of assuming that nvidia-smi success guarantees CUDA functionality. For anyone deploying GPU workloads on virtualized infrastructure, this message — and the failure that followed it — offers a master class in the layers of abstraction between hardware and software, and the careful validation each layer requires.

The Topology Verdict: Why NODE vs PHB Made or Broke Multi-GPU ML