Mapping the Unseen: How a Simple Diagnostic Command Revealed a GPU Topology Discrepancy in Proxmox
In the middle of a complex GPU reconfiguration task on a Proxmox host, a single diagnostic command served as the critical bridge between two different representations of the same hardware. The message at [msg 6040] is deceptively simple — a bash one-liner that maps NVIDIA device files to PCI addresses and NUMA nodes — but it represents a pivotal moment of knowledge discovery that would determine the success or failure of the entire reconfiguration effort.
The Message
The assistant wrote:
Now let me check the nvidia device numbering to understand the mapping between device files and PCI addresses.
>
``bash ssh root@10.1.2.6 'for i in 0 1 2 3 4 5 6 7; do pci=$(readlink -f /sys/class/drm/card${i}/device 2>/dev/null | xargs basename 2>/dev/null); numa=$(cat /sys/class/drm/card${i}/device/numa_node 2>/dev/null); echo "nvidia$i -> PCI $pci, NUMA $numa"; done' ``
>
`` nvidia0 -> PCI 0000:52:00.0, NUMA 0 nvidia1 -> PCI 0000:01:00.0, NUMA 0 nvidia2 -> PCI 0000:11:00.0, NUMA 0 nvidia3 -> PCI 0000:61:00.0, NUMA 0 nvidia4 -> PCI 0000:71:00.0, NUMA 0 nvidia5 -> PCI 0000:81:00.0, NUMA 1 nvidia6 -> PCI 0000:91:00.0, NUMA 1 nvidia7 -> PCI 0000:e1:00.0, NUMA 1 ``
Why This Message Was Written: The Reasoning and Context
The message exists because of a concrete operational need. In [msg 6034], the user stated: "I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)." This is a hardware partitioning task: eight NVIDIA RTX PRO 6000 Blackwell GPUs need to be split between two consumers — an LXC container running SGLang for LLM inference, and a VM (or VMs) that will receive the remaining four GPUs via PCIe passthrough using the vfio-pci driver.
The assistant had already gathered substantial topology information. From [msg 6035], it knew the PCI addresses of all eight GPUs via lspci. From [msg 6036], it knew which PCI addresses belonged to which NUMA node: four GPUs on NUMA 0 (01:00.0, 11:00.0, 61:00.0, 71:00.0) and four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). From [msg 6037], it had the LXC configuration file showing bind mounts for /dev/nvidia0 through /dev/nvidia7.
But there was a critical gap in this knowledge. The LXC configuration references GPUs by their device file names (/dev/nvidia0, /dev/nvidia1, etc.), while the VM passthrough configuration references GPUs by their PCI addresses (0000:01:00.0, etc.). These two naming schemes are not directly correlated — the NVIDIA driver assigns device numbers in an enumeration order that depends on driver initialization, PCI bus scan order, and other factors that may not align with a simple ascending PCI address sort. Without establishing the mapping between "nvidia device number" and "PCI address," the assistant could not determine which device files to keep in the LXC and which PCI addresses to assign to the VM.
This is the fundamental reasoning behind the message. The assistant recognized a missing piece of information — the bridge between two coordinate systems — and designed a targeted command to fill that gap. The command iterates over device indices 0 through 7, follows the symbolic link from the DRM card device to its parent PCI device, extracts the PCI address, and reads the NUMA node from the device's numa_node file. The output is a clean mapping table that connects all three representations: the logical device number, the physical PCI address, and the memory topology (NUMA node).
The Critical Discovery: A Mapping Discrepancy
The output of this command reveals something unexpected. Comparing the PCI addresses from the nvidia device mapping with the earlier lspci output shows a discrepancy:
| nvidia device | PCI address (from nvidia mapping) | PCI address (from lspci) | |---|---|---| | nvidia0 | 0000:52:00.0 | — | | nvidia1 | 0000:01:00.0 | 01:00.0 | | nvidia2 | 0000:11:00.0 | 11:00.0 | | nvidia3 | 0000:61:00.0 | 61:00.0 | | nvidia4 | 0000:71:00.0 | 71:00.0 | | nvidia5 | 0000:81:00.0 | 81:00.0 | | nvidia6 | 0000:91:00.0 | 91:00.0 | | nvidia7 | 0000:e1:00.0 | e1:00.0 | | — | — | f1:00.0 |
The GPU at PCI address 0000:52:00.0 appears in the nvidia device mapping as nvidia0 but was not present in the lspci output from [msg 6035]. Conversely, the GPU at f1:00.0 was listed in lspci but does not appear in the nvidia device mapping. This is a genuine topological anomaly.
Several explanations are possible. The GPU at 52:00.0 might be bound to the nvidia driver but was missed by the lspci grep (perhaps it has a different device class or the PCI bus was rescanned after the initial command). Alternatively, the GPU at f1:00.0 might be bound to a different driver — perhaps it was already assigned to vfio-pci for a different purpose, or it failed to initialize with the nvidia driver. There is also the possibility that the device numbering is simply non-contiguous or that a GPU was hot-added or hot-removed between the two commands.
This discrepancy has immediate practical consequences. The NUMA distribution from the nvidia mapping shows five devices on NUMA 0 (nvidia0 through nvidia4) and only three on NUMA 1 (nvidia5 through nvidia7). This contradicts the earlier finding of a perfect 4+4 split. If the assistant were to blindly follow the NUMA-based split suggested by the user ("nvidia driver to 4 on one numa, other 4 to vfio"), it would need to account for this mismatch. The presence of an extra GPU on NUMA 0 (at 52:00.0) means that either the split is not cleanly 4+4 by NUMA, or one of the NUMA 0 GPUs needs to be moved to the VM side, breaking the NUMA alignment.
Input Knowledge Required
To understand this message, one needs knowledge of several distinct domains. First, an understanding of Proxmox virtualization concepts: how LXC containers access host devices through bind mounts, and how PCI passthrough works with vfio-pci for VM GPU assignment. Second, familiarity with the Linux device model: the /sys filesystem, symbolic links in /sys/class/drm/, and the numa_node attribute that exposes memory locality information. Third, knowledge of NVIDIA GPU device enumeration: how the nvidia driver creates /dev/nvidiaX device files and how they correspond to DRM card entries. Fourth, an understanding of PCI topology: bus addresses, domain numbering, and how lspci enumerates devices. Finally, the concept of NUMA (Non-Uniform Memory Access) nodes and why GPU-to-NUMA affinity matters for performance — keeping GPU memory access local to the CPU socket that drives it avoids cross-socket bandwidth penalties.
Output Knowledge Created
This message produces a concrete mapping table that directly enables the next steps of the reconfiguration. The assistant now knows that to keep NUMA 0 GPUs in the LXC, it needs to retain bind mounts for nvidia0 through nvidia4 (five devices, not four), and to passthrough NUMA 1 GPUs to the VM, it needs to assign PCI addresses 81:00.0, 91:00.0, and e1:00.0 (three devices, not four) to vfio-pci. The mapping also reveals that the GPU at f1:00.0 requires investigation — is it already in vfio-pci, or is there a driver issue?
More subtly, the message creates knowledge about the process of topology discovery. The assistant has demonstrated a pattern for bridging between logical device names and physical PCI addresses that can be reused in any similar reconfiguration task. The command itself is a reusable diagnostic tool: iterate over device indices, resolve the PCI address through the DRM symlink, and read the NUMA node. This pattern is valuable beyond this specific context.
Assumptions and Potential Pitfalls
The message rests on several assumptions that warrant examination. The assistant assumes that the nvidia device numbering (nvidia0 through nvidia7) is contiguous and that iterating indices 0-7 will cover all GPUs. This is generally true for a fresh driver load, but device numbering can be sparse if some GPUs failed to initialize or were bound to different drivers. The command uses /dev/null suppression for missing entries, so it would silently skip missing devices — a design choice that avoids error noise but could mask problems.
The assistant also assumes that the DRM card device (/sys/class/drm/card${i}) corresponds directly to the nvidia device of the same index. This is typically true when only NVIDIA GPUs are present, but if there are other DRM devices (e.g., integrated AMD graphics or a virtual GPU), the indexing could diverge. In this system, the assumption holds because all eight DRM cards are NVIDIA GPUs.
A more subtle assumption is that the numa_node file exists and contains accurate information. On systems without NUMA or with misconfigured BIOS, this file might be missing or report -1 (meaning "no specific NUMA node"). The command handles this gracefully by suppressing errors, but a missing value would leave the NUMA assignment unknown.
The most significant potential pitfall is the discrepancy between the nvidia mapping and the lspci output. If the assistant proceeds without resolving this discrepancy, it could make incorrect decisions about which GPUs to assign to which consumer. The GPU at 52:00.0 needs to be identified and either included in the LXC or moved to vfio-pci, and the GPU at f1:00.0 needs to be accounted for even though it doesn't appear in the nvidia device list.
Conclusion
The message at [msg 6040] is a textbook example of a well-motivated diagnostic step in systems administration. It is not glamorous — it is a simple bash loop that reads a few files from /sys — but it bridges a critical knowledge gap between two representations of the same hardware. The mapping it produces directly enables the GPU split that the user requested, and the discrepancy it reveals (the unexpected presence of 52:00.0 and the absence of f1:00.0) provides a crucial warning that the topology is not as simple as previously assumed. In the complex dance of GPU virtualization, where device files, PCI addresses, NUMA nodes, and driver bindings must all align, this single command provides the essential translation layer that makes the rest of the reconfiguration possible.