The Moment of Truth: Verifying Bare-Metal GPU Topology After the HMM Blocker

The journey to deploy the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs had been a grueling saga of hardware and software obstacles. After migrating from a KVM virtual machine to an LXC container to bypass VFIO's PCIe passthrough limitations, the assistant had hit a critical wall: CUDA refused to initialize, returning error code 3 (CUDA_ERROR_NOT_INITIALIZED) or hanging indefinitely. The root cause, discovered through painstaking investigation, was the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature, which was incompatible with the Proxmox VE kernel. The fix — setting uvm_disable_hmm=1 as a module parameter for nvidia_uvm — had worked immediately, and CUDA was now detecting all eight GPUs inside the container. But the fundamental question remained unanswered: did the LXC approach actually deliver the bare-metal GPU topology it promised? Message 567 is the moment that question gets answered.

The Message: Two Commands, One Pivotal Verification

The message is deceptively brief, containing just two tool calls executed in parallel:

Now let me verify the GPU topology from inside the container with CUDA, 
enable persistence mode, and do a quick P2P test.

The assistant dispatches two bash commands simultaneously. The first runs nvidia-smi topo -m inside the LXC container at internal IP [REDACTED], dumping the full GPU topology matrix. The second runs nvidia-smi -pm 1 on the Proxmox host at [REDACTED] to enable persistence mode across all eight GPUs.

The topology output reveals a matrix of GPU interconnect types:

GPU0    X   NODE  NODE  NODE  SYS   SYS   SYS   SYS   0-31,64-95   0    N/A
GPU1   NODE   X   NODE  NODE  SYS   SYS   SYS   SYS   0-31,64-95   0    N/A
GPU2   NODE  NODE   X   NODE  SYS   SYS   SYS   SYS   0-31,64-95   0    N/A
GPU3   NODE  NODE  NODE   X   SYS   SYS   SYS   SYS   0-31,64-95   0    N/A
GPU4   SYS   SYS   SYS   SYS   X    NODE  NODE  NODE  32-63,96-127 1    N/A
GPU5   SYS   SYS   SYS   SYS  NODE   X    NODE  NODE  32-63,96-127 1    N/A
GPU6   SYS   SYS   SYS   SYS  NODE  NODE   X    NODE  32-63,96-127 1    N/A
GPU7   SYS   SYS   SYS   SYS  NODE  NODE  NODE   X    32-63,96-127 1    N/A

The persistence mode command returns confirmation for all eight GPUs, each identified by its PCIe address (e.g., GPU 00000000:01:00.0), noting "Enabled Legacy persistence mode."

Why This Message Was Written: The Reasoning and Motivation

This message exists because the entire LXC migration strategy hinged on a single hypothesis: that bypassing the VFIO/IOMMU layer would restore direct PCIe peer-to-peer (P2P) access between GPUs. In the earlier KVM VM, nvidia-smi topo -m had shown "PHB" (PCIe Host Bridge) connections, meaning every cross-GPU communication had to traverse the host memory through the virtualized PCIe root complex. This introduced latency of approximately 13 microseconds per transfer, a crippling overhead for the all-to-all communication patterns required by tensor parallelism in large model inference.

The assistant's reasoning, visible in the preceding messages, was methodical. First, confirm CUDA works (done in [msg 565]). Second, verify the topology from inside the container, not from the host, because the container's view of PCIe topology is what matters for CUDA-aware communication libraries like NCCL. Third, enable persistence mode to ensure stable GPU state for the upcoming inference server launch. The P2P test mentioned in the message's preamble would come next ([msg 568]), but the topology verification was the critical prerequisite.

Decoding the Topology Matrix: What NODE and SYS Actually Mean

The nvidia-smi topo -m output uses a compact notation where each cell describes the closest interconnect type between a pair of GPUs. The possible values, ordered from closest to farthest, are: NODE (same PCIe switch or NUMA node), PHB (same PCIe host bridge), PXB (multiple PCIe bridges), PIX (multiple PCIe switches), and SYS (system interconnect, typically across CPU sockets via UPI or similar).

The output shows a clean two-socket NUMA topology. GPUs 0 through 3 are attached to NUMA node 0 (CPU affinity 0-31,64-95), and they see each other as NODE — meaning they share a common PCIe hierarchy, likely through the same PCIe switch or root port. GPUs 4 through 7 are on NUMA node 1 (CPU affinity 32-63,96-127) with the same NODE interconnect among themselves. Cross-NUMA pairs (e.g., GPU0 to GPU4) show SYS, indicating the connection traverses the CPU-to-CPU interconnect (UPI or similar).

This is the canonical topology for a dual-socket server with GPUs evenly split across the two CPU sockets. It is exactly what bare-metal looks like. The absence of any PHB entries is the key validation: the LXC container sees the real PCIe topology, not a virtualized abstraction. This confirmed that the migration from KVM to LXC had succeeded in its primary objective.

The Persistence Mode Decision

The second command, nvidia-smi -pm 1, enables persistence mode on all GPUs. This is a subtle but important operational decision. Without persistence mode, the NVIDIA driver unloads its state from the GPU when no process holds the device open. The next CUDA call then incurs the full initialization overhead — loading firmware, initializing memory, and re-establishing state. For an inference server that may experience gaps between requests, this latency spike can be noticeable.

The assistant chose to run this from the host (the Proxmox server at [REDACTED]) rather than from inside the container, because nvidia-smi -pm modifies a global driver setting that persists across all users of the GPU. Running it from the host ensures the setting survives container restarts. The output's mention of "Legacy persistence mode" is noteworthy — it suggests the driver version 590.48.01 has both a legacy and a modern persistence mechanism, and the legacy path was invoked. This is a minor implementation detail but hints at the driver's transitional state for Blackwell architecture support.

Assumptions and Their Validity

The message rests on several assumptions, most of which proved correct. The assistant assumed that nvidia-smi topo -m would report accurate topology information from inside the LXC container. This assumption was sound because LXC shares the host kernel and device files directly — unlike KVM, there is no emulated PCIe layer to distort the topology. The output confirmed this.

The assistant also assumed that enabling persistence mode from the host would affect GPU behavior inside the container. This is correct because persistence mode is a kernel module setting, not a per-process one. The nvidia-smi -pm command writes to the driver's state, which applies to all subsequent device opens regardless of which namespace or container they originate from.

A more subtle assumption is visible in the message's preamble: "do a quick P2P test." The assistant implicitly assumed that NODE topology implies functional P2P access. While NODE topology is necessary for P2P, it is not sufficient — the GPU driver and CUDA must also support P2P for the specific GPU architecture and driver version. This assumption was validated in the subsequent message ([msg 568]), which confirmed that all 64 GPU pairs (8×8) had P2P access, with bandwidth reaching 53.76 GB/s within the same NUMA node.

Input Knowledge Required to Understand This Message

A reader needs substantial context to grasp the significance of this message. They must understand the difference between KVM virtualization (which emulates PCIe topology through VFIO) and LXC containerization (which shares the host kernel and device files directly). They need to know that nvidia-smi topo -m reports the GPU interconnect topology as seen by the CUDA driver, and that the topology type (NODE vs PHB vs SYS) determines whether direct GPU-to-GPU memory transfers are possible via P2P.

Familiarity with NUMA (Non-Uniform Memory Access) architecture is also essential. The output's CPU affinity ranges (0-31,64-95 and 32-63,96-127) describe which CPU cores and memory controllers are closest to each GPU group. The split into two NUMA domains (0 and 1) reflects the dual-socket server hardware, and the SYS entries between GPU groups indicate that cross-socket communication must traverse the CPU interconnect, which has higher latency and lower bandwidth than same-socket transfers.

The concept of "persistence mode" requires understanding how the NVIDIA driver manages GPU state. Without it, the driver may unload GPU firmware and reset memory allocations when idle, causing latency on the next CUDA call. With it, the driver maintains a persistent connection, reducing initialization overhead.

Output Knowledge Created by This Message

This message produces two distinct pieces of output knowledge. First, it definitively confirms that the LXC container provides bare-metal GPU topology. The NODE/SYS matrix is indistinguishable from what you would see running directly on the host operating system. This validates the entire migration strategy and opens the door to efficient tensor parallelism across all eight GPUs.

Second, it establishes that persistence mode is active across all GPUs, identified by their PCIe bus addresses (00000000:01:00.0, 00000000:11:00.0, etc.). These addresses are useful for correlating GPU identity with physical slot locations on the motherboard, which matters for diagnosing PCIe link issues or planning GPU-to-NIC affinity for networking.

The message also implicitly creates a baseline for comparison. The earlier KVM VM's topology (PHB) can now be contrasted with the LXC container's topology (NODE/SYS), providing a quantitative justification for the migration effort. The subsequent P2P bandwidth test ([msg 568]) builds directly on this foundation, measuring 53.76 GB/s same-NUMA and 40.24 GB/s cross-NUMA — numbers that would have been impossible under the VFIO layer.

The Thinking Process Visible in the Message

The message's reasoning is concise but reveals a clear prioritization. The assistant lists three tasks: "verify the GPU topology from inside the container with CUDA, enable persistence mode, and do a quick P2P test." The ordering is deliberate — topology verification comes first because it determines whether the P2P test is even meaningful. If the topology had shown PHB or PXB instead of NODE, the P2P test would have been a waste of time, and the entire LXC strategy would need reconsideration.

The parallel dispatch of both commands is also telling. The assistant is confident enough in the topology result to proceed with enabling persistence mode simultaneously. There is no conditional logic — "if topology is good, then enable persistence" — because the assistant has already validated CUDA functionality in the previous message. The topology verification is a confirmation, not an experiment.

The use of nvidia-smi -pm 1 from the host rather than the container shows awareness of the tool's scope. nvidia-smi inside the container can query GPU state but modifying driver parameters requires host-level access. This distinction between query and control operations is a subtle but important systems administration insight.

Conclusion

Message 567 is a brief but pivotal moment in a long debugging saga. In two commands and a few lines of output, it validates weeks of effort: the LXC container delivers genuine bare-metal GPU topology with NODE-level interconnects within each NUMA domain, and persistence mode is operational. The foundation for deploying GLM-5-NVFP4 with tensor parallelism across eight Blackwell GPUs is finally solid. The subsequent messages would build on this foundation, launching the sglang inference server and achieving throughput of 806 tokens per second at 128 concurrent requests. But none of that would have been possible without first answering the question posed in this message: does the topology actually work?