The NVLink Investigation: A Critical Fork in the Performance Optimization Journey

Introduction

In the high-stakes world of multi-GPU machine learning inference, the difference between PCIe-bound communication and NVLink's high-speed GPU-to-GPU fabric can mean the difference between a model that serves requests at interactive speeds and one that struggles to keep pace. When a team is deploying the massive GLM-5-NVFP4 Mixture-of-Experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the question of whether NVLink is available is not academic—it is existential to the performance tuning strategy.

This article examines a pivotal sub-investigation within a broader performance optimization session. The team had just computed a theoretical maximum single-stream throughput of 309 tokens per second, only to observe actual performance of a mere 10.36 tokens per second—a staggering 3.4% efficiency gap. A comprehensive parallel system audit had already uncovered numerous misconfigurations: a suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), an outdated kernel (6.8.12), enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of the optimal 4096. But one question loomed largest: were the GPUs connected via NVLink, or was inter-GPU communication limited to PCIe?

The NVLink investigation, spanning messages 0 through 3 of this conversation, represents a masterclass in systematic hardware diagnosis [3][4]. What began with a user's hopeful hypothesis—that a mysterious PCI device might be an NVSwitch or NVLink bridge—ended with a definitive, multi-evidence verdict that shaped the entire optimization strategy going forward [1][2].

The Context: A Performance Crisis Demands Answers

The broader session had been a marathon of environment setup, driver installation, and performance debugging. After installing NVIDIA drivers (590.48.01), CUDA Toolkit 13.1, creating a Python virtual environment with uv, and resolving complex flash-attn build issues (including installing a secondary CUDA 12.8 toolkit and reducing parallel compilation jobs from 128 to 20 to avoid memory exhaustion), the team had finally reached the performance benchmarking phase. The results were devastating.

The theoretical maximum single-stream throughput had been calculated at approximately 309 tokens per second. This calculation accounted for the GPU's compute capability, memory bandwidth, and the model's arithmetic intensity. The actual measured performance was around 10.36 tokens per second. This represented a 3.4% efficiency gap—the system was achieving only one-thirtieth of its theoretical peak. Something was fundamentally wrong.

A comprehensive parallel system audit was launched via ten agents, uncovering numerous misconfigurations across the software and hardware stack. All runtime fixes were applied, and a major kernel upgrade from 6.8.12 to 6.14.11 was performed with amd_pstate=active and processor.max_cstate=1 boot parameters. A post-reboot issue where CUDA failed inside the LXC container was traced to stale NVIDIA device major numbers in the cgroup configuration, which were quickly updated to match the new kernel's device map.

Yet even after these improvements, the performance gap remained enormous. It was at this critical juncture that the user pivoted the investigation toward a hardware-level question that could explain everything: were the GPUs connected via NVLink?

The Hypothesis: NVLink Might Be Dormant

The user's message ([msg 0]) laid out the stakes with clarity and urgency [3]. Earlier topology checks using nvidia-smi topo -m had shown only NODE and SYS connections between GPUs—indicating standard PCIe topology with no NVLink links. However, a mysterious PCI device at address 0000:52:00.0 had been spotted during earlier enumeration, and the user hypothesized it could be an NVSwitch or NVLink bridge.

The reasoning was sophisticated. The RTX PRO 6000 Blackwell Server Edition, being a server-class SKU, might have NVLink bridges on the server board that simply weren't initialized due to a missing fabric manager service. NVLink requires the NVIDIA Fabric Manager (nvidia-fabricmanager) to be running, which manages the NVSwitch fabric and establishes GPU-to-GPU connections. If the fabric manager was simply not installed or not running, the GPUs would fall back to PCIe communication, producing exactly the NODE/SYS topology the earlier investigation had found. The hardware would appear NVLink-incapable when it actually had the capability dormant.

The user's language conveyed the stakes: "This is important because if NVLink IS available but not initialized (missing fabric manager), that would be a MASSIVE performance improvement." If NVLink was available but dormant, installing and starting the NVIDIA fabric manager could unlock a step-change improvement in inter-GPU communication bandwidth, directly attacking the massive efficiency gap. If NVLink was absent from the silicon itself, the team would need to accept PCIe-bound communication and focus optimization efforts elsewhere.

The user dispatched ten specific diagnostic commands to be run via SSH on the Proxmox host at 10.1.2.6, covering every conceivable angle: PCI device enumeration, direct NVLink queries via nvidia-smi, topology display, NVSwitch enumeration, kernel log analysis, fabric manager status checks, and targeted searches for bridge or switch devices.

Round One: Parallel Diagnostics

The assistant's first response ([msg 1]) demonstrated a hallmark of efficient agent design: parallelism [4]. Rather than running diagnostic commands sequentially—which would require multiple rounds of tool calls, each waiting for SSH results to return—the assistant dispatched all ten commands in a single parallel batch. This was possible because none of the commands depended on the output of another; each provided an independent piece of evidence.

The results came back revealing a clear but incomplete picture:

The mystery device was immediately identified. The lspci -s 52:00.0 -vvv output revealed it as an "ASPEED Technology, Inc. ASPEED Graphics Family (rev 52)"—the standard server BMC (Baseboard Management Controller) graphics chip, commonly known as the ASPEED AST2600. This chip provides VGA console output for server management and has nothing to do with NVIDIA, NVLink, or GPU interconnect. The user's suspicion, while reasonable, was definitively put to rest.

nvidia-smi spoke unequivocally. Both nvlink --status and nvlink -s returned the same message for every GPU: "Device does not have or support Nvlink." The NVIDIA driver, which has intimate knowledge of the GPU hardware capabilities, confirmed that the RTX PRO 6000 Blackwell Server Edition (device ID 2bb5) does not possess NVLink connectors. This is a hardware limitation, not a software configuration issue.

The topology command failed. nvidia-smi -q -d TOPOLOGY returned "Failed to parse --display/-d flags"—a syntax error. The correct flag would need to be determined in the next round.

NVSwitch support was absent. nvidia-smi nvswitch --list returned "ERROR: Option nvswitch is not recognized." Either the installed NVIDIA driver version did not support NVSwitch commands, or the GPU SKU itself did not expose NVSwitch functionality.

The kernel module told a story. The dmesg output showed repeated cycles of "nvidia-nvlink: Nvlink Core is being initialized" followed by "Unregistered Nvlink Core." This pattern is characteristic of a kernel module that probes for hardware, finds none, and tears down. If NVLink hardware were present, the module would remain registered and active.

Fabric manager was absent. systemctl status nvidia-fabricmanager reported "Unit nvidia-fabricmanager.service could not be found," and which nvidia-fabricmanager returned nothing. However, in light of the other evidence, this was irrelevant—installing it would not magically enable NVLink on hardware that lacked it.

All eight GPUs were identical. The lspci | grep -i nvidia output listed eight devices, all with device ID 2bb5 (NVIDIA RTX PRO 6000 Blackwell Server Edition). No other NVIDIA devices appeared on the PCI bus.

The bridge/switch grep returned empty. The targeted search for NVIDIA bridge or switch devices produced no output.

Round Two: Closing the Loops

The assistant recognized that one critical piece of data was missing—the topology matrix—and that the mystery device needed kernel-driver-level confirmation. In message [msg 2], the assistant issued four corrected and supplementary commands in parallel.

The topology matrix was retrieved with the correct flag. nvidia-smi topo -m produced a clean matrix showing a two-cluster topology: GPUs 0–3 were connected to each other via NODE (same PCIe root complex, NUMA node 0), GPUs 4–7 were similarly connected via NODE on NUMA node 1, but connections between the two groups were SYS (across PCIe root complexes and NUMA domains). There was no NV (NVLink) or PIX (same PCIe switch) connection anywhere.

The mystery device was definitively identified at the kernel driver level. The lspci -s 52:00.0 -k output confirmed the device used the ast kernel driver with kernel modules ast. The vendor:device IDs were [1a03:2000], unmistakably ASPEED Technology's graphics controller.

Bridge chip information was extracted. The nvidia-smi -q output showed Link Width: Max 16x, Current 16x for all GPUs, confirming full PCIe Gen5 x16 operation. Critically, the Bridge Chip section showed Type: N/A and Firmware: N/A for every GPU—no NVLink bridge chip existed.

The Verdict: Eight Lines of Convergent Evidence

Message [msg 3] represents the culmination of the investigation. The assistant delivered a structured report with a bold opening verdict—"NO NVLink. This hardware does NOT have NVLink capability"—followed by eight enumerated evidence points and a concluding analysis of what this means for performance.

The eight evidence points form a converging case from independent sources:

  1. The mystery device at 52:00.0 is an ASPEED AST2600 BMC/IPMI VGA controller, completely unrelated to NVIDIA.
  2. nvidia-smi explicitly confirms no NVLink on all eight GPUs, from both nvlink --status and nvlink -s commands.
  3. Topology is PCIe-only: GPU0-3 on NUMA node 0, GPU4-7 on NUMA node 1, with NODE connections within groups and SYS connections between groups. No NV# entries anywhere.
  4. No NVSwitch support: the driver/SKU doesn't even recognize the nvswitch command.
  5. No fabric manager installed, but this is irrelevant given the hardware lacks NVLink.
  6. Bridge Chip Type: N/A on all GPUs from nvidia-smi -q.
  7. PCI device enumeration shows only eight GPUs, all device ID 2bb5, with zero NVIDIA bridge/switch devices.
  8. dmesg shows NVLink core init/teardown cycles—the module probes for hardware, finds none, and tears down. This is normal behavior for a driver probing absent hardware. The assistant's interpretation of the dmesg log is particularly insightful. A less experienced investigator might see "Nvlink Core is being initialized" and conclude that NVLink is somehow active. But the assistant correctly reads the full pattern: initialization followed by immediate unregistration. The module loads, probes for NVLink hardware, finds none, and tears down. This cycle repeats across multiple boots, confirming it is a routine probe-and-fail pattern rather than a functional NVLink stack.

What This Means for the Deployment

The verdict has profound implications for the GLM-5-NVFP4 model deployment. The RTX PRO 6000 Blackwell Server Edition (device 2bb5) is the professional/workstation Blackwell SKU. Unlike the GB200 or B200 data center GPUs which have NVLink, this RTX PRO line communicates between GPUs exclusively via PCIe Gen5 x16 (32 GT/s per direction, ~64 GB/s bidirectional per GPU).

For multi-GPU tensor parallel inference, all inter-GPU communication goes through PCIe. The theoretical maximum inter-GPU bandwidth drops from 900 GB/s (NVLink 5.0) to approximately 64 GB/s (PCIe Gen5 x16 bidirectional). This means the deployment strategy must favor pipeline parallelism or data parallelism over tensor parallelism, or at least account for the communication penalty.

The message's final paragraph makes the conclusion explicit: "There is no missing software component—the silicon simply doesn't have NVLink connectors." This is a form of negative knowledge that is just as valuable as positive discovery. Knowing what you cannot change is the first step to optimizing what you can.

The Broader Significance

Within the larger narrative of the opencode session, this NVLink investigation represents a critical inflection point. The team had been pursuing performance optimization under the assumption that NVLink might be an untapped resource. The kernel upgrade, CUDA fix, and diagnostic tooling were all steps toward understanding why actual throughput was so far below theoretical maximum.

By conclusively ruling out NVLink, the assistant freed the team to stop chasing a phantom software fix and instead confront the hardware-imposed constraints. The subsequent diagnostic tooling focused on measuring the latency of individual decode components, revealing that simulated BF16 GEMMs and AllReduces accounted for only 8.9ms of the 95ms decode time. This pointed the finger squarely at the FP4 GEMM kernel overhead, MoE routing, and attention as the primary culprits for the massive efficiency gap—not communication latency.

The investigation also demonstrates the importance of definitive hardware investigation in performance optimization. In complex systems, it is tempting to chase software solutions for what might be hardware limitations. By conclusively ruling out NVLink, the assistant saved the team from potentially wasting days trying to install and configure fabric manager, debugging NVLink initialization, or redesigning the tensor parallel strategy around a non-existent interconnect.

Conclusion

The NVLink investigation across messages 0–3 of this conversation is a masterclass in systematic hardware diagnosis. What began with a user's hopeful hypothesis about a mystery PCI device ended with a definitive, multi-evidence verdict supported by eight independent sources of data. The assistant's approach—parallel diagnostics to maximize information density, adaptive correction of tool syntax errors, and careful interpretation of subtle signals like driver probe patterns in kernel logs—provides a reusable template for any future investigation of NVLink capability on unknown hardware.

The verdict—no NVLink—was delivered with clarity and finality, closing a question that could have otherwise consumed many more cycles of investigation. For anyone working with multi-GPU configurations, the investigative framework demonstrated here is worth studying: check the hardware explicitly, verify with multiple tools, interpret ambiguous signals correctly, and never let a single piece of evidence override a converging body of counter-evidence. Sometimes the most valuable diagnostic result is a definitive negative.## References

[1] "The Verdict on NVLink: A Definitive Diagnosis Through Targeted Diagnostics" — Detailed analysis of message 2, covering the corrected topology flag, mystery device identification, and interconnect details extraction.

[2] "Closing the Book on NVLink: A Definitive Hardware Diagnosis" — Comprehensive examination of message 3, the definitive verdict message with eight lines of convergent evidence.

[3] "The NVLink Investigation: A Critical Fork in the Performance Optimization Journey" — Analysis of the user's initial investigation request (message 0), including the hypothesis, assumptions, and diagnostic toolkit.

[4] "The NVLink Investigation: A Definitive Hardware Probe Through Parallel Diagnostics" — Examination of message 1, the first round of parallel diagnostics and their results.