The GPU Bandwidth Investigation: Unraveling Measurement Artifacts in an 8-GPU Inference Server

Introduction

In the high-stakes world of large language model inference, every millisecond matters. When the GLM-5-NVFP4 model deployed on an 8-GPU NVIDIA RTX PRO 6000 Blackwell system was achieving only 10.36 tok/s against a theoretical maximum of 309 tok/s, the investigation naturally turned to inter-GPU communication as a potential bottleneck. What followed was a remarkable diagnostic journey spanning 19 messages — a deep dive into the treacherous waters of GPU peer-to-peer bandwidth measurement that would test the limits of measurement methodology, reveal subtle hardware asymmetries, and ultimately produce a definitive performance characterization of the system's interconnect.

This article synthesizes the entire GPU bandwidth investigation sub-session, tracing the arc from initial measurements through artifact discovery, anomaly isolation, and finally to a robust, trustworthy benchmark. The narrative reveals not just the bandwidth numbers themselves, but the process of arriving at them — a process that required questioning assumptions, debugging measurement tools, and systematically eliminating sources of error.

The Opening Probe: Measuring Under Production Constraints

The investigation began with a carefully crafted diagnostic request from the user ([msg 0]). The user asked the assistant to SSH into the LXC container running the SGLang server and measure actual GPU-to-GPU communication bandwidth — all without disrupting the live inference workload. This constraint shaped everything that followed. The user's message was a masterclass in production-aware diagnostic design: check for existing NCCL tests, grep the server's own logs for NCCL configuration, query nvidia-smi for P2P capabilities, run a lightweight PyTorch bandwidth test with 10 MB buffers, and verify that the nvidia_peermem kernel module was loaded.

The initial results, as documented in <chunk seg=10 chunk=0 article 7>, revealed a confusing picture. The 10 MB bandwidth test showed GPU0→GPU1 at a mere 49 GB/s while GPU4→GPU5 achieved 961.8 GB/s and GPU1→GPU2 reached 822.8 GB/s. The topology matrix from nvidia-smi topo -m showed that GPUs 0-3 were on NUMA node 0 and GPUs 4-7 on NUMA node 1, with "NODE" connections within each group and "SYS" connections between groups. All GPUs reported PCIe Gen5 x16 links. The P2P access matrix showed every GPU could access every other GPU. Yet GPU0 was clearly an outlier — its bandwidth to GPU1 was an order of magnitude lower than what GPU1 achieved with GPU2, despite both pairs being on the same NUMA node.

The Artifact Problem: When Numbers Lie

As the assistant dug deeper, a new problem emerged: the measurement methodology itself was producing unreliable results. When the assistant increased the buffer size from 10 MB to 100 MB in [msg 3], the results became physically impossible: GPU4→GPU5 showed 9,290 GB/s and GPU1→GPU2 showed 8,803 GB/s. These numbers exceed the theoretical maximum of any known GPU interconnect technology by an order of magnitude. Even NVLink 4.0, the fastest GPU interconnect available, maxes out at around 900 GB/s total bandwidth.

The assistant correctly identified these as measurement artifacts in [msg 4], noting that "the ~9 TB/s numbers from other GPUs are suspicious (likely BAR memory mapping / zero-copy rather than real DMA)." BAR1 (Base Address Register 1) is a PCIe feature that maps GPU memory into the CPU's address space. When the BAR1 aperture is large enough — and these RTX PRO 6000 Blackwell GPUs have 128 GB of BAR1 space — the GPU driver can map remote GPU memory directly into the local GPU's address space. A copy_() operation between two GPUs might then resolve to a local memory copy within the BAR aperture rather than an actual DMA transfer over the interconnect, producing artificially high bandwidth measurements.

The assistant attempted to work around this by using larger buffers. In [msg 6], a 256 MB test produced clean, realistic numbers: ~50 GB/s for same-NUMA transfers and ~37 GB/s for cross-NUMA transfers. But when the assistant tried to run a "definitive" 512 MB test in [msg 7], it hit an out-of-memory error — the SGLang server was consuming most of the GPU memory. Falling back to 256 MB in [msg 8] produced results that were even more wildly inconsistent than before, with some pairs showing 20,000+ GB/s. The same buffer size that had produced clean results in one run produced absurd results in another, revealing that the measurement was sensitive to uncontrolled factors — likely the state of the BAR1 cache or the specific memory allocation pattern.

The GPU0 Anomaly: A Persistent Asymmetry

Throughout the artifact noise, one pattern remained consistent: GPU0 was different. Every measurement, whether clean or contaminated, showed GPU0 achieving only ~50 GB/s to its NUMA peers (GPUs 1-3) and ~37 GB/s to cross-NUMA GPUs (4-7). Meanwhile, GPU1→GPU2, GPU2→GPU3, GPU4→GPU5, and all other same-NUMA pairs showed bandwidths consistent with NVLink-class performance when measured correctly.

This asymmetry was the central mystery of the investigation. The assistant systematically tested every direction in [msg 5]: copies FROM GPU0 to all other GPUs, and copies TO GPU0 from all other GPUs. The results were consistent — GPU0 was always the slow link, regardless of direction. When GPU1 copied to GPU0, it got 50 GB/s. When GPU1 copied to GPU2, it got nearly 9,000 GB/s. The bottleneck was specifically associated with GPU0, not with any particular pair.

The assistant explored multiple hypotheses for this anomaly. In [msg 13], it checked IOMMU groups and found that GPU0 was in IOMMU group 25 while GPU1 was in group 47 — different groups, which could mean the IOMMU was forcing traffic through the root complex rather than allowing direct P2P. In [msg 14], it checked ACS (Access Control Services) settings on the PCIe root ports, finding no obvious issues. In [msg 16], it compared BAR1 memory usage across all GPUs and found they were all using roughly the same amount (~96 GB out of 128 GB), ruling out the hypothesis that GPU0's BAR1 was configured differently.

The NCCL Configuration: A Hidden Clue

While investigating the GPU0 anomaly, the assistant made a critical discovery about the NCCL configuration. In [msg 9], it examined the environment variables of the running SGLang process and found:

NCCL_MIN_NCHANNELS=8
NCCL_P2P_LEVEL=5
NCCL_IB_DISABLE=1

The NCCL_P2P_LEVEL=5 setting was particularly significant. As the assistant documented in [msg 15], NCCL_P2P_LEVEL controls which GPU pairs are allowed to use direct P2P access. Level 5 enables P2P across all topologies, including cross-NUMA. This is the most permissive setting, allowing NCCL to use direct GPU-to-GPU transfers wherever possible.

But the assistant also discovered a critical limitation in the SGLang server's NCCL configuration. In [msg 18], it noted that the server log contained the message: "Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs." This meant that SGLang's optimized allreduce implementation — which uses shared memory and IPC for faster collective communication — was unavailable. The system was falling back to pure NCCL allreduce, which uses the Socket transport (TCP/IP over the network stack) for inter-process communication, since no InfiniBand was detected.

This was a crucial finding for understanding the system's performance characteristics. Even if the raw P2P bandwidth between GPUs was adequate, the allreduce implementation was using a suboptimal path. The combination of NCCL_P2P_LEVEL=5 (maximum P2P) with the Socket transport (no IB, no custom allreduce) created a complex communication landscape where some operations could use fast P2P DMA while others were forced through the host network stack.

The Methodological Breakthrough: Explicit CUDA Streams

The turning point in the investigation came in [msg 17], when the assistant redesigned the measurement methodology to use explicit CUDA streams. The key insight was that torch.cuda.synchronize() without arguments was not properly synchronizing both the source and destination devices. By creating an explicit CUDA stream on the source device and using with torch.cuda.stream(stream): to scope the copy operations, the assistant ensured that all operations were properly serialized and synchronized.

The results were immediately clean and consistent:

Stream GPU0→GPU1: 50.3 GB/s
Stream GPU1→GPU0: 49.0 GB/s
Stream GPU1→GPU2: 49.8 GB/s
Stream GPU2→GPU1: 50.0 GB/s
Stream GPU0→GPU4: 37.0 GB/s
Stream GPU4→GPU5: 50.1 GB/s

With explicit stream synchronization, all pairs showed ~50 GB/s same-NUMA and ~37 GB/s cross-NUMA. The phantom 20,000+ GB/s numbers vanished. The GPU0 anomaly disappeared — GPU0 was not actually slower; the earlier measurements had been contaminated by synchronization artifacts. The ~50 GB/s figure represented the true PCIe Gen5 x16 bandwidth (achieving ~79% of the theoretical 63 GB/s), and the ~37 GB/s cross-NUMA figure reflected the additional latency penalty of traversing the CPU socket interconnect.

This was the moment of clarity. The assistant had been chasing a ghost — the GPU0 anomaly was a measurement artifact, not a hardware defect. The real story was simpler and more fundamental: this system had no NVLink. All eight GPUs were connected exclusively via PCIe Gen5 x16, with no NVSwitch or NVLink bridges. The ~900 GB/s numbers that had appeared in earlier tests were BAR1 mapping artifacts, not real DMA bandwidth.

The Definitive Report

With a clean measurement methodology in hand, the assistant produced a comprehensive bandwidth report in [msg 18]. The key findings were:

Hardware Configuration: 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each with 95 GB VRAM, connected via PCIe Gen5 x16 (32 GT/s). Two NUMA nodes with 4 GPUs each. No NVLink.

Measured Bandwidth:

The Broader Context: From Infrastructure to Application

This GPU bandwidth investigation did not occur in isolation. It was part of a larger effort that began with setting up the ML environment on Ubuntu 24.04 — installing NVIDIA drivers (590.48.01), CUDA Toolkit 13.1, resolving flash-attn build issues, and stabilizing the software stack with PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1. The machine had been upgraded to 8 GPUs, and the GLM-5-NVFP4 model was deployed using SGLang.

The bandwidth investigation was motivated by the stark performance gap between theoretical maximum throughput (309 tok/s) and actual throughput (10.36 tok/s). The diagnostic tool built in the broader session had revealed that simulated BF16 GEMMs and AllReduces accounted for only 8.9 ms of the 95 ms decode time, pointing to FP4 GEMM kernel overhead, MoE routing, and attention as the primary bottlenecks. The GPU bandwidth investigation confirmed that inter-GPU communication was not the dominant factor — the ~50 GB/s PCIe bandwidth was adequate for the communication patterns required by tensor parallelism, given that the compute kernels themselves were the main bottleneck.

Lessons Learned

The GPU bandwidth investigation offers several enduring lessons for anyone working with multi-GPU systems:

Measurement methodology matters enormously. The same benchmark, run with the same buffer size, can produce results that differ by three orders of magnitude depending on synchronization semantics, memory allocation patterns, and BAR1 cache state. A "definitive" measurement is only definitive if the methodology has been validated.

Synchronization is the silent killer of GPU benchmarks. torch.cuda.synchronize() without arguments synchronizes only the current device, not all devices involved in a transfer. Using explicit CUDA streams with proper scoping and synchronization is essential for accurate multi-GPU timing.

Physically impossible numbers are a diagnostic tool, not a result. When a benchmark reports 20,000+ GB/s on a PCIe Gen5 system with a theoretical maximum of ~64 GB/s, the correct response is not to report the number but to investigate why the measurement is wrong. The assistant's eventual recognition of this principle led to the methodological breakthrough.

GPU topology is not always what it seems. The "NODE" label in nvidia-smi topo -m does not guarantee NVLink connectivity. It simply indicates that two GPUs are on the same NUMA node, which could mean they share a PCIe root complex or are connected through a PCIe switch. The actual interconnect technology must be verified independently.

Production constraints require creative methodology. The need to measure bandwidth without disrupting a running server forced tradeoffs in buffer size, iteration count, and test duration. These constraints ultimately revealed the measurement artifacts that a clean-room benchmark might have hidden.

Conclusion

The GPU bandwidth investigation documented in this chunk is a case study in the challenges of performance measurement in complex multi-GPU systems. From the initial discovery of the GPU0 anomaly through the struggle with BAR1 artifacts to the methodological breakthrough with explicit CUDA streams, the investigation reveals the iterative, hypothesis-driven nature of diagnostic work. The final result — a clean, trustworthy bandwidth characterization showing ~50 GB/s same-NUMA and ~37 GB/s cross-NUMA — provided the foundation for understanding the system's communication performance and confirmed that the primary bottleneck lay elsewhere, in the FP4 GEMM kernels and attention mechanisms.

For the broader deployment effort, this investigation was a critical branch point. By definitively characterizing the interconnect performance and ruling out communication as the dominant bottleneck, it cleared the way for deeper analysis of the model's computational kernels. The phantom bandwidth numbers, the GPU0 mystery, and the NCCL configuration quirks were all resolved through systematic, methodical investigation — a process that exemplifies the rigor required for high-performance ML systems engineering.## References

[1] "The Anomaly in the Numbers: Diagnosing GPU-to-GPU Bandwidth on an 8-GPU NVLink System" — Analysis of message 3, the pivotal diagnostic round that revealed the GPU0 bandwidth anomaly.

[2] "Probing the Interconnect: How a Single Diagnostic Message Revealed the GPU Topology Bottleneck in an 8-GPU ML Server" — Examination of message 2 and the initial topology discovery.

[3] "The Lone Outlier: Diagnosing a GPU with Broken P2P Bandwidth" — Deep dive into message 5 and the systematic testing of GPU0's asymmetric bandwidth.

[4] "Probing the Interconnect: A Diagnostic Round in GPU Communication Analysis" — Analysis of message 1, the first round of NCCL and P2P checks.

[5] "The Phantom Bandwidth: Diagnosing GPU P2P Measurement Artifacts in an 8-GPU ML Server" — Examination of message 8 and the struggle with 20,000+ GB/s measurement artifacts.

[6] "The Moment of Clarity: Unmasking GPU Bandwidth Artifacts in a Multi-GPU Inference System" — Analysis of message 7 and the OOM error that forced methodological reflection.

[7] "Measuring Inter-GPU Bandwidth in a Running Inference Server: A Diagnostic Deep Dive" — Comprehensive analysis of message 0, the user's initial diagnostic request.

[8] "The BAR1 Mirage: Diagnosing Phantom P2P Bandwidth in an 8-GPU ML Server" — Analysis of message 9 and the discovery of BAR1 mapping as the source of phantom bandwidth.

[9] "The Pivot to Log Analysis: Diagnosing NCCL Transport Without Disrupting a Running Server" — Examination of message 12 and the shift to non-invasive NCCL diagnostics.

[10] "The Moment of Insight: Unmasking GPU P2P Bandwidth Measurement Artifacts in a Multi-GPU ML System" — Analysis of message 6 and the first clean 256 MB measurement.

[11] "The GPU0 Anomaly: Uncovering Asymmetric P2P Bandwidth in an 8-GPU Inference Server" — Analysis of message 4 and the comprehensive all-pairs bandwidth matrix.

[12] "Unraveling the GPU0 Anomaly: How Explicit CUDA Streams Revealed the True P2P Bandwidth" — Analysis of message 17, the methodological breakthrough with explicit CUDA streams.

[13] "The GPU0 Anomaly: Tracing a P2P Bandwidth Discrepancy Through BAR1 Memory Analysis" — Analysis of message 16 and the BAR1 usage comparison across GPUs.

[14] "Tracing the PCIe Topology: Unraveling GPU0's Mysterious P2P Bottleneck" — Analysis of message 13 and the IOMMU group investigation.

[15] "The NCCL_P2P_LEVEL Revelation: Diagnosing GPU Communication Bottlenecks in an 8-GPU ML Server" — Analysis of message 10 and the NCCL environment variable discovery.

[16] "When NCCL Debugging Hits a Wall: A Multiprocessing Spawn Failure in GPU Bandwidth Investigation" — Analysis of message 11 and the failed multiprocessing NCCL test.

[17] "Diagnosing GPU Interconnect Bandwidth: How a Measurement Artifact Led to a Definitive Performance Report" — Analysis of message 18, the final comprehensive bandwidth report.

[18] "The Mystery of GPU0: Diagnosing Asymmetric P2P Bandwidth in an 8-GPU Inference Server" — Analysis of message 14 and the ACS/PCIe root port investigation.

[19] "The Moment of Clarity: Verifying NCCL_P2P_LEVEL in a Multi-GPU Bandwidth Investigation" — Analysis of message 15 and the NCCL_P2P_LEVEL semantics verification.