Systematic Optimization of Allreduce Strategies for Blackwell GPUs on PCIe: From Dead Ends to a CUDA 13 Pivot

Introduction

In the high-stakes world of large-scale ML inference, the performance of collective communication operations like allreduce can determine whether a deployment meets its throughput targets or falls short. When eight NVIDIA RTX PRO 6000 Blackwell GPUs are connected via PCIe rather than NVLink, the communication topology introduces unique bottlenecks that challenge conventional optimization strategies. This article chronicles a systematic investigation into allreduce optimization approaches for precisely such a configuration, documenting a journey through multiple dead ends, a serendipitous throughput discovery, and a strategic pivot toward upgrading the CUDA toolkit to unlock Blackwell-native capabilities.

The work described here took place within a broader session focused on deploying the GLM-5-NVFP4 model using SGLang on an 8-GPU Ubuntu 24.04 system. The preceding segments had already resolved complex build issues—installing NVIDIA drivers 590.48.01, setting up CUDA Toolkit 13.1, wrestling with flash-attn compilation by reducing parallel jobs from 128 to 20, and stabilizing a compatible stack of PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1. With the environment hardened, the team turned to the critical question: how can allreduce performance be optimized for this specific hardware configuration?

The Optimization Landscape: Why Allreduce Matters

Allreduce is the fundamental collective operation that enables tensor parallelism in distributed inference. When a transformer layer processes a batch of tokens across multiple GPUs, each GPU holds a shard of the model parameters. After computing its partial results, each GPU must sum its contributions with those from all other GPUs and distribute the result back—this is the allreduce operation. In a typical transformer forward pass, dozens of allreduce calls are made per layer, and their cumulative latency directly impacts end-to-end throughput.

For the 8× RTX PRO 6000 Blackwell system, the challenge is amplified by the PCIe interconnect. Unlike NVLink, which provides high-bandwidth, low-latency GPU-to-GPU communication, PCIe introduces significant contention and higher latency. The PCIe bus is a shared resource—when all eight GPUs attempt to communicate simultaneously, the bus becomes a bottleneck, and the all-to-all communication pattern of allreduce exacerbates this contention. Understanding which allreduce backend performs best under these constraints was the central question driving this investigation.

The Systematic Elimination of Optimization Approaches

The assistant's approach to optimization was methodical: rather than guessing which technique might work, it systematically tested each candidate approach, documented the results, and eliminated the non-viable options. This scientific methodology ensured that no stone was left unturned and that the team could make informed decisions about where to invest further effort.

FlashInfer Allreduce Fusion: Blocked by Architecture Support

The first candidate was FlashInfer's allreduce fusion, which promised to combine multiple allreduce operations into a single, optimized kernel invocation. This technique is particularly valuable for transformer inference, where multiple allreduce calls occur in quick succession during attention and MLP computations.

However, the test immediately hit a wall: FlashInfer's JIT compiler does not support the SM120 architecture, which is the compute capability of Blackwell GPUs (compute capability 12.0). The JIT compiler is responsible for generating optimized CUDA kernels at runtime based on the specific tensor shapes and hardware configuration. Without SM120 support, FlashInfer fusion was simply unavailable. This was not a performance issue—it was a compatibility issue that could only be resolved by either upgrading FlashInfer to support Blackwell or finding an alternative approach.

Custom Allreduce Kernel on PCIe: 2× Slower Than NCCL

The second candidate was SGLang's custom allreduce kernel, which uses shared memory to perform allreduce without invoking NCCL. This kernel is designed to be faster than NCCL for small-to-medium tensors by avoiding NCCL's internal overhead and leveraging GPU shared memory for fast intra-node communication.

The assistant forced the custom allreduce kernel to operate on the PCIe-connected system and measured the results. The throughput was a mere 38 tok/s—more than 2× slower than the NCCL baseline. The root cause was massive PCIe bus contention. The custom allreduce kernel uses an all-to-all communication pattern where each GPU sends data to every other GPU simultaneously. On a PCIe bus, this pattern creates a perfect storm of contention: the shared bus must handle 8×7 = 56 simultaneous data transfers, each competing for bandwidth. The result is catastrophic throughput degradation.

This finding was particularly instructive because it demonstrated that the custom allreduce kernel, while effective on NVLink-connected systems where dedicated GPU-to-GPU links avoid bus contention, is fundamentally unsuitable for PCIe topologies. The optimization technique that works best on one interconnect can be the worst on another.

Torch Symmetric Memory: Another SM120 Roadblock

The third candidate was PyTorch's symmetric memory feature, which allows GPUs to directly read from each other's memory without explicit communication operations. This technique can dramatically reduce allreduce latency by eliminating the need for NCCL calls—each GPU simply reads the data it needs from the other GPUs' memory.

Unfortunately, this approach also failed due to Blackwell architecture support. SM120 is not present in PyTorch's architecture lookup table for symmetric memory, meaning the feature simply does not recognize Blackwell GPUs as compatible hardware. Like FlashInfer fusion, this was a compatibility issue rather than a performance issue, but the result was the same: the approach was non-functional.

Expert Parallelism with FlashInfer A2A: Assertion Errors and OOM

The fourth candidate was Expert Parallelism (EP) using FlashInfer's all-to-all (A2A) backend. Expert Parallelism is a technique for distributing the expert layers of mixture-of-experts (MoE) models across GPUs, and it relies on efficient A2A communication to route tokens to the correct expert.

The test encountered two failures: first, an assertion error in the FlashInfer A2A backend, indicating that the code path was not properly handling the Blackwell configuration; second, an out-of-memory (OOM) error, suggesting that the memory allocation strategy for the A2A buffers was not compatible with the GPU memory layout. Together, these failures made Expert Parallelism non-functional for this system.

The Serendipitous Discovery: Reducing cuda-graph-max-bs

Amid the string of dead ends, a significant positive discovery emerged. The assistant experimented with the --cuda-graph-max-bs parameter, which controls the maximum batch size for CUDA graph capture. CUDA graphs are a performance optimization that pre-records a sequence of GPU operations and replays them with minimal overhead, bypassing the CPU launch latency for each individual kernel.

The default value of 512 was consuming a large amount of GPU memory for graph storage, leaving less memory available for the KV cache—the critical memory structure that stores attention key-value pairs for the ongoing inference. By reducing --cuda-graph-max-bs from 512 to 128, the assistant freed GPU memory for KV cache, which allowed larger batch sizes and improved throughput.

The results were striking: baseline throughput improved from 82 tok/s to 89.5 tok/s—a 9% gain from a single parameter change. This discovery was particularly valuable because it required no code changes, no new backends, and no hardware modifications. It was a pure configuration optimization that could be applied immediately.

The EAGLE-3 Speculative Decoding Bottleneck

With the baseline throughput improved, the assistant turned to EAGLE-3 speculative decoding, a technique that uses a small draft model to predict multiple tokens in parallel, which are then verified by the main model. Speculative decoding can theoretically increase throughput by reducing the number of main model forward passes.

However, the EAGLE-3 implementation achieved only 54.1 tok/s, well below the baseline of 89.5 tok/s. The bottleneck was in the verify pass, where the main model must process the draft tokens and verify their correctness. This verification requires allreduce operations to synchronize the model's hidden states across GPUs, and the verify pass was performing approximately 122 NCCL allreduce calls, consuming roughly 30ms per verification step.

The root cause was the same PCIe contention that had plagued the custom allreduce kernel. Each NCCL allreduce call required communication across all eight GPUs over the shared PCIe bus, and 122 such calls created a massive cumulative latency that dwarfed the savings from speculative decoding. The assistant concluded that until the verify pass cost could be reduced—either by reducing the number of allreduce calls or by finding a faster communication backend—speculative decoding would remain unprofitable on this hardware.

Updating the Optimization Plan

Throughout this investigation, the assistant maintained a comprehensive optimization plan document that tracked all experimental results, including both successful and failed approaches. This document served as the team's shared knowledge base, ensuring that:

  1. No effort was duplicated: Each approach was documented with its results, so future investigations would not retest eliminated candidates.
  2. Patterns could be identified: The consistent theme of SM120 compatibility issues became apparent when viewing the results together.
  3. The pivot to CUDA 13 was data-driven: The decision to upgrade CUDA was supported by clear evidence that multiple optimization paths were blocked by architecture support limitations. The optimization plan document included detailed performance numbers, configuration parameters, error messages, and analysis for each tested approach. This systematic documentation transformed the investigation from a series of ad-hoc experiments into a structured research program.

The CUDA 13 Pivot: A Promising Direction

The user, reviewing the experimental results, proposed a strategic pivot: upgrade CUDA to version 13, which has native SM120 support. This single change could potentially unblock multiple dead ends simultaneously.

The assistant confirmed that the existing NVIDIA driver (590.48.01) already supports CUDA 13.1, but the installed toolkit was only version 12.8. Research into the upgrade path revealed encouraging findings:

Lessons Learned and Methodology

This investigation offers several lessons for ML infrastructure optimization:

1. Systematic elimination is more efficient than random experimentation. By testing each candidate approach in a structured manner and documenting results, the team avoided the common pitfall of repeatedly revisiting the same dead ends.

2. Architecture compatibility is a first-order concern. Multiple optimization approaches failed not because of performance issues but because of basic compatibility with the Blackwell architecture. Checking architecture support should be the first step in any optimization effort for new hardware.

3. PCIe topology fundamentally changes optimization priorities. Techniques that work well on NVLink-connected systems (custom allreduce kernels, symmetric memory) can be worse than the baseline on PCIe. The interconnect topology must be a primary consideration in optimization strategy.

4. Configuration optimization can yield surprising gains. The 9% improvement from reducing cuda-graph-max-bs demonstrates that significant performance gains can come from parameter tuning rather than code changes.

5. Speculative decoding is not universally beneficial. The profitability of speculative decoding depends on the ratio of verification cost to generation savings. On systems where communication is expensive, the verification cost can outweigh the benefits.

Conclusion

The systematic investigation into allreduce optimization for PCIe-connected Blackwell GPUs eliminated four candidate approaches (FlashInfer fusion, custom allreduce kernel, torch symmetric memory, and Expert Parallelism with flashinfer A2A), discovered a 9% throughput improvement from configuration tuning, identified the EAGLE-3 verify pass as the critical bottleneck for speculative decoding, and culminated in a strategic pivot to upgrade CUDA to version 13.

The CUDA 13 upgrade path offers the promise of unblocking multiple Blackwell-native optimizations simultaneously, potentially transforming the performance landscape for this system. Whether this promise is realized depends on the actual compatibility and performance of the upgraded stack, but the investigation has provided a clear, data-driven rationale for pursuing this direction.

For anyone working on ML inference optimization for Blackwell GPUs—especially on PCIe-connected systems—this investigation provides a roadmap of what to try, what to avoid, and where to focus effort. The combination of systematic experimentation, thorough documentation, and strategic pivoting offers a model for navigating the complex landscape of distributed communication optimization.