The Optimization Campaign: Testing Boundaries on PCIe-Bound Blackwell GPUs

Introduction

The path from a working ML environment to a high-performance inference deployment is rarely a straight line. In this chunk of the opencode session, the assistant and user found themselves deep in the trenches of distributed GPU optimization, systematically testing and eliminating one allreduce approach after another on an unusual hardware configuration: 8× RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink. What emerged from this campaign was not a single silver bullet, but a nuanced understanding of the hardware's constraints, a valuable 9% throughput improvement through memory configuration tuning, and a strategic pivot toward upgrading CUDA to version 13 to unlock Blackwell-native optimizations.

This article synthesizes the full arc of the optimization work in this chunk, drawing on the detailed message-level analyses in [1], [2], and [3] to tell the broader story of how the assistant and user navigated a landscape of dead ends, partial successes, and promising new directions.

The Hardware Reality: PCIe as the Bottleneck

The machine at the center of this work is a beast: 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each a powerhouse of compute, connected via PCIe Gen5. But there is a catch — no NVLink. In the world of distributed ML inference, NVLink provides direct GPU-to-GPU communication with far lower latency and higher bandwidth than PCIe. Without it, every allreduce operation — the fundamental communication primitive that synchronizes gradients and activations across GPUs during tensor-parallel inference — must traverse the PCIe bus.

This is not merely a performance concern; it is a structural constraint that shapes the entire optimization strategy. As documented in the optimization plan file eagle-fast-verify.md (see [3]), the EAGLE-3 verify pass on this hardware takes approximately 30ms per cycle, with 97% of that time consumed by communication. The root cause is stark: 122 NCCL allreduces per forward pass (61 layers × 2 each), each carrying only 42 KB of data but incurring 150–300 µs of fixed NCCL latency. When message sizes are small, latency dominates over bandwidth, and PCIe Gen5's impressive throughput cannot compensate for the per-operation overhead.

The baseline throughput without speculative decoding was 82 tok/s. For EAGLE-3 to be worthwhile, the verify pass would need to be faster than the baseline generation rate. The optimization plan set an ambitious best-case target: reducing verify time to 10–12 ms, enabling 167+ tok/s — a 103% improvement. Achieving this would require attacking the communication bottleneck from multiple angles.

Systematic Elimination: Testing the Optimization Priority List

The optimization plan (analyzed in depth in [2]) enumerated seven approaches ranked by estimated effort and impact. The assistant worked through the most promising candidates systematically, and the results were largely negative — but each failure produced valuable diagnostic information.

FlashInfer Allreduce Fusion for SM120

The first approach was deceptively simple: a two-line change to enable FlashInfer's allreduce fusion for SM120 (Blackwell) GPUs. The estimated savings were 2–8 ms — a meaningful but not transformative improvement. However, the attempt failed immediately because FlashInfer's JIT compiler does not support the SM120 architecture. This was not a fundamental limitation of the approach, but a practical one: the software ecosystem had not yet caught up to the hardware. Blackwell GPUs were simply too new for FlashInfer's compilation pipeline.

Custom Allreduce Kernel on PCIe

The second approach was more ambitious: force the custom allreduce kernel to work on PCIe by removing the gatekeeping code that disables it for systems with more than two PCIe-connected GPUs. The estimated savings were 10–18 ms — enough to make speculative decoding profitable. But when tested, the custom kernel produced only 38 tok/s, more than 2× slower than NCCL's baseline performance.

The reason was PCIe bus contention. The custom allreduce kernel uses an all-to-all communication pattern, which requires every GPU to communicate with every other GPU simultaneously. On a system with NVLink, this is efficient because the NVLink topology provides dedicated point-to-point connections. On PCIe, however, all traffic must share the same bus, creating massive contention. The all-to-all pattern saturates the PCIe bus with overlapping transfers, and the resulting congestion destroys throughput. NCCL's Ring algorithm, by contrast, uses a carefully orchestrated sequence of point-to-point transfers that avoids bus contention — which is why it performed better despite its per-operation latency overhead.

This result was particularly instructive because it confirmed a counterintuitive insight: on PCIe-connected systems, the simpler, more structured communication pattern (Ring) can outperform a theoretically more efficient pattern (all-to-all) because it avoids the physical limitations of the shared bus.

Torch Symmetric Memory

The third approach was Torch symmetric memory, a feature that allows GPUs to directly access each other's memory without explicit communication. The estimated savings were 5–10 ms. But this approach also failed because SM120 is not in PyTorch's architecture lookup table for symmetric memory support. Like FlashInfer, PyTorch's software stack had not yet been updated to recognize Blackwell GPUs.

Expert Parallelism with FlashInfer A2A

The fourth approach was Expert Parallelism, which replaces MoE (Mixture of Experts) allreduces with all-to-all communication. This is a more fundamental architectural change that could eliminate many allreduces entirely. However, when tested with the FlashInfer A2A backend, it hit an assertion error followed by an out-of-memory (OOM) condition, making it completely non-functional.

A Partial Victory: The cuda-graph-max-bs Discovery

After four approaches had been eliminated, the optimization campaign appeared to be running out of options. But then came an unexpected breakthrough — not from a new communication algorithm, but from memory configuration.

The assistant discovered that reducing the --cuda-graph-max-bs parameter from 512 to 128 improved the baseline throughput from 82 tok/s to 89.5 tok/s — a 9% gain. The mechanism was straightforward: by limiting the CUDA graph batch size, the system freed GPU memory that was previously reserved for larger batch processing. This freed memory could then be allocated to the KV cache, which is the primary memory consumer during autoregressive inference. A larger KV cache means more tokens can be processed without recomputation, directly improving throughput.

This discovery is a classic example of the kind of optimization that only emerges from hands-on experimentation. No theoretical analysis would have predicted that reducing a batch size parameter would improve throughput, because the relationship is mediated by the complex interplay between CUDA graph caching, memory allocation, and KV cache utilization. It took actual measurement on the real hardware to uncover this lever.

However, even with this improvement, EAGLE-3 speculative decoding still only reached 54.1 tok/s — well below the 89.5 tok/s baseline. The verify pass bottleneck (~30ms for 122 NCCL allreduces) remained unresolved, and the 9% baseline improvement was not enough to make speculative decoding profitable. The fundamental communication problem persisted.

The Strategic Pivot: CUDA 13 and Blackwell-Native Optimizations

With four approaches eliminated and one partial success in hand, the user and assistant faced a strategic decision. The pattern across the failures was clear: every approach that failed did so because the software ecosystem did not yet support SM120 (Blackwell). FlashInfer's JIT compiler lacked SM120 support. PyTorch's symmetric memory lookup table lacked SM120. The custom allreduce kernel had been tested but was fundamentally limited by PCIe contention.

The common thread was CUDA version. The system was running CUDA Toolkit 12.8, which predates Blackwell support. CUDA 13, however, has native SM120 support. If the system could be upgraded to CUDA 13, all of the dead ends might become viable.

The user proposed this upgrade, and the assistant investigated its feasibility. The results were promising: the NVIDIA driver already supported CUDA 13.1 (version 590.48.01), but the toolkit was still at 12.8. Further research revealed that PyTorch nightly builds provide cu130 wheels, sgl-kernel has a dedicated cu130 index, and FlashInfer also supports CUDA 13. This meant the upgrade path was not only possible but well-supported by the ecosystem.

The implications were significant. With CUDA 13, FlashInfer allreduce fusion could work on SM120. Torch symmetric memory would recognize Blackwell GPUs. The custom allreduce kernel might be compiled with Blackwell-specific optimizations. Even the Expert Parallelism approach might function correctly with updated CUDA libraries. The CUDA 13 upgrade could unblock multiple optimization paths simultaneously, potentially providing the cumulative improvements needed to make speculative decoding profitable.

The Optimization Plan as a Living Document

Throughout this campaign, the optimization plan document — eagle-fast-verify.md — served as the central artifact tracking progress. As analyzed in [1], the user's instruction to read this file was far more than a simple file retrieval; it was a strategic pause to reorient before proceeding. The document encoded the team's collective understanding of the problem, the prioritized approaches, and the performance targets.

After completing the experimental work described above, the assistant updated the optimization plan with the results. FlashInfer fusion was marked as "failed — SM120 not supported." The custom allreduce kernel was marked as "failed — 38 tok/s on PCIe." Torch symmetric memory was marked as "failed — SM120 not in lookup table." Expert Parallelism was marked as "failed — assertion error and OOM." The cuda-graph-max-bs discovery was recorded as a successful optimization yielding 9% improvement. And the CUDA 13 upgrade was added as a new priority — not as a direct optimization, but as an enabler for all the other approaches that had been blocked by the CUDA version.

This process of updating the plan is itself a form of knowledge creation. The document evolves from a speculative roadmap into an empirical record, capturing not just what was tried but what was learned. Future optimization efforts — whether by the same team or by others facing similar hardware configurations — can benefit from these documented results.

The Broader Lessons

The optimization campaign in this chunk offers several lessons for distributed ML inference:

Software maturity matters as much as hardware capability. The RTX PRO 6000 Blackwell GPUs are cutting-edge hardware, but the software ecosystem (FlashInfer, PyTorch, CUDA toolkit) lags behind. The most powerful optimizations are useless if the software stack cannot support them. The CUDA 13 upgrade strategy recognizes this: sometimes the best path forward is to upgrade the foundation rather than hack around its limitations.

PCIe-connected multi-GPU systems have unique constraints. Most high-end ML deployments use NVLink or NVSwitch to connect GPUs, and the optimization ecosystem is built around that assumption. PCIe-connected systems face bus contention issues that are poorly handled by communication patterns designed for NVLink topologies. The custom allreduce kernel's failure on PCIe is a cautionary tale: an algorithm that works well on one interconnect can be catastrophically worse on another.

Small configuration changes can yield surprising gains. The cuda-graph-max-bs discovery is a reminder that not all optimizations come from algorithmic innovation. Sometimes the biggest wins come from understanding how the runtime system allocates resources and finding the right configuration knobs.

Negative results are valuable data. Every failed optimization in this campaign produced diagnostic information that guided the next steps. The FlashInfer failure revealed the CUDA version dependency. The custom allreduce failure revealed the PCIe contention pattern. The Torch symmetric memory failure confirmed the ecosystem gap. Without these negative results, the team might have pursued the wrong approaches for much longer.

Conclusion

The optimization campaign documented in this chunk is a microcosm of the challenges facing ML engineers working at the frontier of hardware and software. The assistant and user tested four major optimization approaches, found three blocked by software ecosystem limitations and one fundamentally incompatible with the PCIe interconnect, discovered a modest but real improvement through memory configuration tuning, and ultimately pivoted to a CUDA 13 upgrade strategy that could unblock multiple paths simultaneously.

The work is not complete — the CUDA 13 upgrade and subsequent re-testing of the blocked approaches lies ahead. But the foundation has been laid: a systematic understanding of what works, what doesn't, and why. The optimization plan document has been updated with empirical results. The strategic direction is clear. And the lessons learned — about PCIe contention, software maturity, and the value of negative results — will inform not just this deployment but future work on similar hardware configurations.

In the high-stakes world of large model inference, where milliseconds translate directly into user experience and operational cost, this kind of systematic optimization work is essential. The path to 167 tok/s is not a straight line, but it is now better understood than it was at the start of this chunk.