The Systematic Elimination: Methodical Optimization of Blackwell GPUs and the Pivot to CUDA 13

Introduction

In the high-stakes world of large language model inference optimization, progress rarely follows a straight line. It is a landscape of dead ends, false starts, unexpected interactions, and the occasional serendipitous discovery that keeps the whole endeavor moving forward. Segment 35 of this opencode session captures exactly such a journey: a systematic optimization campaign for the Kimi-K2.5 model—a 1-trillion-parameter Mixture-of-Experts architecture—running on eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 without NVLink.

The goal was ambitious: make EAGLE-3 speculative decoding profitable by reducing the verify-pass bottleneck. What unfolds across this segment is a masterclass in methodical engineering. The assistant tests one optimization hypothesis after another, each failing for a different reason—architecture incompatibility, PCIe bus contention, missing software support, unexpected memory interactions—until a single positive discovery (reducing --cuda-graph-max-bs improves baseline throughput by 9%) and a user-proposed pivot (upgrading CUDA to version 13) open a new path forward. This article traces that journey, examining the reasoning, assumptions, and methodology that turned a series of failures into actionable knowledge.

The Problem: When Speculation Hurts Throughput

The optimization campaign was driven by a stark performance gap. The baseline throughput for Kimi-K2.5 INT4 on the 8× Blackwell system had been boosted to approximately 89.5 tokens per second (tok/s) through earlier tuning. But EAGLE-3 speculative decoding—a technique that uses a lightweight draft model to predict multiple tokens ahead of the target model—was achieving only 54.1 tok/s, a 40% regression from the baseline. The bottleneck had been meticulously identified: the "verify pass," where the target model checks the draft tokens, consumed approximately 30 milliseconds per cycle. Of that, roughly 24 milliseconds was pure NCCL allreduce latency.

The root cause was architectural. Kimi-K2.5 has 61 layers, each performing two allreduce operations (one for attention, one for the Mixture-of-Experts routing), totaling 122 allreduces per forward pass. Each allreduce operates on tiny tensors—just 42 KB—but NCCL's Ring LL protocol imposes a latency floor of roughly 150–300 microseconds per operation regardless of tensor size. The math was brutal: 122 × ~200 µs ≈ 24 ms of pure communication overhead, with actual compute accounting for only 5–8 ms. The GPUs were spending 70% of their time idle, waiting for NCCL synchronization across the PCIe bus.

The optimization plan, documented across multiple improvement documents, ranked several approaches by expected impact. The assistant began working through them systematically, testing each hypothesis against the unforgiving reality of the hardware.

The Hardware Reality: PCIe-Connected Blackwell GPUs

To understand why this optimization campaign was so challenging, one must first understand the hardware configuration. The system consisted of eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture, compute capability 12.0) connected via PCIe Gen5, with no NVLink interconnect. This is a fundamentally different topology from the NVLink-connected DGX systems that most large-scale inference optimizations target.

The Blackwell architecture was extremely new at the time of this session. Many libraries—FlashInfer's JIT compiler, PyTorch's symmetric memory module, and various custom kernels—had not yet added support for SM120. This created a recurring pattern: promising optimization approaches would fail not because of algorithmic flaws but because the software ecosystem hadn't caught up to the hardware.

The PCIe topology added another layer of constraint. PCIe Gen5 provides roughly 32 GB/s bidirectional bandwidth per GPU, shared across all communication. Unlike NVLink, which provides dedicated high-speed GPU-to-GPU links, PCIe requires all cross-GPU traffic to go through the CPU's root complex. This makes communication patterns that work well on NVLink—such as the all-to-all pattern used by custom allreduce kernels—catastrophically slow on PCIe due to bus contention.

The Graveyard of Optimization: Six Approaches That Failed

The assistant systematically tested six distinct approaches to reducing the allreduce overhead, and nearly all of them failed. The pattern of failure is instructive, revealing both the current limitations of the software ecosystem and the fundamental constraints of PCIe-connected multi-GPU systems.

Dead End 1: FlashInfer Allreduce Fusion on SM120

The highest-priority approach was FlashInfer allreduce fusion, a technique that promised to combine the allreduce, residual addition, and RMS normalization into a single fused kernel using flashinfer's TRTLLM IPC-based allreduce. If successful, this could bypass NCCL entirely for small tensors, potentially reducing per-allreduce latency from ~200 µs to ~30–50 µs.

The assistant applied two surgical code changes to enable this fusion on SM120 architecture, which was previously restricted to SM90 (Hopper) and SM100. In communicator.py, the condition _is_sm90_supported or _is_sm100_supported was extended to include _is_sm120_supported. In server_args.py, the auto-enable logic was similarly modified.

The server was launched combining these fusion changes with an experimental NCCL tuning configuration (fewer channels, smaller buffer). It never became ready. The assistant's log inspection revealed the root cause: flashinfer's JIT compiler reported "No supported CUDA architectures found for major versions [9, 10]". SM120 corresponds to CUDA compute capability 12.0, and flashinfer's TRTLLM communication module only knew about SM 9.x and 10.x. Blackwell's SM120 was simply not in the lookup table.

This was a hard dead end. No amount of configuration tuning could fix it—the flashinfer library itself would need to be updated by its maintainers. The assistant cleanly reverted the two code changes using sed commands, verified the reverts by grepping for remaining SM120 references, and documented the failure in the optimization plan.

Dead End 2: Custom Allreduce Kernel on PCIe

With flashinfer fusion eliminated, the assistant pivoted to the custom allreduce kernel. The custom allreduce kernel in SGLang normally requires NVLink for GPU-to-GPU communication, but the assistant hypothesized that it could be forced to work over PCIe by modifying the architecture checks in custom_all_reduce.py.

This required a deeper investigation. The assistant read the Python wrapper code, traced the kernel compilation pipeline, and discovered that the custom allreduce kernel is pre-compiled for specific architectures. SM120 (Blackwell) was not among them. The assistant found an escape hatch—the SGLANG_CUSTOM_ALLREDUCE_ALGO environment variable—that could force the kernel to use a different algorithm. After patching the Python gates to allow PCIe usage, the assistant launched a server with the custom allreduce enabled.

The result was catastrophic for performance: 38 tok/s, more than 2× slower than the NCCL baseline. The all-to-all communication pattern of the custom kernel caused massive PCIe bus contention. With 8 GPUs sharing the PCIe fabric, the simultaneous all-to-all transfers saturated the bus, creating a bottleneck far worse than NCCL's Ring protocol. This experiment definitively ruled out the custom allreduce approach for PCIe-connected Blackwell GPUs.

Dead End 3: NCCL Tree Algorithm

The NCCL Tree algorithm was ruled out because it is incompatible with CUDA graphs, which SGLang relies on for efficient GPU kernel launch. This was a framework conflict rather than a hardware issue, but equally terminal. CUDA graphs capture repetitive computation patterns to reduce kernel launch overhead, and NCCL Tree cannot operate within that framework.

Dead End 4: Torch Symmetric Memory

Priority 5 in the optimization plan was torch symmetric memory (--enable-torch-symm-mem), a PyTorch feature that enables efficient peer-to-peer GPU communication. The assistant attempted to enable it, but the server failed during initialization. The cause: SM120 (Blackwell) is not in PyTorch's architecture lookup table for symmetric memory support. Like flashinfer fusion, this was a software compatibility issue that could only be resolved by upstream changes to PyTorch.

Dead End 5: Expert Parallelism with Flashinfer A2A

Priority 6 was Expert Parallelism (EP), which distributes MoE experts across GPUs and uses all-to-all communication instead of allreduce. The assistant attempted to enable EP with the flashinfer A2A backend (--moe-a2a-backend flashinfer). This hit an assertion error during initialization, followed by an out-of-memory condition. The flashinfer A2A implementation had compatibility issues with the specific model configuration, making it non-functional.

Dead End 6: NCCL Fewer Channels

The assistant then turned to a hypothesis grounded in NCCL internals: reducing the number of NCCL communication channels from 16 to 1–2 could reduce per-allreduce latency for tiny 42 KB tensors. The theory was sound—for latency-bound small-message allreduce, fewer channels mean less per-operation overhead. The assistant configured NCCL_MIN_NCHANNELS=1, NCCL_MAX_NCHANNELS=2, NCCL_BUFFSIZE=131072 (down from 16 MB), and NCCL_NTHREADS=64 (down from 512).

The server launched but crashed during memory pool initialization with a RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. This was deeply puzzling because the available GPU memory after weight loading was 21.78 GB—slightly more than the 21.71 GB available in the working baseline run. The assistant was confronting what one article in this segment called "The Paradox of Plenty": more free memory, yet an OOM error.

The assistant's diagnostic reasoning at this point is worth examining in detail. The error message said to increase --mem-fraction-static, but the assistant recognized that this would make the problem worse, not better. The memory fraction controls how much GPU memory is reserved for KV cache; if 0.55 × 96 GB = 52.8 GB was being requested but only 21.78 GB was available, the fraction was too high, not too low. The error message was misleading.

The assistant read the source code of model_runner_kv_cache_mixin.py around the init_memory_pool function to understand the max_total_num_tokens calculation. The code showed that this value was computed from a tensor result, and if it came out as zero or negative, the error was raised. The assistant hypothesized that the NCCL buffer size change was altering the memory layout in a way that affected this calculation—perhaps NCCL pre-allocated its buffers differently with fewer channels, consuming memory that the KV cache pool expected to be available.

After several rounds of investigation—checking GPU memory with nvidia-smi, comparing log outputs, tracing the memory calculation formula—the assistant made a strategic decision. The fewer-channels approach was not promising enough to justify extensive debugging. The expected benefit was marginal (a few percent at best), and the OOM interaction suggested a fundamental incompatibility between the reduced channel count and SGLang's memory management. The assistant reverted to the known-working NCCL configuration and moved on.

A Serendipitous Discovery: The 9% Baseline Improvement

Amidst this cascade of failures, a genuine positive result emerged. The assistant had been using --cuda-graph-max-bs 512 (the default) in all baseline runs. During one experiment, the assistant reduced this to 128 to free GPU memory. The result was unexpected: baseline throughput jumped from 82 to 89.5 tok/s—a 9% improvement.

The mechanism was clear in retrospect. CUDA graphs capture repetitive computation patterns to reduce kernel launch overhead, but each captured graph consumes GPU memory. With --cuda-graph-max-bs 512, the graphs captured operations for batch sizes up to 512, consuming significant memory that could otherwise be used for KV cache. Reducing to 128 freed enough memory to expand the KV cache, allowing larger batch sizes and higher throughput.

This discovery was significant for two reasons. First, it provided an immediate, no-cost throughput improvement—no code changes, no NCCL tuning, no new libraries. Second, it demonstrated that memory allocation patterns were a critical lever for performance on this hardware configuration, a lesson that would prove valuable in subsequent debugging.

Yet the core problem remained unsolved. Even at 89.5 tok/s, EAGLE-3 speculation was still at 54.1 tok/s—a 40% deficit. The verify pass bottleneck had not been touched by the baseline improvement.

The Allreduce Coalescing Insight

While waiting for a server to load during one of the experiments, the assistant had a moment of strategic reflection. The real bottleneck was 122 NCCL allreduces per forward pass, each taking approximately 200 microseconds. Every approach so far had been a drop-in replacement for NCCL allreduce, attempting to shave microseconds off each individual call.

The assistant then articulated a fundamentally different idea: what if instead of making each allreduce faster, you reduced the number of allreduces? The arithmetic was compelling: 122 separate allreduces of 42KB each could be batched into 2 large allreduces of 2.5MB each (by grouping all 61 attention allreduces together and all 61 MoE allreduces together). The projected savings were dramatic: from 122 × 200µs = 24.4ms down to 2 × 200µs = 400µs—a 60× improvement in communication overhead.

The assistant recognized that this would require a fundamental architectural change to the model forward pass, but the insight itself was valuable: the problem was not just that each allreduce was slow, but that there were too many of them. This reframing opened the door to Expert Parallelism (which replaces MoE allreduces with all-to-all communication) and, ultimately, to the CUDA 13 upgrade.

The Moment of Reckoning: Six Paths Collapse

After all six approaches had been tested and eliminated, the assistant produced a summary that stands as a model of intellectual honesty in engineering. The message listed each approach, its result, and the reason for failure in a structured table. Five approaches were labeled "FAILED," one was "Inconclusive," and the only unambiguous achievement was the 9% baseline improvement.

The assistant then listed the remaining viable options: install DeepEP, install MSCCL++, retry NCCL fewer channels with correct settings, attempt allreduce coalescing (an invasive code change), train more drafter data to improve acceptance rates, or simply accept the 89.5 tok/s baseline as the best achievable result for this hardware.

This moment of reckoning was crucial. By formally acknowledging that an entire family of optimization approaches had been exhausted, the assistant created the intellectual space for a fundamentally different strategy. The todo list was updated with each approach marked as "DEAD END" with a parenthetical explanation of why. The optimization plan document was updated with all experimental results.

The Common Thread: SM120 Support Gaps

Looking across all six failed approaches, a striking pattern emerges. Every failure can be traced to the same root cause: the CUDA 12.8 toolchain and its associated libraries lack native support for the Blackwell SM120 architecture.

The Pivot: CUDA 13

Then came the user's question—eight words that would reshape the entire campaign: "Should we update cuda to 13 with more proper support for sm120?"

This question was not a random suggestion. The user had absorbed the experimental results and recognized the pattern: two of the most promising approaches (FlashInfer fusion and torch symmetric memory) had failed specifically because SM120 wasn't recognized by the CUDA 12.8 toolchain. If the root cause was incomplete architecture support, then upgrading to the native toolkit for Blackwell could unblock those paths without requiring any additional algorithmic innovation.

The assistant's response was immediate and thorough. It validated the user's intuition, provided the technical rationale (CUDA 12.8 has "early/partial" SM120 support while CUDA 13 is the native toolkit), and connected the upgrade to the specific dead ends encountered. Then it gathered evidence: checking the current CUDA version, the installed toolkit directories, and the operating system.

The investigation revealed encouraging findings. The NVIDIA driver (version 590.48.01) already supported CUDA 13.1—only the toolkit was behind at 12.8.1. PyTorch nightly offered cu130 wheels at a dedicated download URL. sgl-kernel had a dedicated cu130 install index, and a recently-fixed issue suggested Blackwell compatibility was now functional. flashinfer was listed in the cu130 nightly index. The upgrade path was real.

The user's response to this research was an empty message—a silent nod that served as authorization to proceed. The assistant interpreted this as a green light and began planning the upgrade: install CUDA 13 toolkit, create a new virtual environment, rebuild SGLang from source, test the baseline, and then retry the failed optimizations one by one.

What This Campaign Teaches About Systematic Optimization

The work in this segment offers several enduring lessons for anyone engaged in systems optimization.

Hypothesis-driven experimentation. Every approach was grounded in a specific, testable hypothesis about why it would improve performance. The fewer-channels hypothesis was based on NCCL latency-bandwidth tradeoffs for small tensors. The custom allreduce hypothesis was based on GPU IPC shared memory having lower latency than NCCL protocols. Each hypothesis was tested, and the results were used to update the plan.

Clean state management. After every failed experiment, the assistant reverted code changes, restored configuration files, and verified the system was in a known-good state before proceeding. This prevented cascading failures where one experiment's residue contaminated the next.

Variable isolation. The assistant learned from the early mistake of combining two changes (flashinfer fusion + NCCL tuning) in a single experiment. After that, each variable was tested independently, making it possible to attribute failures to specific causes.

Cost-benefit triage. Not every dead end was investigated exhaustively. The assistant evaluated the expected benefit of each approach against the debugging cost and abandoned approaches that weren't worth the time. The fewer-channels experiment was abandoned after one OOM failure because the expected gain was small.

Negative results as knowledge. Every failed experiment produced valuable information. The flashinfer fusion failure revealed that SM120 support was missing from flashinfer's JIT compiler. The custom allreduce failure revealed that PCIe bus contention made all-to-all communication patterns worse than Ring. These negative results prevented future wasted effort and narrowed the search space.

The pivot as a skill. Perhaps the most important pattern is the willingness to pivot. The assistant did not become attached to any particular approach. When flashinfer fusion failed, it moved to custom allreduce. When custom allreduce failed, it moved to NCCL tuning. When NCCL tuning hit an OOM, it documented the results and pivoted to CUDA 13. This flexibility is essential in optimization work, where most ideas don't work and the key to success is finding the few that do.

Documentation is essential. Throughout the campaign, the assistant maintained a todo list, an optimization plan document, and detailed benchmark results. This discipline ensured that no effort was wasted on re-testing dead ends, and that the reasoning behind each decision was preserved for future reference.

Conclusion

Segment 35 of this opencode session is a case study in methodical optimization work. The assistant and user systematically tested six approaches to reducing allreduce overhead on a challenging hardware configuration—PCIe-connected Blackwell GPUs—and found that five failed outright while one was inconclusive. The only unambiguous success was a 9% baseline improvement from a configuration parameter change: reducing --cuda-graph-max-bs from 512 to 128.

But the true value of the campaign was not in the results but in the knowledge created. The team now knows exactly which optimizations do not work on this hardware, and why. They know that NCCL Ring is the best available allreduce strategy for PCIe-connected Blackwell GPUs. They know that the software ecosystem has not yet caught up to the hardware. And they have a clear path forward: upgrade CUDA to version 13, which has native SM120 support, and retry the optimizations that were blocked by architecture detection issues.

The pivot to CUDA 13 represents a fundamental shift in strategy—from working within the constraints of CUDA 12.8 to upgrading the foundation itself. Whether this upgrade ultimately unblocks the Blackwell-native optimizations and makes EAGLE-3 speculative decoding profitable remains to be seen. But the campaign has already demonstrated something equally valuable: the discipline to systematically eliminate dead ends, the insight to recognize when the problem is the foundation rather than the superstructure, and the willingness to make a strategic pivot when the evidence demands it.

In the end, the most important skill demonstrated in this segment is not the ability to write code or tune parameters—it is the ability to learn from failure, update one's mental model, and pivot to a more promising direction. That is the essence of systematic optimization, and it is what separates effective engineering from random experimentation.

References

[1] Segment 35, Chunk 0: "The Systematic Elimination: How an Optimization Campaign for Blackwell GPUs Navigated Dead Ends to Discover the CUDA 13 Path" — Detailed account of the FlashInfer fusion, custom allreduce, torch symmetric memory, and EP failures, plus the 9% baseline improvement discovery.

[2] Segment 35, Chunk 1: "The Systematic Elimination of Allreduce Optimizations: A Methodical Campaign Across Dead Ends and the Pivot to CUDA 13" — Comprehensive analysis of all six optimization approaches, the allreduce coalescing insight, and the CUDA 13 upgrade feasibility research.