The Systematic Elimination: How an Optimization Campaign for Blackwell GPUs Navigated Dead Ends to Discover the CUDA 13 Path

Introduction

In the high-stakes world of large language model inference optimization, progress is rarely a straight line. It is a landscape of dead ends, false starts, unexpected interactions, and the occasional serendipitous discovery that keeps the whole endeavor moving forward. This article synthesizes a critical segment of a multi-day optimization campaign for the Kimi-K2.5 model—a 1-trillion-parameter Mixture-of-Experts architecture—running on eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 without NVLink. The goal was ambitious: make EAGLE-3 speculative decoding profitable by reducing the verify-pass bottleneck from ~30 milliseconds to ~12–15 milliseconds.

What unfolds across this chunk is a masterclass in systematic engineering. The assistant tests one optimization hypothesis after another, each failing for a different reason—architecture incompatibility, PCIe bus contention, missing software support, unexpected memory interactions—until a single positive discovery (reducing --cuda-graph-max-bs improves baseline throughput by 9%) and a user-proposed pivot (upgrading CUDA to version 13) open a new path forward. This article traces that journey, examining the reasoning, assumptions, and methodology that turned a series of failures into actionable knowledge.

The Problem: When Speculation Hurts Throughput

The optimization campaign was driven by a stark performance gap. The baseline throughput for Kimi-K2.5 INT4 on the 8× Blackwell system was a respectable 82 tokens per second. But EAGLE-3 speculative decoding—a technique that uses a lightweight draft model to predict multiple tokens ahead of the target model—was achieving only 60 tok/s, a 27% regression from the baseline. The bottleneck had been meticulously identified: the "verify pass," where the target model checks the draft tokens, consumed approximately 30 milliseconds per cycle. Of that, roughly 24 milliseconds was pure NCCL allreduce latency.

The root cause was architectural. Kimi-K2.5 has 61 layers, each performing two allreduce operations (one for attention, one for the Mixture-of-Experts routing), totaling 122 allreduces per forward pass. Each allreduce operates on tiny tensors—just 42 KB—but NCCL's Ring LL protocol imposes a latency floor of roughly 150–300 microseconds per operation regardless of tensor size. The math was brutal: 122 × ~200 µs ≈ 24 ms of pure communication overhead, with actual compute accounting for only 5–8 ms. The GPUs were spending 70% of their time idle, waiting for NCCL synchronization across the PCIe bus.

The optimization plan, documented in eagle-fast-verify.md, ranked seven approaches by expected impact. The assistant began working through them systematically.

Dead End 1: FlashInfer Allreduce Fusion on SM120

The highest-priority approach was FlashInfer allreduce fusion, a technique that promised to combine the allreduce, residual addition, and RMS normalization into a single fused kernel using flashinfer's TRTLLM IPC-based allreduce. If successful, this could bypass NCCL entirely for small tensors, potentially reducing per-allreduce latency from ~200 µs to ~30–50 µs.

The assistant applied two surgical code changes to enable this fusion on SM120 (Blackwell) architecture, which was previously restricted to SM90 (Hopper) and SM100. In communicator.py, the condition _is_sm90_supported or _is_sm100_supported was extended to include _is_sm120_supported. In server_args.py, the auto-enable logic was similarly modified.

The server was launched combining these fusion changes with an experimental NCCL tuning configuration (fewer channels, smaller buffer). It never became ready. The assistant's log inspection revealed the root cause: flashinfer's JIT compiler reported "No supported CUDA architectures found for major versions [9, 10]". SM120 corresponds to CUDA compute capability 12.0, and flashinfer's TRTLLM communication module only knew about SM 9.x and 10.x. Blackwell's SM120 was simply not in the lookup table.

This was a hard dead end. No amount of configuration tuning could fix it—the flashinfer library itself would need to be updated by its maintainers. The assistant cleanly reverted the two code changes using sed commands, verified the reverts by grepping for remaining SM120 references, and documented the failure in the optimization plan.

Dead End 2: Custom Allreduce Kernel on PCIe

With flashinfer fusion eliminated, the assistant pivoted to Priority 3: custom allreduce for PCIe. The custom allreduce kernel in SGLang normally requires NVLink for GPU-to-GPU communication, but the assistant hypothesized that it could be forced to work over PCIe by modifying the architecture checks in custom_all_reduce.py.

This required a deeper investigation. The assistant read the Python wrapper code, traced the kernel compilation pipeline, and discovered that the custom allreduce kernel is pre-compiled for specific architectures. SM120 (Blackwell) was not among them. The assistant found an escape hatch—the SGLANG_CUSTOM_ALLREDUCE_ALGO environment variable—that could force the kernel to use a different algorithm. After patching the Python gates to allow PCIe usage, the assistant launched a server with the custom allreduce enabled.

The result was catastrophic for performance: 38 tok/s, more than 2× slower than the NCCL baseline. The all-to-all communication pattern of the custom kernel caused massive PCIe bus contention. With 8 GPUs sharing the PCIe fabric, the simultaneous all-to-all transfers saturated the bus, creating a bottleneck far worse than NCCL's Ring protocol. This experiment definitively ruled out the custom allreduce approach for PCIe-connected Blackwell GPUs.

Dead End 3: Torch Symmetric Memory

Priority 5 in the optimization plan was torch symmetric memory (--enable-torch-symm-mem), a PyTorch feature that enables efficient peer-to-peer GPU communication. The assistant attempted to enable it, but the server failed during initialization. The cause: SM120 (Blackwell) is not in PyTorch's architecture lookup table for symmetric memory support. Like flashinfer fusion, this was a software compatibility issue that could only be resolved by upstream changes to PyTorch.

Dead End 4: Expert Parallelism with Flashinfer A2A

Priority 6 was Expert Parallelism (EP), which distributes MoE experts across GPUs and uses all-to-all communication instead of allreduce. The assistant attempted to enable EP with the flashinfer A2A backend (--moe-a2a-backend flashinfer). This hit an assertion error during initialization, followed by an out-of-memory condition. The flashinfer A2A implementation had compatibility issues with the specific model configuration, making it non-functional.

A Serendipitous Discovery: The 9% Baseline Improvement

Amidst this cascade of failures, a genuine positive result emerged. The assistant had been using --cuda-graph-max-bs 512 (the default) in all baseline runs. During one experiment, the assistant reduced this to 128 to free GPU memory. The result was unexpected: baseline throughput jumped from 82 to 89.5 tok/s—a 9% improvement.

The mechanism was clear in retrospect. CUDA graphs capture repetitive computation patterns to reduce kernel launch overhead, but each captured graph consumes GPU memory. With --cuda-graph-max-bs 512, the graphs captured operations for batch sizes up to 512, consuming significant memory that could otherwise be used for KV cache. Reducing to 128 freed enough memory to expand the KV cache, allowing larger batch sizes and higher throughput.

This discovery was significant for two reasons. First, it provided an immediate, no-cost throughput improvement—no code changes, no NCCL tuning, no new libraries. Second, it demonstrated that memory allocation patterns were a critical lever for performance on this hardware configuration, a lesson that would prove valuable in subsequent debugging.

The Fewer-Channels Experiment and the OOM Paradox

The assistant then turned to a hypothesis grounded in NCCL internals: reducing the number of NCCL communication channels from 16 to 1–2 could reduce per-allreduce latency for tiny 42 KB tensors. The theory was sound—for latency-bound small-message allreduce, fewer channels mean less per-operation overhead. The assistant configured NCCL_MIN_NCHANNELS=1, NCCL_MAX_NCHANNELS=2, NCCL_BUFFSIZE=131072 (down from 16 MB), and NCCL_NTHREADS=64 (down from 512).

The server launched but crashed during memory pool initialization with a RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. This was deeply puzzling because the available GPU memory after weight loading was 21.78 GB—slightly more than the 21.71 GB available in the working baseline run. The assistant was confronting what one article in this chunk called "The Paradox of Plenty": more free memory, yet an OOM error.

The assistant's diagnostic reasoning at this point is worth examining in detail. The error message said to increase --mem-fraction-static, but the assistant recognized that this would make the problem worse, not better. The memory fraction controls how much GPU memory is reserved for KV cache; if 0.55 × 96 GB = 52.8 GB was being requested but only 21.78 GB was available, the fraction was too high, not too low. The error message was misleading.

The assistant read the source code of model_runner_kv_cache_mixin.py around the init_memory_pool function to understand the max_total_num_tokens calculation. The code showed that this value was computed from a tensor result, and if it came out as zero or negative, the error was raised. The assistant hypothesized that the NCCL buffer size change was altering the memory layout in a way that affected this calculation—perhaps NCCL pre-allocated its buffers differently with fewer channels, consuming memory that the KV cache pool expected to be available.

After several rounds of investigation—checking GPU memory with nvidia-smi, comparing log outputs, tracing the memory calculation formula—the assistant made a strategic decision. The fewer-channels approach was not promising enough to justify extensive debugging. The expected benefit was marginal (a few percent at best), and the OOM interaction suggested a fundamental incompatibility between the reduced channel count and SGLang's memory management. The assistant reverted to the known-working NCCL configuration and moved on.

The Pivot to CUDA 13

At this point, the optimization campaign had systematically eliminated every approach in the plan except for MSCCL++ (which turned out to be unavailable—the Python package didn't exist). The assistant updated the optimization plan document with all experimental results, documenting each dead end and its cause.

Then the user proposed a new direction: upgrade CUDA from version 12.8 to version 13. The reasoning was that CUDA 13 has native SM120 (Blackwell) support, which could unblock every dead end simultaneously. FlashInfer fusion might work with CUDA 13's updated JIT compiler. Torch symmetric memory might support SM120 in the CUDA 13-compatible PyTorch builds. The custom allreduce kernel might have Blackwell-specific optimizations.

The assistant investigated the feasibility. The driver already supported CUDA 13.1, but the toolkit was only 12.8. Research revealed that PyTorch nightly provides cu130 wheels, sgl-kernel has a dedicated cu130 index, and flashinfer also supports CUDA 13. The upgrade path was viable.

This pivot represents a fundamental shift in strategy. Instead of trying to work around the limitations of CUDA 12.8 on Blackwell hardware—which had proven to be a series of dead ends—the assistant would upgrade the entire CUDA stack to gain native SM120 support. This could enable flashinfer fusion, torch symmetric memory, and other Blackwell-native optimizations that were previously unavailable. The upgrade offered a promising direction to finally reduce the verify cost and make speculative decoding profitable.

What the Campaign Reveals About Systematic Optimization

This chunk of the optimization campaign is a textbook example of how systematic engineering works in practice. Several key patterns emerge:

Hypothesis-driven experimentation. Every approach was grounded in a specific, testable hypothesis about why it would improve performance. The fewer-channels hypothesis was based on NCCL latency-bandwidth tradeoffs for small tensors. The custom allreduce hypothesis was based on GPU IPC shared memory having lower latency than NCCL protocols. Each hypothesis was tested, and the results were used to update the plan.

Clean state management. After every failed experiment, the assistant reverted code changes, restored configuration files, and verified the system was in a known-good state before proceeding. This prevented cascading failures where one experiment's residue contaminated the next.

Variable isolation. The assistant learned from the early mistake of combining two changes (flashinfer fusion + NCCL tuning) in a single experiment. After that, each variable was tested independently, making it possible to attribute failures to specific causes.

Cost-benefit triage. Not every dead end was investigated exhaustively. The assistant evaluated the expected benefit of each approach against the debugging cost and abandoned approaches that weren't worth the time. The fewer-channels experiment was abandoned after one OOM failure because the expected gain was small.

Negative results as knowledge. Every failed experiment produced valuable information. The flashinfer fusion failure revealed that SM120 support was missing from flashinfer's JIT compiler. The custom allreduce failure revealed that PCIe bus contention made all-to-all communication patterns worse than Ring. These negative results prevented future wasted effort and narrowed the search space.

The pivot as a skill. Perhaps the most important pattern is the willingness to pivot. The assistant did not become attached to any particular approach. When flashinfer fusion failed, it moved to custom allreduce. When custom allreduce failed, it moved to NCCL tuning. When NCCL tuning hit an OOM, it documented the results and pivoted to CUDA 13. This flexibility is essential in optimization work, where most ideas don't work and the key to success is finding the few that do.

Conclusion

The optimization campaign documented in this chunk is a story of systematic elimination. The assistant tested seven approaches to reduce the verify-pass bottleneck in EAGLE-3 speculative decoding. Four were eliminated as dead ends (flashinfer fusion, custom allreduce, torch symmetric memory, expert parallelism). One produced a positive result (reducing --cuda-graph-max-bs improved baseline throughput by 9%). One was abandoned after an OOM interaction (fewer NCCL channels). One was unavailable (MSCCL++).

But the campaign did not end in failure. The accumulated knowledge from these experiments—combined with the user's insight about CUDA 13—pointed to a new path forward. By upgrading the CUDA toolkit to version 13, the assistant could gain native SM120 support and potentially unblock all the Blackwell-native optimizations that were previously unavailable. The dead ends were not wasted effort; they were necessary steps in narrowing the search space and identifying the fundamental constraint: CUDA 12.8's lack of native Blackwell support.

In the end, the most important skill demonstrated in this chunk is not the ability to write code or tune parameters—it is the ability to learn from failure, update one's mental model, and pivot to a more promising direction. That is the essence of systematic optimization, and it is what separates effective engineering from random experimentation.