From Net-Negative to Net-Positive: The CUDA 13 Breakthrough That Transformed EAGLE-3 Speculative Decoding on Blackwell

Introduction

In the high-stakes world of large language model inference, few challenges are as humbling as speculative decoding on PCIe-connected multi-GPU systems. The promise is seductive: a small "draft" model proposes candidate tokens while the large "target" model verifies them in parallel, ideally generating more tokens per second than the target model alone. But the reality is often brutal — the verify pass, which checks the draft tokens, introduces communication overhead that can turn speculation from an accelerator into a drag.

This is the story of how an 8× NVIDIA RTX PRO 6000 Blackwell GPU system running the Kimi-K2.5-NVFP4 model with EAGLE-3 speculative decoding was transformed from a net-negative 54.1 tok/s (40% slower than baseline) to a net-positive 96.1 tok/s (3.8% faster). The journey spanned CUDA toolkit upgrades, SGLang source code patching, ABI compatibility battles, and a fundamental rethinking of when speculation should be used at all. It is a case study in systematic systems optimization — and a reminder that the most important optimization decision is often not how to make something faster, but when to use it.

The CUDA 13 Stack Upgrade: A Foundation for Breakthrough

The entire optimization campaign was blocked by a single constraint: CUDA 12.8 did not support Blackwell's SM120 compute capability for certain JIT-compiled kernels. This meant that two critical optimizations — FlashInfer allreduce fusion and Torch symmetric memory — were completely inaccessible. The assistant had systematically tested every alternative: NCCL Tree algorithm (failed due to CUDA graph incompatibility), custom allreduce kernels (impractical for the architecture), and Expert Parallelism (not viable for the PCIe topology). Each dead end pointed to the same conclusion: the CUDA stack had to be upgraded.

Upgrading from CUDA 12.8 to 13.0.1 was not a trivial undertaking. The assistant navigated significant ABI compatibility challenges to assemble a stable stack, documented in the milestone summary at <msg id=5412>:

Patching SGLang for SM120: Enabling Blackwell-Native Optimizations

With CUDA 13 installed, the assistant discovered that SGLang's source code did not recognize SM120 (Blackwell's compute capability) in several critical locations. The torch_symm_mem module and the all_reduce_utils.py capability maps both needed patches to add SM120 support. These patches, documented in <msg id=5412>, were surgical but essential:

  1. SM120 added to all_reduce_utils.py and torch_symm_mem.py capability maps
  2. sitecustomize.py updated with CUDA 13 environment variables
  3. libnvrtc.so.13 made available via ldconfig The FlashInfer allreduce fusion optimization was the key enabler. In a tensor-parallel deployment across 8 GPUs, every forward and backward pass requires allreduce operations to synchronize partial results. For large batches, these costs are amortized. But for the tiny batches characteristic of the EAGLE-3 verify step (often just 1–3 tokens), each allreduce's fixed latency dominates. FlashInfer allreduce fusion addresses this by combining multiple small allreduce operations into a single, larger operation, reducing the number of synchronization points and improving PCIe bandwidth utilization. The assistant's reasoning about when these optimizations matter was precise. As noted in <msg id=5412>: "While they don't help the single-stream baseline (allreduces are hidden behind compute), FlashInfer fusion dramatically reduced the EAGLE-3 verify cost." This distinction between compute-bound and communication-bound workloads was essential to understanding why the optimizations worked for speculation but not for baseline inference.

The EAGLE-3 Delegation Fix: Bridging the Model Gap

The first attempt to launch the EAGLE-3 server with FlashInfer fusion (<msg id=5393>) ended in a crash after 97 polling cycles (roughly 8 minutes). The error was an AttributeError:

'KimiK25ForConditionalGeneration' object has no attribute 'set_eagle3_layers_to_capture'

This crash revealed a fundamental integration gap. The KimiK25ForConditionalGeneration class — a custom wrapper around DeepseekV3ForCausalLM — did not inherit from the base model class that implemented EAGLE-3's interface methods. Instead, it wrapped the DeepseekV3 model as self.language_model, and every method that the EAGLE-3 infrastructure expected on the outer class had to be explicitly delegated.

The assistant's response to this crash demonstrated a methodical debugging approach that spanned several messages (<msg id=5395> through <msg id=5400>). Rather than blindly adding the single missing method, the assistant stepped back and asked a more fundamental question: What other methods might be missing?

In <msg id=5397>, the assistant ran a grep command that extracted every method call pattern from the EAGLE-3 worker source code, filtering for calls on target_worker., self.model_runner.model., or .model. — the three patterns by which the worker accesses the target model object. The output revealed eight methods, including get_embed_and_head, set_embed_and_head, forward_batch_generation, and set_eagle3_layers_to_capture. This was the complete interface contract between the EAGLE-3 worker and the target model.

The assistant then traced where set_eagle3_layers_to_capture was actually called (<msg id=5399>), discovering that the call sites were in model_runner.py and cuda_graph_runner.py — not directly in the eagle worker as initially assumed. This diagnostic grep transformed an opaque error into a well-understood integration gap.

The fix itself (<msg id=5400>) was elegant in its minimalism: a single sed command that inserted a delegation method before the EntryClass line in kimi_k25.py:

def set_eagle3_layers_to_capture(self, layer_ids=None):
    self.language_model.set_eagle3_layers_to_capture(layer_ids)

The assistant recognized the pattern from mllama4.py (line 954), which had already solved the same architectural problem — a wrapper model delegating EAGLE-3 methods to an inner language model. This pattern recognition is a hallmark of experienced software engineering: recognizing when a problem has already been solved elsewhere in the codebase and adopting the same approach for consistency.

The Breakthrough: From 54.1 to 96.1 tok/s

With the delegation fix applied and the server restarted (<msg id=5401>), the assistant waited through 116 polling cycles — 580 seconds — for the server to come up (<msg id=5402>). This wait, documented in the article "The 580-Second Wait" [6], was the last barrier before the payoff.

The benchmark results were transformative. In <msg id=5404>, the assistant reported:

97.8 tok/s with EAGLE-3 + flashinfer fusion! This is a dramatic improvement: - Previous EAGLE-3 on CUDA 12.8: 54.1 tok/s (40% slower than baseline) - Now EAGLE-3 on CUDA 13 + fusion: 97.8 tok/s (5.6% faster than the 92.6 baseline!)

A follow-up 10-run benchmark (<msg id=5405>) confirmed a more stable average of 96.1 tok/s (range: 87.5–103.5), representing a 77.6% improvement from the previous EAGLE-3 configuration and a 3.8% gain over the baseline.

The high variance (87.5–103.5 tok/s) was characteristic of speculative decoding, where acceptance rates vary depending on how predictable the generated text is. When the draft model happens to propose tokens that the target model accepts, the effective throughput skyrockets. When the draft model guesses poorly, the verify step rejects most candidates and the system falls back to generating one token at a time.

This was the moment of truth — the first time in the entire optimization campaign that EAGLE-3 speculative decoding had beaten the baseline. The CUDA 13 upgrade, the SM120 patches, the FlashIner fusion enablement, and the delegation methods had all been necessary but individually insufficient. Only when the last piece fell into place did the system deliver its breakthrough performance.

The Pivot: Discovering Speculation's Load Sensitivity

The celebration was brief. In <msg id=5413>, the user issued a request that would reshape the entire project's trajectory:

"Try benchmarking parallel inference C=2,5,10,30,70,100,250"

This was not a request for more data — it was a probe into a fundamental unknown. The single-stream victory at 96.1 tok/s was necessary but not sufficient. The real question was whether EAGLE-3 speculation still helped when the server was under realistic multi-user load.

The assistant immediately recognized that the existing benchmark infrastructure was inadequate (<msg id=5414>). The benchmark_eagle3.py script was a serial benchmark — it sent one request at a time and measured per-request latency. It could not answer the question: "What happens to total system throughput when 30 clients are hammering the server simultaneously?"

In <msg id=5415>, the assistant wrote a new parallel benchmark from scratch: benchmark_parallel.py. This script managed concurrent HTTP requests, collected timing data, computed aggregate statistics, and handled error cases gracefully. It was designed to measure total throughput across all concurrent requests — a fundamentally different metric from the serial benchmark's per-request latency.

The results (<msg id=5417>) were striking:

| Concurrency | Throughput (tok/s) | Per-Req (tok/s) | Avg Latency (s) | |---|---|---|---| | 1 | 77.5 | 77.7 | 6.6 | | 2 | 125.1 | 62.9 | 8.2 | | 5 | 183.2 | 37.3 | 13.8 | | 10 | 239.6 | 24.8 | 20.8 | | 30 | 299.5 | 10.4 | 49.6 | | 70 | 337.7 | 5.7 | 96.2 | | 100 | 338.8 | 4.1 | 134.3 | | 250 | 340.9 | 2.9 | 243.6 |

The data told a clear story: throughput scaled nearly linearly from C=1 to C=10, then began to saturate. By C=70, the system hit a ceiling around 338-340 tok/s, and adding more concurrency yielded almost no additional throughput. Meanwhile, per-request throughput collapsed from 77.7 tok/s at C=1 to just 2.9 tok/s at C=250, and latency ballooned from 6.6 seconds to over 4 minutes.

This was the critical insight: speculation's value depended on how busy the GPUs were. At low concurrency, the GPU had idle cycles that the draft model could exploit. At high concurrency, every cycle was already occupied, and the verify pass became pure overhead.

The Strategic Pivot: Dynamic Speculation Disabling

The user's response in <msg id=5418> demonstrated a sophisticated understanding of the system's behavior:

"Can we tell (or mod) sglang to disable speculation at certain concurrency? Probably need to bench w/o speculation to decide a threshold (also maybe for 2-3 tok reduced speculation)"

This message was the strategic pivot. The user recognized that the optimization problem had fundamentally changed — from "make speculation work" to "know when to use it." The assistant's response (<msg id=5419>) immediately embraced the insight:

"Good idea — speculation helps at low concurrency but likely hurts at high concurrency where the GPU is already saturated. Let me first get the baseline (no speculation) numbers at the same concurrency levels so we can find the crossover point."

The assistant laid out a complete investigation plan with four high-priority todos:

  1. Benchmark baseline (no speculation) at C=1,2,5,10,30,70,100,250
  2. Compare EAGLE-3 vs baseline at each concurrency to find crossover point
  3. Investigate SGLang dynamic speculation disable based on concurrency/load
  4. Test reduced speculation (fewer steps/draft tokens) at high concurrency This plan reflected a nuanced understanding of the problem. The crossover point — where speculation's benefit equals its cost — would be empirically determined by comparing the two throughput curves. The reduced speculation option acknowledged that the optimal policy might not be binary (on/off) but continuous (reduce speculation intensity as load increases). The empty user message at <msg id=5420> — a continuation signal that said nothing and meant everything — authorized the assistant to proceed without further approval. This was the product of a mature collaboration where trust had been earned through dozens of rounds of successful debugging.

Documentation: The Discipline of Recording Results

Throughout the optimization campaign, the assistant maintained a meticulous experiment log at /home/theuser/glm-kimi-sm120-rtx6000bw/eagle-fast-verify.md. This document had grown to 399 lines (<msg id=5407>) and contained the complete history of attempted optimizations, their results, and the reasoning behind each approach.

The documentation process was itself a multi-step ritual. In <msg id=5406>, the assistant read the experiment log to understand its current state. In <msg id=5408>, it reviewed the results table — a scoreboard of failure that showed entries like "EAGLE-3 2-step: 54.1 tok/s (40% slower than baseline)" and "FlashInfer fusion SM120: FAILED (JIT doesn't recognize SM120)." In <msg id=5409> and <msg id=5410>, the assistant applied edits to update both the narrative and the results table, transforming the document from a record of failure into a record of success.

The todo list update at <msg id=5411> marked the administrative closing of the CUDA 13 upgrade chapter. The milestone summary at <msg id=5412> consolidated everything into a clean, referenceable format: stack specification, key patches, and performance results. This documentation discipline ensured that the hard-won knowledge would survive beyond the current session.

Conclusion

The CUDA 13 upgrade transformed EAGLE-3 speculative decoding from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s — a 77.6% improvement driven by FlashInfer allreduce fusion and Torch symmetric memory, both unblocked by the stack upgrade. But the parallel benchmarks revealed a deeper truth: speculation's value is not constant. It helps at low concurrency where the GPU has idle cycles, but becomes a liability at high concurrency where the GPU is already saturated.

This insight shifted the project's trajectory from "making speculation work" to "deploying speculation intelligently." The next phase — dynamic speculation disabling based on server load — represents a more sophisticated approach to optimization: not a static configuration, but a load-aware policy that adapts to conditions.

The journey documented in this chunk is a testament to systematic engineering. Every dead end was explored and eliminated. Every hypothesis was tested with measurement. Every breakthrough was recorded. And when the data revealed a more complex reality than the single-stream benchmark suggested, the team pivoted — not in panic, but with a clear plan and a methodical approach to finding the crossover point where speculation should yield to the baseline.

In the end, the most valuable optimization was not a faster kernel or a better algorithm — it was the recognition that the right question is not "how do we make this faster?" but "when should we use this at all?"