The Victory Lap: How a CUDA 13 Upgrade Turned EAGLE-3 Speculative Decoding from Liability to Asset

Introduction

In the course of a long and grueling optimization campaign for an 8× NVIDIA RTX PRO 6000 Blackwell GPU system running the Kimi K2.5 NVFP4 model, message [msg 5405] stands as a quiet milestone — the moment when a persistent performance deficit was finally reversed. The message is brief, almost understated: "Over 10 runs: 96.1 tok/s average (87.5 - 103.5 range). This is 3.8% faster than the 92.6 tok/s baseline without speculation." But to understand why this single paragraph matters, one must trace the arc of failures, dead ends, and incremental breakthroughs that preceded it.

This article examines message [msg 5405] in depth: what it reports, why it was written, the decisions and assumptions embedded in its numbers, and the technical context that gives it meaning. It is a story about how a carefully orchestrated stack upgrade — from CUDA 12.8 to CUDA 13, from stock SGLang to a patched version with Blackwell support — transformed EAGLE-3 speculative decoding from a net-negative 54.1 tok/s (40% slower than baseline) to a net-positive 96.1 tok/s (3.8% faster), a 77.6% improvement in speculative throughput.

The Message Itself

The complete text of the subject message is:

Over 10 runs: 96.1 tok/s average (87.5 - 103.5 range). This is 3.8% faster than the 92.6 tok/s baseline without speculation. And remember, the old EAGLE-3 on CUDA 12.8 was only 54.1 tok/s — a 77.6% improvement from the CUDA 13 upgrade + fusion!

Following this, the assistant emits a structured todowrite block listing completed tasks: backing up the Python environment, installing CUDA 13.0, installing PyTorch cu130, installing sgl-kernel 0.3.21+cu130, and several other infrastructure steps that had been accomplished over the preceding rounds.

On its surface, the message is a benchmark report. But its subtext is one of triumph — the assistant is not merely reporting numbers but contextualizing them against the history that the user (and anyone reading the conversation log) would recognize. The phrase "remember, the old EAGLE-3 on CUDA 12.8 was only 54.1 tok/s" is a deliberate callback to the failure state that had plagued the project for days.

Why This Message Was Written: The Reasoning and Motivation

The message was written because the assistant had just completed a 10-run benchmark of the EAGLE-3 speculative decoding server, and the results were significant enough to warrant explicit documentation. But the deeper motivation is more interesting: this message serves as a validation point for an entire chain of technical decisions.

The assistant had been working on EAGLE-3 speculative decoding for the Kimi K2.5 model across many rounds. Earlier attempts had been disastrous — the verify pass (where the target model checks the draft tokens produced by the smaller drafter model) was so slow that speculation became a net loss. The bottleneck was the all-reduce communication during verification: with 8 GPUs connected via PCIe rather than NVLink, the NCCL all-reduce operations dominated latency. Every optimization tried — FlashInfer allreduce fusion on CUDA 12.8, custom all-reduce kernels, Torch symmetric memory — had failed because Blackwell's SM120 architecture wasn't recognized by the software stack.

The CUDA 13 upgrade was the key that unlocked everything. Once CUDA 13 was installed and SGLang was patched to recognize SM120, both FlashInfer allreduce fusion and Torch symmetric memory began functioning correctly. The message at [msg 5405] is the first time the assistant could point to a benchmark and say: it worked.

The todowrite block reinforces this narrative. It lists completed tasks in reverse chronological order of the upgrade process: backup, CUDA 13 install, PyTorch cu130 install, sgl-kernel install, flashinfer install, SGLang v0.5.9 install, and the various patches. This is the assistant's way of saying "here is the recipe that produced this result."

How Decisions Were Made

While the subject message itself does not contain decision-making (it is a report of outcomes), the decisions that led to this moment are visible in the surrounding context. Let me trace them.

Decision 1: Upgrade CUDA to Version 13

The most consequential decision was to abandon CUDA 12.8 and install CUDA 13.0 alongside it. This was not a trivial choice — CUDA 13 was bleeding-edge, with potential ABI compatibility issues and unknown interactions with PyTorch, flashinfer, and SGLang. The assistant had to assemble a compatible stack piece by piece: CUDA 13.0.1, PyTorch 2.9.1+cu130, sgl-kernel 0.3.21+cu130, flashinfer 0.6.4, and SGLang v0.5.9. Each component had to be verified against the others.

The decision was driven by necessity: Blackwell's SM120 compute capability was not recognized by CUDA 12.8's FlashInfer allreduce fusion or Torch symmetric memory paths. The assistant had systematically tested and eliminated every other approach (see [msg 5395] through [msg 5400]), and CUDA 13 was the remaining option.

Decision 2: Patch SGLang for SM120 Support

Once CUDA 13 was installed, the assistant discovered that SGLang's torch_symm_mem and kimi_k25.py modules did not recognize SM120. The decision to patch these files rather than wait for an official release was pragmatic: the patches were small and well-understood, and the alternative (waiting weeks or months) was unacceptable.

Decision 3: Add Delegation Methods to KimiK25

When the first EAGLE-3 server attempt crashed with AttributeError: 'KimiK25ForConditionalGeneration' object has no attribute 'set_eagle3_layers_to_capture' (see [msg 5394]), the assistant diagnosed the problem and added delegation methods. The KimiK25 model wraps DeepseekV3ForCausalLM as self.language_model, but the EAGLE-3 worker calls methods like get_embed_and_head(), set_embed_and_head(), and set_eagle3_layers_to_capture() directly on the top-level model class. The assistant recognized the pattern from mllama4.py (line 954) and added simple delegation methods that forward these calls to self.language_model.

Assumptions Made

Several assumptions underpin the message's claims:

The Baseline is Still Valid

The assistant compares the 96.1 tok/s EAGLE-3 result against a 92.6 tok/s baseline measured without speculation. This assumes that the baseline throughput has not changed due to the CUDA 13 upgrade or any other environmental factor. In reality, the CUDA 13 upgrade itself improved baseline throughput from 89.5 to 92.6 tok/s (a 3.5% gain), so the comparison is against the new baseline, not the old one. This is correct methodology.

10 Runs is Sufficient for a Stable Benchmark

The assistant ran 10 iterations and reported the average. The range (87.5 to 103.5 tok/s) shows significant variance — about ±8% from the mean. This is typical for speculative decoding, where acceptance rates vary based on the prompt and the drafter's luck. The assistant implicitly assumes that 10 runs smooth out this variance enough to draw conclusions. A more rigorous benchmark might use 50 or 100 runs, but 10 is reasonable for a progress check.

The Improvement is Attributable to CUDA 13 + Fusion

The assistant attributes the 77.6% improvement to "the CUDA 13 upgrade + fusion." This is a reasonable attribution but not a controlled experiment. The upgrade involved multiple simultaneous changes: a new CUDA toolkit, new PyTorch build, new flashinfer version, new sgl-kernel, and new SGLang version. Any of these components could have contributed. The assistant's reasoning is that the specific Blackwell-native optimizations (FlashInfer allreduce fusion and Torch symmetric memory) were the key enablers, and these only became functional after the CUDA 13 upgrade.

Mistakes and Incorrect Assumptions

The Missing Delegation Methods

The most concrete mistake visible in the context was the assumption that KimiK25ForConditionalGeneration would work with EAGLE-3 out of the box. The first server launch ([msg 5393]) crashed because set_eagle3_layers_to_capture was not defined on the KimiK25 class. The assistant had to iterate: first adding get_embed_and_head and set_embed_and_head ([msg 5388]), then discovering the need for set_eagle3_layers_to_capture ([msg 5394]), and finally adding it ([msg 5400]).

This mistake stemmed from an assumption that the EAGLE-3 worker only called methods listed in the eagle_worker.py grep results ([msg 5397]). But set_eagle3_layers_to_capture is called from model_runner.py and cuda_graph_runner.py, not directly from the eagle worker. The assistant had to broaden the search ([msg 5399]) to find all callers.

The Overwritten File

Another minor mistake occurred at [msg 5389] when the assistant appended delegation methods after the EntryClass line (i.e., outside the class definition). This was caught and fixed by truncating the file and re-inserting the methods at the correct location. The mistake was harmless but shows the iterative nature of the patching process.

Input Knowledge Required

To understand this message, one needs knowledge of:

  1. Speculative decoding architecture: The concept of a target model (Kimi K2.5) and a draft model (EAGLE-3 drafter), where the draft model proposes tokens and the target model verifies them in parallel.
  2. The verify pass bottleneck: The all-reduce communication required during verification, which is particularly expensive on PCIe-connected GPUs.
  3. CUDA versioning and compute capabilities: CUDA 12.8 vs 13.0, and the SM120 compute capability of Blackwell GPUs. The fact that certain optimizations (FlashInfer allreduce fusion, Torch symmetric memory) require specific CUDA versions and architecture support.
  4. SGLang's model architecture: The pattern of wrapper models (like KimiK25ForConditionalGeneration) that delegate to an inner language model (DeepseekV3ForCausalLM), and the need for explicit delegation methods for EAGLE-3 integration.
  5. The history of failures: The reader must know that previous EAGLE-3 attempts achieved only 54.1 tok/s (40% slower than baseline) to appreciate the 77.6% improvement claim.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A validated recipe: The todowrite block documents the exact sequence of steps that produced a working EAGLE-3 deployment on Blackwell GPUs. This is actionable knowledge for anyone facing similar hardware.
  2. A performance baseline: 96.1 tok/s for EAGLE-3 speculation on Kimi K2.5 with 8× RTX PRO 6000 GPUs, TP=8, with flashinfer allreduce fusion enabled.
  3. A confirmation that CUDA 13 unblocks Blackwell optimizations: The message proves that the CUDA 13 upgrade was the correct decision and that the Blackwell-native optimizations (FlashInfer allreduce fusion, Torch symmetric memory) are now functional.
  4. A benchmark methodology: Running 10 iterations and reporting mean and range provides a template for future performance testing.

The Thinking Process Visible in Reasoning

The assistant's thinking is visible in how it structures the message. It leads with the headline number (96.1 tok/s), then provides context (3.8% faster than baseline), then adds historical perspective (77.6% improvement from the old 54.1 tok/s). This is a deliberate rhetorical structure: establish the result, contextualize it, then emphasize the magnitude of improvement.

The todowrite block that follows is not just a status update — it is a chain of causality. By listing the completed tasks in order, the assistant implicitly argues that each step was necessary for the final result. The backup, the CUDA install, the PyTorch install, the kernel installs, the SGLang install, the patches — all of these were prerequisites for the benchmark result reported above.

The assistant also shows awareness of the variance issue. The range (87.5-103.5) is reported alongside the mean, and the assistant does not overclaim. It says "3.8% faster" rather than rounding up to 4%, and it acknowledges the range explicitly. This intellectual honesty is important for credibility.

Conclusion

Message [msg 5405] is a victory lap, but a measured one. The assistant does not declare the problem solved — it reports the numbers and lets them speak. The real work was in the preceding rounds: the CUDA 13 upgrade, the SM120 patches, the delegation methods, the server restarts, the crash debugging. This message is the moment when all that effort crystallizes into a single number: 96.1 tok/s.

The message also contains the seeds of future work. The high variance (87.5-103.5) suggests that speculation is not uniformly beneficial — it depends on the prompt and the acceptance rate. The assistant would later explore dynamic speculation disabling based on server load, recognizing that at high concurrency (C≥30), speculation becomes a liability. But for this moment, the message stands as a testament to a successful optimization campaign: EAGLE-3 speculative decoding, once a net-negative 54.1 tok/s, now delivers a solid 96.1 tok/s — faster than the baseline and finally fulfilling its promise.