The CUDA 13 Breakthrough: How a Stack Upgrade Transformed EAGLE-3 Speculative Decoding from Net-Negative to Net-Positive
Introduction
In any high-performance computing project, there comes a moment when a single infrastructure decision unlocks a cascade of previously blocked optimizations. For the 8× RTX PRO 6000 Blackwell GPU system running the Kimi-K2.5-NVFP4 model with EAGLE-3 speculative decoding, that moment arrived with the CUDA 13 stack upgrade. Message [msg 5412] in this opencode session is a milestone summary — a concise record of what was accomplished after an intense period of debugging, patching, and system-level integration work. The message consolidates the outcome of dozens of preceding tool calls into a clean statement of fact: speculative decoding had gone from a net-negative 54.1 tok/s (40% slower than baseline) to a net-positive 96.1 tok/s (3.8% faster), a 77.6% improvement driven entirely by the CUDA 13 upgrade and the Blackwell-native optimizations it unblocked.
This article examines message [msg 5412] in depth: why it was written, the decisions it reflects, the assumptions it encodes, and the knowledge it creates for the next phase of the project.
The Message: A Milestone Summary
The subject message reads in full:
## Summary
>
The CUDA 13 upgrade was a major success. Here's what was accomplished:
>
### CUDA 13 Stack Upgrade - CUDA 13.0.1 toolkit installed at /usr/local/cuda-13.0 (alongside existing 12.8) - PyTorch 2.9.1+cu130 (stable — nightly 2.12.0 had ABI incompatibility with sgl-kernel) - sgl-kernel 0.3.21+cu130 (pre-built wheel from GitHub) - flashinfer 0.6.4 + jit-cache cu130 - SGLang v0.5.9 (clean checkout + KimiK25 EAGLE-3 patches)
>
### Key Patches Applied 1. SM120 added toall_reduce_utils.pyandtorch_symm_mem.pycapability maps 2.get_embed_and_head()andset_eagle3_layers_to_capture()delegation added tokimi_k25.py3.sitecustomize.pyupdated with CUDA 13 env vars 4.libnvrtc.so.13made available via ldconfig
>
### Performance Results
>
| Config | tok/s | Change | |---|---|---| | Old baseline (CUDA 12.8) | 89.5 | — | | Old EAGLE-3 (CUDA 12.8) | 54.1 | -40% | | New baseline (CUDA 13) | 92.6 | +3.5% | | New EAGLE-3 + fusion (CUDA 13) | 96.1 | +7.4% vs old baseline |
>
The two previously-blocked optimizations (FlashInfer allreduce fusion, Torch symmetric memory) both work now on CUDA 13 with SM120. While they don't help the single-stream baseline (allreduces are hidden behind compute), FlashInfer fusion dramatically reduced the EAGLE-3 verify cost, taking EAGLE-3 from 40% slower than baseline to 3.8% faster. The EAGLE-3 server is currently running and serving at ~96 tok/s.
This is not a message that makes decisions — it is a message that records decisions already made and validates the path taken. Its structure reveals the assistant's understanding of what matters most to the project at this juncture: a clean stack specification, a list of source-code patches that future readers will need to reproduce the setup, and a performance table that tells the story in numbers.
Why This Message Was Written
The message serves several distinct purposes, each visible in its structure and content.
First, it is a checkpoint. The preceding segment (segment 35) had systematically tested and eliminated several allreduce optimization approaches for PCIe-connected Blackwell GPUs. FlashInfer allreduce fusion had failed on CUDA 12.8 because its JIT compiler didn't recognize SM120. Torch symmetric memory had failed for the same reason. Custom allreduce kernels had been tested and rejected. The team had reached a dead end on CUDA 12.8, and the decision to upgrade to CUDA 13 was a high-risk, high-reward gamble. This message confirms that the gamble paid off.
Second, it is a knowledge capture document. The four patches listed under "Key Patches Applied" are not obvious. They represent hours of debugging — discovering that kimi_k25.py lacked get_embed_and_head() and set_eagle3_layers_to_capture() delegation methods (see [msg 5388] through [msg 5400]), that SM120 needed to be added to capability maps in two separate files, that libnvrtc.so.13 needed to be made available via ldconfig. Without this summary, a future developer attempting to reproduce the setup would have to rediscover each of these patches by reading through hundreds of messages.
Third, it is a transition document. The final sentence — "The EAGLE-3 server is currently running and serving at ~96 tok/s" — signals that the infrastructure work is complete and the project can move to the next phase. Indeed, the chunk summary for segment 36 reveals that the next task identified was implementing dynamic speculation disabling based on server load, since benchmarking showed EAGLE-3 becomes a liability at high concurrency (C≥30). The summary message creates the clean handoff point for that transition.
Decisions Reflected in the Message
Although the message itself does not make new decisions, it encodes several critical decisions that were made during the CUDA 13 upgrade process.
The PyTorch version choice. The message notes "PyTorch 2.9.1+cu130 (stable — nightly 2.12.0 had ABI incompatibility with sgl-kernel)." This reflects a real debugging episode. The nightly PyTorch 2.12.0 was likely the first cu130 build available, but it broke ABI compatibility with the pre-built sgl-kernel wheel. The decision to fall back to 2.9.1 (a stable release that had been rebuilt for CUDA 13) was a pragmatic tradeoff: newer is not always better when the dependency chain isn't aligned.
The SGLang version choice. The message specifies "SGLang v0.5.9 (clean checkout + KimiK25 EAGLE-3 patches)." This tells us that the team chose to stay on a known-good SGLang version rather than upgrading to a newer version that might have different EAGLE-3 interfaces. The patches applied to kimi_k25.py were minimal — just delegation methods — suggesting the team valued stability over feature freshness.
The optimization targeting decision. The message explicitly states that FlashInfer allreduce fusion and Torch symmetric memory "don't help the single-stream baseline (allreduces are hidden behind compute)." This is a non-obvious insight. On a single-stream workload, the GPU is compute-bound, so allreduce communication happens in the shadow of computation and doesn't affect latency. But EAGLE-3 speculative decoding introduces a verify pass that is communication-bound — the allreduce becomes the bottleneck. The decision to pursue these optimizations despite their lack of baseline benefit was correct, but only in hindsight. The message captures this nuance.
Assumptions and Their Validity
The message rests on several assumptions, most of which proved correct.
Assumption: CUDA 13 would unblock Blackwell-native optimizations. This was the central hypothesis driving the entire upgrade effort. It proved correct: SM120 was recognized by the FlashInfer JIT compiler under CUDA 13, and both FlashInfer allreduce fusion and Torch symmetric memory began functioning. The assumption was grounded in the knowledge that CUDA 13 added SM120 support to its JIT runtime (libnvrtc), whereas CUDA 12.8's JIT compiler predated Blackwell support.
Assumption: FlashInfer allreduce fusion would help EAGLE-3 verify. This was the secondary hypothesis. It proved correct, but the message's careful wording — "While they don't help the single-stream baseline... FlashInfer fusion dramatically reduced the EAGLE-3 verify cost" — reveals that the assistant understood the distinction between compute-bound and communication-bound workloads. The verify pass in EAGLE-3 is a small forward pass that doesn't saturate compute, so communication overhead dominates. This assumption was validated by the 77.6% improvement in speculative throughput.
Assumption: The stack would be stable enough for production. The message ends with "The EAGLE-3 server is currently running and serving at ~96 tok/s." This implies the stack is stable enough for benchmarking and potentially for production use. However, the chunk summary reveals that the next task is dynamic speculation disabling, suggesting the team recognized that the current configuration is not optimal at all concurrency levels.
One incorrect assumption worth noting: The initial EAGLE-3 server launch with flashinfer fusion (see [msg 5393]) crashed because KimiK25ForConditionalGeneration lacked set_eagle3_layers_to_capture(). The assistant had assumed that the EAGLE-3 worker only needed get_embed_and_head() and set_embed_and_head(), but the model runner also calls set_eagle3_layers_to_capture() during initialization. This was a gap in the assistant's understanding of the SGLang codebase, quickly corrected by adding the delegation method in [msg 5400]. The summary message in [msg 5412] includes this fix in its list of key patches, showing that the lesson was absorbed.
Input Knowledge Required
To fully understand message [msg 5412], a reader needs substantial context from the preceding conversation.
Understanding of the hardware topology. The system has 8× RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink. This topology makes allreduce communication particularly expensive — a fact that explains why FlashInfer allreduce fusion matters so much for this specific configuration. A reader unfamiliar with the PCIe bottleneck might wonder why allreduce optimization is the central concern.
Knowledge of EAGLE-3 speculative decoding. EAGLE-3 is a speculative decoding algorithm that uses a small draft model to propose tokens, then verifies them against the target model. The verify pass is a forward pass of the full target model, which on an 8-GPU TP=8 configuration requires allreduce communication between GPUs. The reader needs to understand that the verify pass is communication-bound because it processes only a small batch (draft tokens), making it latency-sensitive to allreduce performance.
Awareness of the CUDA 12.8 dead end. The preceding segment (segment 35) documented systematic attempts to optimize allreduce on CUDA 12.8 — FlashInfer fusion failed because the JIT didn't recognize SM120, NCCL Tree algorithm failed due to CUDA graph incompatibility, Torch symmetric memory failed for the same SM120 reason. Without this context, the CUDA 13 upgrade might seem like an arbitrary infrastructure change rather than the necessary response to a blocked optimization path.
Understanding of ABI compatibility in the PyTorch ecosystem. The mention of "nightly 2.12.0 had ABI incompatibility with sgl-kernel" requires knowledge that PyTorch and its extension libraries (like sgl-kernel) must be compiled against the same C++ ABI. When PyTorch is rebuilt for a new CUDA version, extension libraries must also be rebuilt. The pre-built sgl-kernel wheel was compiled against PyTorch 2.9.1's ABI, not 2.12.0's.
Output Knowledge Created
Message [msg 5412] creates several pieces of valuable knowledge that the project will build upon.
A validated stack specification. The message provides a precise recipe: CUDA 13.0.1 + PyTorch 2.9.1+cu130 + sgl-kernel 0.3.21+cu130 + flashinfer 0.6.4 + SGLang v0.5.9. This is the "known good" configuration that future work will use. Any deviation from this stack risks reintroducing the ABI or compatibility issues that were resolved.
A performance baseline for future optimization. The table of four configurations (old baseline, old EAGLE-3, new baseline, new EAGLE-3) provides reference points for any future optimization work. If someone later improves the verify pass further, they can compare against the 96.1 tok/s figure. If someone optimizes the baseline, they can compare against 92.6 tok/s.
Documentation of required source patches. The four patches listed are essential for reproducing the setup. Without this list, a future developer would need to rediscover that SM120 must be added to two capability maps, that kimi_k25.py needs delegation methods, and that libnvrtc.so.13 must be made available. The message transforms tacit knowledge (learned through debugging) into explicit knowledge.
Understanding of the optimization landscape. The message's observation that FlashInfer fusion helps EAGLE-3 but not the baseline is a non-trivial insight that shapes future strategy. It tells the team that further optimization efforts should focus on the verify pass specifically, not on general-purpose allreduce improvements.
The Thinking Process Visible in the Message
While message [msg 5412] is a summary rather than a reasoning trace, its structure reveals the assistant's thinking process.
The message is organized into three sections — stack, patches, results — which reflects a mental model of "what you need, what you must change, and what you get." This is the thinking of a systems engineer who views the project as a layered stack where each layer must be correctly configured before the next can function.
The decision to include the nightly PyTorch 2.12.0 failure ("stable — nightly 2.12.0 had ABI incompatibility") shows a thinking process that values negative results as much as positive ones. The assistant could have simply listed the final stack without mentioning the failed alternative, but including it provides crucial context for anyone who might wonder "why not use the latest PyTorch?"
The careful qualification of the FlashInfer fusion benefit — "While they don't help the single-stream baseline... FlashInfer fusion dramatically reduced the EAGLE-3 verify cost" — reveals a thinking process that distinguishes between different workload types. The assistant is not claiming a universal improvement; it is claiming a targeted improvement for a specific workload characteristic (communication-bound verify pass). This precision of thought is essential for correct optimization targeting.
The final sentence — "The EAGLE-3 server is currently running and serving at ~96 tok/s" — is a subtle signal that the assistant considers the infrastructure phase complete. The use of "serving" (present continuous) rather than "was benchmarked at" (past tense) suggests readiness for the next phase: load testing, concurrency analysis, and dynamic speculation policy.
Conclusion
Message [msg 5412] is a milestone document in the opencode session — a moment where the assistant pauses the rapid cycle of debugging and benchmarking to consolidate what was learned. The CUDA 13 upgrade transformed the project's trajectory, shifting the bottleneck from "making the verify pass work" to "optimally deploying the working verify pass." The message captures this transition with precision: a clean stack specification, a list of required patches, and a performance table that tells the story in numbers.
For a reader who hasn't followed the preceding 35 segments of conversation, this message provides a compressed but complete picture of a complex systems engineering achievement. It shows that the right infrastructure decision — upgrading CUDA to unlock Blackwell-native optimizations — can transform a speculative decoding system from a net liability (40% slower) to a net asset (3.8% faster). And it sets the stage for the next challenge: making speculation smart enough to know when to turn itself off.