The CUDA 13 Breakthrough: Transforming EAGLE-3 Speculative Decoding from Net-Negative to Net-Positive on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference optimization, few stories capture the interplay between systematic debugging, strategic risk-taking, and deep architectural understanding as vividly as the CUDA 13 upgrade saga documented in Segment 36. What began as a project stuck in a frustrating impasse—with EAGLE-3 speculative decoding performing 40% slower than the baseline on an 8× NVIDIA RTX PRO 6000 Blackwell GPU system running the Kimi-K2.5 model—culminated in a stunning reversal: speculative throughput of 96.1 tok/s, a 77.6% improvement that finally made EAGLE-3 a net-positive optimization.

But the story does not end with this triumph. Parallel benchmarking revealed a deeper truth: speculation's value is not constant across all workloads. At low concurrency, where GPUs have idle cycles, EAGLE-3 provides a clear advantage. At high concurrency, where the GPUs are already saturated with compute, the verify pass overhead becomes pure liability. This discovery transformed the project's trajectory from "making speculation work" to "deploying speculation intelligently"—a far more nuanced and impactful optimization challenge.

This article synthesizes the entire journey, tracing the arc from strategic pivot through infrastructure ordeal to triumphant validation, and culminating in the discovery that the most important optimization decision is often not how to make something faster, but when to use it.

The Strategic Pivot: Recognizing the Root Cause

The project's turning point came not with a clever code change or a new algorithm, but with a moment of diagnostic clarity. The assistant had been systematically testing six different approaches to accelerate the EAGLE-3 verify pass—the critical bottleneck where the large target model checks the draft model's predictions in parallel. Every approach had failed, but the failures shared a common pattern: they crashed with errors related to missing SM120 (Blackwell) architecture support [1].

FlashInfer allreduce fusion failed because its JIT compiler reported "No supported CUDA architectures found for major versions [9, 10]." Torch symmetric memory threw a KeyError: 12 because SGLang's architecture lookup table only went up to compute capability 10. Custom allreduce on PCIe produced a disastrous 38 tok/s. NCCL Tree was incompatible with CUDA graphs. Expert Parallelism hit assertion errors. Each dead end pointed to the same root cause: the CUDA 12.8 software stack simply did not recognize Blackwell GPUs for the JIT-compiled kernels that these optimizations required.

The assistant's analysis connected these dots into a coherent theory. The verify pass required 122 NCCL all-reduce operations per forward pass—one for each of the 61 layers, times two for attention and MoE. Each all-reduce was tiny (just 42 KB), but NCCL's Ring LL protocol imposes a latency floor of roughly 150–300 microseconds per operation regardless of tensor size. The math was brutal: 122 all-reduces × ~200 µs ≈ 24 milliseconds of pure NCCL latency overhead, accounting for roughly 80% of the verify pass time.

The two most promising solutions—FlashInfer allreduce fusion (which would fuse multiple small all-reduces into larger ones) and Torch symmetric memory (which would enable peer-to-peer GPU memory access over PCIe)—were blocked not by architectural limitations but by a software stack that simply didn't recognize Blackwell. The user's question—"Should we update cuda to 13 with more proper support for sm120?"—had identified the precise intervention needed [2].

The Authorization: From Analysis to Action

The user's response was deceptively simple: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed." This 14-word message was the fulcrum on which the entire project pivoted. It was not a command but a delegation—a signal that the user trusted the assistant's analysis and authorized autonomous execution.

This moment reveals a crucial dynamic in human-AI collaboration. The assistant had earned the right to proceed through exhaustive due diligence: verifying that the NVIDIA driver already supported CUDA 13.1 (version 590.48.01), researching PyTorch cu130 nightly wheel availability, confirming sgl-kernel cu130 compatibility, and discovering that a critical SM120 fix had landed just 20 days prior [3]. A comprehensive state-of-the-project document served as both a knowledge transfer and a risk assessment, laying out a seven-step upgrade plan with explicit rollback strategy.

The assistant chose to proceed, a decision that reveals its assessment that the research was sufficient, the risks manageable, and the potential reward—unblocking two dead-end optimizations that could transform EAGLE-3 from a net-negative to a net-positive—justified the investment.

The Infrastructure Ordeal: Navigating ABI Compatibility

The CUDA 13 upgrade itself was a multi-hour ordeal that tested the assistant's diagnostic skills to their limit. The goal was to assemble a stable stack comprising CUDA 13.0.1, PyTorch 2.9.1+cu130, sgl-kernel 0.3.21+cu130, flashinfer 0.6.4, and SGLang v0.5.9. What followed was a cascade of compatibility issues that would have derailed a less persistent engineer.

The first major challenge was the ABI (Application Binary Interface) boundary. When the assistant installed sgl-kernel 0.3.21+cu130 and flashinfer 0.6.4, the server crashed with a symbol mismatch error: _ZNK3c106SymInt6symbolEv—a mangled C++ symbol from PyTorch's c10::SymInt class [4]. The root cause was that flashinfer-python 0.6.4's dependency resolver had silently downgraded PyTorch from 2.9.1+cu130 to 2.10.0, which was compiled against CUDA 12.8 rather than CUDA 13. The result was a stack where sgl-kernel (compiled against CUDA 13) and PyTorch (compiled against CUDA 12.8) used different C++ ABI conventions, causing symbol resolution failures at runtime.

The assistant diagnosed this through a methodical process: using nm to inspect symbol tables, checking library paths with ldconfig, and tracing the dependency chain through pip's resolver [5]. The fix required pinning flashinfer to a specific version that didn't conflict with the cu130 PyTorch, and ensuring that all components were compiled against the same CUDA toolkit.

A second ABI crisis emerged when the assistant discovered that libnvrtc.so.13 was not being found by the runtime linker. The CUDA 13 toolkit had been installed alongside CUDA 12.8, but the system's ldconfig cache still pointed to the old library paths. The assistant resolved this by running ldconfig to update the linker cache and by setting LD_LIBRARY_PATH to prioritize the CUDA 13 libraries [6].

These ABI battles consumed the majority of the upgrade effort, but they were essential. Without a fully consistent stack where every component was compiled against the same CUDA version, the Blackwell-native optimizations would remain inaccessible. The assistant's persistence in diagnosing and fixing each ABI mismatch was what ultimately made the breakthrough possible.

The Baseline Breakthrough: 92.6 tok/s

After the stack was finally assembled and verified, the assistant ran the first baseline benchmark. The result: 92.6 tok/s—a 3.5% improvement over the previous baseline of 89.5 tok/s [7]. This was a welcome validation that the CUDA 13 upgrade had delivered immediate value even before testing the Blackwell-native optimizations.

But the path to this result was not straightforward. The first benchmark attempt produced only 83.9 tok/s—a 6% regression from the old baseline. The assistant diagnosed the cause by inspecting the server log and discovering that SGLang v0.5.9 was defaulting to the triton attention backend, whereas the old baseline had used flashinfer. Restarting with --attention-backend flashinfer recovered and exceeded the old baseline [8].

This diagnostic episode is instructive. The assistant did not panic at the regression. Instead, it systematically enumerated four possible explanations (SGLang version overhead, attention backend selection, PyTorch version differences, NCCL version changes) and tested each one by examining the server log. The fix was a single flag change, but the diagnostic process—observe, hypothesize, test, conclude—is the pattern that drove the entire project forward.## The Blackwell-Native Optimizations: FlashInfer Allreduce Fusion

With the CUDA 13 baseline validated, the assistant turned to the optimizations that had motivated the entire upgrade. The first test was FlashInfer allreduce fusion. The assistant restarted the server with --enable-flashinfer-allreduce-fusion and ran the benchmark.

The result: 92.7 tok/s—essentially identical to the 92.6 tok/s baseline [9]. A casual observer might have been disappointed. After all the effort of the CUDA 13 upgrade, the optimization that was supposed to be the prize delivered zero benefit for the single-stream baseline.

But the assistant's analysis revealed a deeper understanding. In the baseline decode path, the all-reduce operations are already well-hidden behind compute. Each GPU has enough independent matrix multiplication work to keep the streaming multiprocessors busy while the all-reduce is in flight. Fusing the all-reduces doesn't change this overlap; it just changes how the communication is implemented. Since compute is the bottleneck, not communication, fusion provides no benefit [10].

The assistant correctly predicted that the real benefit would appear in the EAGLE-3 verify pass, where tiny batches (1–3 tokens) make each all-reduce's latency dominate. With minimal compute to overlap with communication, the all-reduce latency becomes the primary bottleneck, and fusing multiple small all-reduces into larger ones can dramatically reduce overhead. This prediction would soon be validated.

Torch Symmetric Memory: Patching SGLang for SM120

The second Blackwell-native optimization was Torch symmetric memory. Unlike FlashInfer allreduce fusion, which worked immediately after the CUDA 13 upgrade, Torch symmetric memory required source code patches to SGLang itself.

The assistant traced the crash to two files. The first was all_reduce_utils.py, which defined TORCH_SYMM_MEM_ALL_REDUCE_MAX_SIZES only for compute capabilities 9 and 10, not 12 (Blackwell). The second was torch_symm_mem.py, which had a _WORLD_SIZES_MULTIMEM mapping that also lacked SM120 [11].

The assistant's diagnostic approach was methodical. A grep for "12" in the relevant files revealed the missing entries. A cat of the 15-line all_reduce_utils.py confirmed the structure. The fix was a twelve-line patch that added compute capability 12 entries with the same buffer sizes as SM100 (64 MiB for 2 and 4 GPUs, 128 MiB for 6 and 8 GPUs) and added SM120 to the _WORLD_SIZES_MULTIMEM mapping [12].

After clearing the Python cache to ensure the patches took effect, the assistant benchmarked Torch symmetric memory. The result: 92.6 tok/s—again identical to the baseline [13]. Like FlashInfer allreduce fusion, Torch symmetric memory provided no benefit for single-stream decode. But the crucial thing was that it no longer crashed. Both optimizations were now available for the EAGLE-3 verify pass, where their impact would be transformative.

The EAGLE-3 Integration: Bridging the Interface Gap

With both Blackwell-native optimizations validated, the assistant turned to the final challenge: integrating EAGLE-3 speculative decoding with the patched stack. This required not just enabling the right flags but also patching SGLang's model code to support the EAGLE-3 interface.

The KimiK25 model (KimiK25ForConditionalGeneration) wraps a DeepseekV3ForCausalLM as self.language_model rather than inheriting from it. This wrapper pattern creates an interface gap: any methods that the EAGLE-3 worker expects to call on the target model must be explicitly delegated through the wrapper. The assistant discovered this gap through trial and error, with each server crash revealing the next missing method [14].

The first missing method was get_embed_and_head, which the EAGLE-3 worker calls to access the target model's embedding and language model head weights. The assistant added a delegation method that forwarded the call to self.language_model.get_embed_and_head(). The second missing method was set_eagle3_layers_to_capture, which configures which transformer layers should expose their hidden states for the draft model. The assistant located the canonical implementation in the DeepseekV2 source via grep and replicated it via delegation [15].

The third missing method was set_embed_and_head, which the assistant had initially implemented incorrectly in a six-line patch that had to be surgically reverted and rewritten [16]. Each fix followed the same pattern: identify the error, locate the canonical implementation in the parent model, understand the method's signature and behavior, and add a delegation method on the KimiK25 wrapper.

This trial-and-error approach was necessary because the full EAGLE-3 interface contract was not documented anywhere. The assistant had to discover each method by attempting to launch the server and observing which AttributeError occurred next. This is a common challenge in working with complex ML frameworks: the interface between components is defined implicitly by what methods are called, not by an explicit specification.

The Moment of Truth: EAGLE-3 at 96.1 tok/s

With all patches applied and all delegation methods in place, the assistant launched the EAGLE-3 server with the full configuration: --enable-flashinfer-allreduce-fusion, --attention-backend flashinfer, --speculative-algorithm EAGLE3, and the draft model path pointing to the fine-tuned EAGLE-3 drafter [17].

The benchmark result was a breakthrough: 96.1 tok/s—a 77.6% improvement over the pre-CUDA-13 EAGLE-3 performance of 54.1 tok/s, and a 3.8% improvement over the baseline of 92.6 tok/s [18]. For the first time in the project's history, speculative decoding was a net positive.

The mechanism was exactly what the assistant had predicted. FlashInfer allreduce fusion slashed the verify pass latency by merging the 122 tiny all-reduces into fewer, larger operations. Where the verify pass had previously been spending ~80% of its time waiting on NCCL acknowledgments, the fusion optimization dramatically reduced this overhead, making the verify pass fast enough that the draft model's predictions could provide a net throughput gain.

The high variance in results (87.5–103.5 tok/s across 10 runs) was characteristic of speculative decoding, where acceptance rates vary depending on how predictable the generated text is. When the draft model happens to propose tokens that the target model accepts, the effective throughput skyrockets. When the draft model guesses poorly, the verify step rejects most candidates and the system falls back to generating one token at a time.

The Discovery: EAGLE-3's Concurrency Dependency

The triumph of 96.1 tok/s was followed by an important discovery. The user requested parallel benchmarking at multiple concurrency levels (C=2, 5, 10, 30, 70, 100, 250) to understand how EAGLE-3 behaved under realistic multi-user load [19].

The assistant recognized that the existing benchmark infrastructure was inadequate for this task. The benchmark_eagle3.py script was a serial benchmark—it sent one request at a time and measured per-request latency. It could not answer the question: "What happens to total system throughput when 30 clients are hammering the server simultaneously?"

In response, the assistant wrote a new parallel benchmark from scratch: benchmark_parallel.py. This script managed concurrent HTTP requests, collected timing data, computed aggregate statistics, and handled error cases gracefully. It was designed to measure total throughput across all concurrent requests—a fundamentally different metric from the serial benchmark's per-request latency.

The results were striking:

| Concurrency | Throughput (tok/s) | Per-Req (tok/s) | Avg Latency (s) | |---|---|---|---| | 1 | 77.5 | 77.7 | 6.6 | | 2 | 125.1 | 62.9 | 8.2 | | 5 | 183.2 | 37.3 | 13.8 | | 10 | 239.6 | 24.8 | 20.8 | | 30 | 299.5 | 10.4 | 49.6 | | 70 | 337.7 | 5.7 | 96.2 | | 100 | 338.8 | 4.1 | 134.3 | | 250 | 340.9 | 2.9 | 243.6 |

The data told a clear story: throughput scaled nearly linearly from C=1 to C=10, then began to saturate. By C=70, the system hit a ceiling around 338–340 tok/s, and adding more concurrency yielded almost no additional throughput. Meanwhile, per-request throughput collapsed from 77.7 tok/s at C=1 to just 2.9 tok/s at C=250, and latency ballooned from 6.6 seconds to over 4 minutes.

This was the critical insight: speculation's value depended on how busy the GPUs were. At low concurrency, the GPU had idle cycles that the draft model could exploit. At high concurrency, every cycle was already occupied, and the verify pass became pure overhead.

The Strategic Pivot: Dynamic Speculation Disabling

The user's response demonstrated a sophisticated understanding of the system's behavior: "Can we tell (or mod) sglang to disable speculation at certain concurrency? Probably need to bench w/o speculation to decide a threshold (also maybe for 2-3 tok reduced speculation)" [20].

This message was the strategic pivot. The user recognized that the optimization problem had fundamentally changed—from "make speculation work" to "know when to use it." The assistant immediately embraced the insight, laying out a complete investigation plan:

1. Benchmark baseline (no speculation) at C=1, 2, 5, 10, 30, 70, 100, 250
2. Compare EAGLE-3 vs baseline at each concurrency to find crossover point
3. Investigate SGLang dynamic speculation disable based on concurrency/load
4. Test reduced speculation (fewer steps/draft tokens) at high concurrency This plan reflected a nuanced understanding of the problem. The crossover point—where speculation's benefit equals its cost—would be empirically determined by comparing the two throughput curves. The reduced speculation option acknowledged that the optimal policy might not be binary (on/off) but continuous (reduce speculation intensity as load increases).

Documentation: The Discipline of Recording Results

Throughout the optimization campaign, the assistant maintained a meticulous experiment log at /home/theuser/glm-kimi-sm120-rtx6000bw/eagle-fast-verify.md. This document had grown to 399 lines and contained the complete history of attempted optimizations, their results, and the reasoning behind each approach.

The documentation process was itself a multi-step ritual. The assistant read the experiment log to understand its current state, reviewed the results table—a scoreboard of failure that showed entries like "EAGLE-3 2-step: 54.1 tok/s (40% slower than baseline)" and "FlashInfer fusion SM120: FAILED (JIT doesn't recognize SM120)"—and applied edits to update both the narrative and the results table, transforming the document from a record of failure into a record of success.

The milestone summary consolidated everything into a clean, referenceable format: stack specification, key patches, and performance results. This documentation discipline ensured that the hard-won knowledge would survive beyond the current session and could be referenced by future optimization efforts.

Lessons Learned

This session offers several enduring lessons for ML infrastructure engineering:

1. Know when to invest in foundational upgrades. The assistant could have continued trying tactical fixes—tweaking NCCL parameters, trying different all-reduce algorithms, adjusting batch sizes. But the diagnostic evidence pointed to a root cause that was deeper than any tactical fix could address. The CUDA 13 upgrade was a high-risk, high-effort intervention, but it was the only intervention that could unblock the two most promising optimizations. The willingness to invest in foundational infrastructure when the evidence supports it is a hallmark of mature engineering judgment.

2. ABI compatibility is a silent project killer. The ABI mismatches that plagued the CUDA 13 upgrade are a reminder that software stacks are held together by fragile conventions. A single component compiled against the wrong CUDA version can cause symbol resolution failures that manifest as cryptic runtime crashes. The assistant's methodical approach to diagnosing ABI issues—using nm to inspect symbol tables, checking library paths with ldconfig, tracing dependency chains through pip—is a template for anyone facing similar issues.

3. Null results are data, not failures. When FlashInfer allreduce fusion and Torch symmetric memory both produced throughput identical to the baseline, the assistant did not conclude that the optimizations were useless. Instead, it reasoned about why they didn't help the baseline and predicted where they would help. This ability to extract information from null results is what separated the successful optimization from a random search.

4. Interface contracts must be explicitly documented or discovered. The trial-and-error approach to discovering the EAGLE-3 interface methods was inefficient but necessary. In complex ML frameworks, the interface between components is often defined implicitly by what methods are called. This creates a maintenance burden for wrapper models and a debugging challenge for anyone integrating new architectures. Explicit documentation of interface contracts would dramatically reduce this friction.

5. Optimization is workload-dependent. The same optimization that provides zero benefit for single-stream decode can be transformative for speculative decoding verification. Understanding which workloads benefit from which optimizations is more important than knowing the raw performance numbers. The assistant's ability to reason about the compute/communication overlap characteristics of different workloads was the key to correctly targeting the optimizations.

6. The most important optimization decision is when to use it. The parallel benchmarking revealed that speculation's value is not constant—it helps at low concurrency but hurts at high concurrency. This insight transformed the optimization problem from a static configuration choice into a dynamic, load-aware policy decision. The next phase of the project—implementing dynamic speculation disabling based on server load—represents a fundamentally more sophisticated approach to optimization.

Conclusion

The CUDA 13 upgrade saga is a testament to the power of systematic reasoning in ML infrastructure optimization. What began as a project stuck in a frustrating impasse—with every optimization attempt crashing on SM120 incompatibility—was transformed through a strategic pivot, a grueling infrastructure overhaul, and a series of targeted source code patches into a decisive breakthrough.

The numbers tell the story: from 54.1 tok/s (40% slower than baseline) to 96.1 tok/s (3.8% faster than baseline)—a 77.6% improvement in speculative throughput. But the real achievement is not the final number; it is the reasoning process that made that number possible. The assistant's ability to diagnose the root cause, formulate a hypothesis, execute a high-risk upgrade, navigate ABI compatibility challenges, patch source code for a new GPU architecture, and systematically validate each optimization is a model for how complex infrastructure problems should be approached.

The session ends with a clear direction for future work: dynamic speculation disabling based on server load. But the foundation has been laid. The CUDA 13 upgrade unblocked the Blackwell-native optimizations that were the key to making EAGLE-3 work. The parallel benchmarking revealed the nuanced, load-dependent nature of speculation's value. The rest is optimization on a working foundation—a far more comfortable position than the dead-end impasse where the project began.

In the end, the most valuable optimization was not a faster kernel or a better algorithm—it was the recognition that the right question is not "how do we make this faster?" but "when should we use this at all?"## References

[1] "The Strategic Pivot: How a Comprehensive State-of-the-Project Message Charted the Course from Dead Ends to the CUDA 13 Breakthrough" — Message 5262 analysis covering the root cause diagnosis and upgrade plan.

[2] "The Pivot Point: How a Single Sentence Authorized a High-Stakes CUDA 13 Upgrade" — Message 5263 analysis of the user's delegation and authorization.

[3] "The Pivot Point: How Two WebFetches Transformed a CUDA 13 Upgrade from Investigation to Execution" — Message 5268 analysis of the research phase.

[4] "The ABI Detective: Diagnosing a C++ Symbol Mismatch in the CUDA 13 Stack Upgrade" — Message 5306 analysis of the _ZNK3c106SymInt6symbolEv symbol mismatch.

[5] "The Symbol Table Tells the Truth: Diagnosing an ABI Mismatch with nm" — Message 5304 analysis of the nm diagnostic approach.

[6] "The Library Path That Almost Broke the Stack: Diagnosing libnvrtc.so.13 After a CUDA 13 Upgrade" — Message 5321 analysis of the runtime linker issue.

[7] "The CUDA 13 Breakthrough: How a 3.5% Baseline Gain Unlocked the Path Forward" — Message 5357 analysis of the 92.6 tok/s baseline result.

[8] "The Diagnostic Pivot: When a CUDA 13 Upgrade Looked Like a Regression" — Message 5352 analysis of the attention backend regression diagnosis.

[9] "The Pivot Point: Validating FlashInfer Allreduce Fusion on Blackwell and Charting the Path Forward" — Message 5361 analysis of the 92.7 tok/s fusion result.

[10] "The Pivot: Why Blackwell-Native Optimizations Don't Help Baseline But Transform EAGLE-3 Speculative Decoding" — Message 5375 analysis of the compute/communication overlap reasoning.

[11] "Diagnosing the Torch Symmetric Memory Crash: A Pivotal Debugging Step in the CUDA 13 Upgrade" — Message 5363 analysis of the KeyError: 12 crash.

[12] "The Twelve-Line Patch: Unblocking Torch Symmetric Memory on Blackwell GPUs" — Message 5368 analysis of the SM120 patch.

[13] "The Quiet Confirmation: When Torch Symmetric Memory Finally Worked on Blackwell" — Message 5374 analysis of the 92.6 tok/s symmetric memory result.

[14] "The Architecture Gap: When a Wrapper Model Breaks Speculative Decoding" — Message 5387 analysis of the KimiK25 interface gap.

[15] "The Missing Method: How a Single AttributeError Uncovered the Fragile Interface Between SGLang and KimiK25" — Message 5395 analysis of the set_eagle3_layers_to_capture discovery.

[16] "The Six-Line Mistake: A Micro-Surgery on SGLang's EAGLE-3 Integration" — Message 5389 analysis of the incorrect patch and revert.

[17] "The Moment of Truth: Launching EAGLE-3 Speculative Decoding on a Patched CUDA 13 Stack" — Message 5393 analysis of the full launch command.

[18] "The CUDA 13 Breakthrough: Unlocking Blackwell-Native Performance on 8× RTX PRO 6000 GPUs" — Subagent summary of the final 96.1 tok/s result.

[19] "From 404 to Actionable Intelligence: How Systematic Research Unlocked SGLang's CUDA 13 Compatibility on Blackwell GPUs" — Subagent summary of the concurrency dependency discovery.

[20] "The Pivot: How a Single User Message Reshaped the EAGLE-3 Optimization Strategy" — Message 5418 analysis of the dynamic speculation disabling request.