The CUDA Graph Benchmark: When Speculative Decoding Still Can't Beat the Baseline

Introduction

In the long arc of debugging and optimizing EAGLE-3 speculative decoding for the Kimi-K2.5 model, message [msg 3635] arrives as a moment of both validation and sobering reality. The message is deceptively brief — a single benchmark run with a short preamble — but it represents the culmination of a multi-day debugging odyssey spanning hidden state concatenation bugs, flag mismatches, draft model architecture issues, and now, finally, CUDA graph optimization. The results it reports are a mixed bag: CUDA graphs work, they improve throughput dramatically, but they still fall short of the non-speculative baseline. This message is the data point that forces a strategic pivot from tuning hyperparameters to scaling training data.

The Message

The assistant writes:

CUDA graphs captured successfully for both target and draft models. Now let me benchmark:

It then executes the benchmark script and receives:

=== Single-stream EAGLE3 benchmark ===
  Prompt 1: 886 tokens in 12.7s = 69.8 tok/s (prompt: 19 tok)
  Prompt 2: 1024 tokens in 11.3s = 90.5 tok/s (prompt: 20 tok)
  Prompt 3: 1024 tokens in 14.5s = 70.6 tok/s (prompt: 24 tok)
  Prompt 4: 995 tokens in 13.4s = 74.2 tok/s (prompt: 18 tok)
  Prompt 5: 1024 tokens in 14.2s = 72.3 tok/s (prompt: 18 tok)

  Average: 74.9 tok/s (4953 tokens / 66.1s)

Five prompts, five data points, one average. The numbers tell a story that the assistant will need to unpack in subsequent reasoning.

The Road to This Benchmark

To understand why this message matters, one must trace the path that led to it. The team had been working on deploying EAGLE-3 speculative decoding — a technique where a lightweight "draft" model predicts multiple future tokens, which the full model then verifies in parallel. The promise is higher throughput by generating multiple tokens per forward pass of the large model.

The journey had been fraught. In <msg id=3611-3614>, the team discovered a critical bug: the SGLang server had been started with --speculative-algorithm EAGLE instead of EAGLE3. This seemingly trivial flag mismatch had devastating consequences. The is_eagle3() check in the codebase is strict — only the string &#34;EAGLE3&#34; triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only hidden states instead of the expected 21504-dimensional concatenated states (7168 × 3 layers). This caused the fusion layer (fc) to be silently bypassed, rendering all trained weights useless. The acceptance length was stuck at 1.0 — meaning no draft tokens were ever accepted.

After fixing the flag and restarting the server (<msg id=3615-3618>), the acceptance length jumped to ~2.0-2.5. The draft model was finally receiving correct hidden states and making useful predictions. But the throughput was only 56.7 tok/s — far below the 90 tok/s non-speculative baseline. The speculation overhead (running the draft model, verifying candidates) was consuming more time than the ~2.3× acceptance length saved.

The assistant then methodically explored alternatives. Reducing draft tokens from 16 to 5 (<msg id=3622-3623>) improved the accept rate (from ~15% to ~41%) but actually reduced throughput to 53.2 tok/s. The AQ-MedAI drafter, a general-purpose K2 drafter trained on more data, performed even worse at 50.5 tok/s (<msg id=3628-3629>). Each experiment confirmed the same fundamental problem: with an acceptance length of ~2.1, speculation overhead was too high.

The CUDA Graph Hypothesis

The one remaining lever was CUDA graphs. Without CUDA graphs, every step of the speculative decoding pipeline — draft model forward pass, verification, token acceptance — incurs the full overhead of launching individual CUDA kernels. CUDA graphs capture the entire sequence of operations as a single fused graph, eliminating kernel launch overhead and reducing synchronization costs. The assistant had previously avoided CUDA graphs due to stability concerns (earlier attempts had caused hangs, as seen in segment 24's debugging), but the performance gap was large enough to warrant the risk.

The preamble of message [msg 3635] — "CUDA graphs captured successfully for both target and draft models" — is itself a significant achievement. It confirms that the SGLang server's CUDA graph capture mechanism works correctly for both the large target model (Kimi-K2.5 with 8-way tensor parallelism) and the small draft model. The capture process, visible in the log lines from [msg 3634], took approximately 1.4 seconds per GPU and consumed about 0.49 GB of memory, leaving ample headroom on the 80 GB GPUs.## Interpreting the Numbers

The benchmark results in message [msg 3635] show a significant improvement over the previous CUDA-graph-disabled runs. The average throughput of 74.9 tok/s represents a 40% improvement over the 53.2 tok/s seen with CUDA graphs disabled and 5 draft tokens ([msg 3622]), and a 32% improvement over the 56.7 tok/s with 16 draft tokens ([msg 3612]). CUDA graphs are clearly effective at reducing speculation overhead.

However, the 74.9 tok/s average still falls short of the 90 tok/s non-speculative baseline. The gap is approximately 17%. More revealing is the variance across prompts: Prompt 2 achieved 90.5 tok/s — essentially matching the baseline — while Prompt 1 achieved only 69.8 tok/s and Prompt 3 only 70.6 tok/s. This variance suggests that the draft model's acceptance rate depends on the specific content being generated. Some sequences are more predictable by the draft model than others.

The assistant's thinking at this point, visible in the surrounding context, reveals a crucial insight: the acceptance length of ~2.1 is fundamentally insufficient. The EAGLE-3 paper's scaling curves suggest that acceptance length scales with training data quantity. The draft model was trained on only 10,000 samples — a tiny fraction of what the paper used. The assistant's reasoning correctly identifies that hyperparameter tuning (draft tokens, steps, CUDA graphs) has been exhausted as a lever. The only remaining path to meaningful speedup is improving the draft model itself through more training data.

The Strategic Pivot

This message is the inflection point. The assistant had been iterating on configuration parameters — trying different numbers of draft tokens, enabling and disabling CUDA graphs, testing alternative draft models. Each experiment produced incremental data but no breakthrough. Message [msg 3635] provides the clearest evidence yet that the configuration is no longer the bottleneck: even with CUDA graphs working correctly and the hidden state bug fixed, the draft model's predictions are simply not good enough.

The assistant's subsequent actions confirm this pivot. In the messages that follow the target message, the focus shifts dramatically from tuning to data scaling. The team launches a massive inference pipeline to generate responses for 83,288 prompts through the Kimi-K2.5 model, creating a 10× larger training dataset. Ten parallel agents search for agentic coding, reasoning, and general chat datasets. The full pipeline plan is documented in train_plan_v4.md. A progress monitor script is created to track the ~24-55 hour inference run.

Assumptions and Their Validation

Several assumptions underpin this message. First, the assistant assumes that CUDA graphs would capture correctly — an assumption that was not trivial given earlier stability issues with the SM120 architecture. The successful capture validates that SGLang's CUDA graph support works for this specific model and hardware configuration.

Second, the assistant assumes that the benchmark script (/tmp/eagle3_bench.py) provides a representative measure of single-stream throughput. The five prompts cover different topics and response lengths, but they are all single-turn completions with temperature 0. This is a reasonable proxy for the target use case but may not capture multi-turn or higher-temperature scenarios.

Third, there is an implicit assumption that the 90 tok/s baseline is stable and reproducible. Earlier tuning work (segment 25) had achieved 90 tok/s through NCCL protocol tuning, but this baseline was measured under specific conditions. The comparison assumes these conditions are equivalent.

What Knowledge This Message Creates

Message [msg 3635] creates several pieces of output knowledge:

  1. Empirical validation: CUDA graphs work correctly for EAGLE-3 speculative decoding on the Kimi-K2.5 model with 8-way tensor parallelism on RTX PRO 6000 Blackwell GPUs.
  2. Quantified performance: The best achievable throughput with the current 10K-sample draft model is 74.9 tok/s, approximately 17% below the non-speculative baseline.
  3. Variance characterization: Single-stream performance varies significantly across prompts (69.8 to 90.5 tok/s), indicating content-dependent acceptance behavior.
  4. Diminishing returns: Configuration tuning has reached its limit. The acceptance length of ~2.1 is a hard constraint imposed by draft model quality, not by system configuration.
  5. Decision trigger: The data justifies the significant investment required for 10× data scaling. Without this benchmark result, the team might have continued chasing configuration improvements indefinitely.

The Broader Context

This message sits at the intersection of several technical domains: speculative decoding algorithms, CUDA optimization, distributed inference systems, and machine learning training pipelines. The assistant must reason across all of these simultaneously — understanding how a flag string mismatch affects tensor dimensions, how CUDA graph capture interacts with tensor parallelism, how acceptance rate translates to throughput, and how training data quantity affects model quality.

The message also reveals the iterative nature of systems optimization. Each experiment eliminates one variable and narrows the search space. The hidden state bug was the first major discovery. The flag mismatch was the second. CUDA graph effectiveness was the third. Each layer of debugging revealed a new constraint, and message [msg 3635] represents the point where the team finally reaches the fundamental limit: model quality, not system configuration.

Conclusion

Message [msg 3635] is a turning point in the EAGLE-3 optimization effort. It confirms that the system is now correctly configured — CUDA graphs work, hidden states are properly concatenated, and the draft model is making predictions that get accepted at a reasonable rate. But it also delivers the sobering news that these improvements are not enough. The 74.9 tok/s result, while a significant improvement over earlier attempts, still trails the baseline by a meaningful margin. The message forces a strategic shift from configuration tuning to data scaling, setting the stage for the massive inference pipeline that follows. In the narrative of this coding session, it is the moment when the team accepts that there are no more easy wins — only hard work.