The Benchmark That Confirmed the Gap: 82.3 tok/s and the Limits of EAGLE-3 Speculation

Introduction

In the course of deploying EAGLE-3 speculative decoding for the Kimi-K2.5 language model on an 8-GPU server, a single benchmark result arrived as message [msg 3640] in the conversation. At first glance, it appears to be nothing more than a routine performance measurement: five prompts evaluated, an average throughput of 82.3 tokens per second, and a notable burst of 101.8 tok/s on one code-generation prompt. But this message sits at a critical inflection point in the project. It represents the culmination of a multi-hour debugging and optimization marathon, and it delivers a clear verdict: even with the hidden state concatenation bug fixed, even with CUDA graphs enabled, even with the draft token count tuned down to reduce overhead, EAGLE-3 speculative decoding still cannot beat the non-speculative baseline of 90 tok/s. The gap is small—roughly 9%—but it is definitive. This article examines the message in depth, exploring the reasoning that produced it, the decisions embedded in its configuration, the assumptions it validates and invalidates, and the pivot it forces.

The Message Itself

The message is a direct transcript of a benchmark execution on the remote server:

=== Single-stream EAGLE3 benchmark ===
  Prompt 1: 937 tokens in 12.4s = 75.6 tok/s (prompt: 19 tok)
  Prompt 2: 1024 tokens in 10.1s = 101.8 tok/s (prompt: 20 tok)
  Prompt 3: 1024 tokens in 13.4s = 76.5 tok/s (prompt: 24 tok)
  Prompt 4: 1024 tokens in 12.9s = 79.4 tok/s (prompt: 18 tok)
  Prompt 5: 1024 tokens in 12.4s = 82.4 tok/s (prompt: 18 tok)

  Average: 82.3 tok/s (5033 tokens / 61.2s)

The assistant ran a bash command executing a Python benchmark script (/tmp/eagle3_bench.py) via SSH on the remote machine root@10.1.230.174. The script generated five prompts of varying types (short prompts of 18–24 tokens) and measured the throughput of the EAGLE-3 speculative decoding server that had been started moments earlier. The results show a clear pattern: Prompt 2, which the assistant later identifies as a code-generation task, achieves a remarkable 101.8 tok/s—the only configuration in the entire session to exceed the non-speculative baseline. But the other four prompts cluster in the 75–82 tok/s range, and the average settles at 82.3 tok/s.

Why This Message Was Written: The Optimization Trajectory

To understand why this particular benchmark was run, we must trace the trajectory of the preceding messages. The assistant had spent the prior hour systematically working through a chain of debugging and optimization steps, each motivated by a clear hypothesis.

Step 1: The Hidden State Concatenation Bug (Messages [msg 3613]–[msg 3615]). The assistant had discovered that the EAGLE-3 draft model was receiving 7168-dimensional hidden states (single final layer) instead of the expected 21504-dimensional concatenated states (three layers: 2, 30, and 58). The root cause was a flag mismatch: the server had been started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in SGLang's codebase is strict—only the string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states. With EAGLE, the draft model received single-layer states, causing the trained fc fusion layer to be silently bypassed and all trained weights to be useless. After restarting with the correct flag, the accept_len jumped from 1.0 (no tokens accepted) to ~2.1–2.5, confirming the fix worked.

Step 2: Without CUDA Graphs (Messages [msg 3618]–[msg 3623]). The assistant tested the corrected configuration with num_draft_tokens=16 and CUDA graphs disabled. The result was 53.2 tok/s—far below the 90 tok/s baseline. The assistant correctly diagnosed the bottleneck: without CUDA graphs, each speculation step incurred significant Python-level overhead for launching CUDA kernels, and the accept_len of ~2.1 was insufficient to compensate.

Step 3: The AQ-MedAI Drafter Comparison (Messages [msg 3624]–[msg 3629]). The assistant tested an alternative draft model from AQ-MedAI, trained on DeepSeek-K2 data. It achieved 50.5 tok/s with accept_len ~1.8–2.1, slightly worse than the custom K2.5-trained drafter. This confirmed that the custom drafter was better but still data-limited.

Step 4: CUDA Graphs with 16 Draft Tokens (Messages [msg 3630]–[msg 3636]). The assistant enabled CUDA graphs, which dramatically reduced per-step overhead by pre-recording GPU kernel launch sequences. The result: 74.9 tok/s average, a 40% improvement over the non-CUDA-graph configuration. But still below the 90 tok/s baseline.

Step 5: CUDA Graphs with 5 Draft Tokens (Messages [msg 3637]–[msg 3640]). This is where message [msg 3640] enters. The assistant hypothesized that reducing draft tokens from 16 to 5 would reduce verification overhead. With accept_len ~2.1, verifying 16 draft tokens meant that ~14 tokens were being rejected per step—a massive waste. With 5 draft tokens, the accept rate would rise from ~13% to ~42%, reducing the overhead per accepted token. The assistant restarted the server with --speculative-num-draft-tokens 5 and ran the benchmark, producing the 82.3 tok/s result.

How Decisions Were Made: The Configuration Choices

The benchmark result in message [msg 3640] is not a random data point—it is the product of deliberate configuration choices, each reflecting a hypothesis about the system's behavior.

Choice of draft token count (5 instead of 16). The assistant's reasoning is visible in the preceding messages. With 16 draft tokens and accept_len ~2.1, the accept rate was ~0.13, meaning 87% of draft tokens were rejected. Each rejected token still required a verification forward pass through the full target model. By reducing to 5 draft tokens, the accept rate rose to ~0.42 (as confirmed in message [msg 3641]), meaning 42% of draft tokens were accepted. The trade-off is that the maximum possible speedup is lower with fewer draft tokens, but the efficiency per verification step is higher.

Choice of CUDA graphs enabled. The assistant explicitly removed the --disable-cuda-graph flag from the server launch command. CUDA graphs capture the entire sequence of GPU kernel launches into a single graph that can be replayed with minimal CPU overhead. This is critical for speculative decoding, where the draft model and verification steps involve many small kernel launches that would otherwise incur significant Python-to-CUDA dispatch overhead.

Choice of num_steps=3 and topk=4. These parameters control the tree attention structure of EAGLE-3. With 3 steps and top-4 branching, the draft model generates up to 4 candidate tokens at each of 3 positions, producing a tree of up to 1 + 4 + 4 = 9 candidate sequences (though the actual implementation may differ). The assistant kept these at the same values used in previous tests, focusing the optimization on draft token count and CUDA graphs.

Choice of single-stream benchmark. The assistant used a single-stream benchmark (one request at a time) rather than a throughput benchmark with concurrent requests. This was deliberate: the goal was to measure the latency improvement (or lack thereof) from speculative decoding, which is most relevant for interactive single-user scenarios. The assistant had previously measured batch throughput at ~830 tok/s for the non-speculative server, but that metric is less relevant for evaluating speculation quality.

Assumptions Made by the Assistant

The benchmark and its interpretation rest on several assumptions, some explicit and some implicit.

Assumption 1: The benchmark script is representative. The assistant assumes that /tmp/eagle3_bench.py generates prompts that are representative of real usage patterns. The five prompts produce outputs of 886–1024 tokens from short inputs of 18–24 tokens. This tests the steady-state decoding performance but may not capture the behavior of longer prompts or more varied generation lengths.

Assumption 2: Reducing draft tokens improves throughput. The assistant assumes that the overhead of verifying additional draft tokens outweighs the benefit of having more candidates. This is a reasonable assumption given the accept_len of ~2.1, but it implicitly assumes that accept_len is independent of the number of draft tokens—that is, the draft model's predictions are equally good regardless of how many candidates are generated. In practice, EAGLE-3's tree attention may produce better candidates with more draft tokens, so reducing the count could also reduce the quality of the best candidate.

Assumption 3: The 90 tok/s baseline is stable. The assistant repeatedly references a non-speculative baseline of 90 tok/s, established in earlier benchmarks (segment 24). This assumes that the baseline server configuration is identical to the speculative configuration except for the speculation flags—same model, same tensor parallelism, same memory fraction, same NCCL settings. If the baseline was measured under slightly different conditions (e.g., different server load, different CUDA graph state), the comparison may not be perfectly fair.

Assumption 4: CUDA graphs are fully compatible with EAGLE-3. The assistant had previously encountered CUDA graph hangs with EAGLE-3 (segment 23), and the fact that they worked in this test was treated as a success. The assumption is that once CUDA graphs are captured, they remain valid for the lifetime of the server. This is generally true, but changes in input shapes or batch sizes can invalidate captured graphs.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption is that reducing draft tokens from 16 to 5 would bridge the gap to the baseline. The improvement from 74.9 tok/s to 82.3 tok/s is meaningful—a 10% gain—but it still leaves the speculative configuration 9% slower than the non-speculative baseline. The assistant's implicit model was that speculation overhead scales linearly with draft token count, and that reducing tokens would proportionally reduce overhead. But the data shows diminishing returns: the overhead is not purely proportional to draft token count, because there are fixed costs (loading the draft model, managing the speculation state) that do not scale down.

A more subtle issue is that the assistant may have been comparing against an imperfect baseline. The 90 tok/s non-speculative figure was measured with CUDA graphs enabled on the same hardware. But the assistant's earlier benchmarks (segment 24) showed that the baseline itself varied depending on prompt type and system state. The 90 tok/s figure may represent an optimistic upper bound rather than a stable average. If the true average baseline is closer to 85 tok/s, then the 82.3 tok/s speculative result is essentially competitive.

The assistant also assumed that accept_len would remain constant when reducing draft tokens. In fact, message [msg 3641] shows accept_len ~1.9–2.4 with 5 draft tokens, compared to ~2.1–2.5 with 16 draft tokens—a slight decrease. This suggests that having fewer candidates does reduce the best-case acceptance length, though the effect is small.

Input Knowledge Required

To fully understand message [msg 3640], a reader needs knowledge spanning several domains:

Speculative decoding. The core concept: a small "draft" model generates candidate tokens cheaply, and the large "target" model verifies them in parallel. The speedup depends on the acceptance length (how many draft tokens are accepted per verification step) and the relative costs of the draft and target models.

EAGLE-3 architecture. EAGLE-3 is a specific speculative decoding method that uses a lightweight transformer with tree attention to generate multiple candidate sequences. It requires hidden states from intermediate layers of the target model, which are concatenated and fed into a fusion layer (fc). The hidden state dimension is critical: 7168 (single layer) vs. 21504 (three layers concatenated).

CUDA graphs. A CUDA feature that allows a sequence of kernel launches to be captured and replayed as a single unit, eliminating CPU-side launch overhead. Essential for workloads with many small kernels, such as speculative decoding.

SGLang server flags. The benchmark configuration uses flags like --speculative-algorithm, --speculative-draft-model-path, --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens, --num-continuous-decode-steps, --disable-custom-all-reduce, and --disable-cuda-graph. Understanding these flags is necessary to interpret the configuration choices.

The Kimi-K2.5 model and hardware setup. The target model is Kimi-K2.5, a large language model running on 8 RTX PRO 6000 Blackwell GPUs with tensor parallelism. The NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.) reflect the specific multi-GPU communication configuration.

Output Knowledge Created

Message [msg 3640] produces several concrete pieces of knowledge:

  1. Best achievable EAGLE-3 throughput with current drafter: 82.3 tok/s average, with a peak of 101.8 tok/s on code generation tasks.
  2. The gap to the non-speculative baseline: 82.3 tok/s vs. ~90 tok/s, a deficit of approximately 9%.
  3. The effect of reducing draft tokens from 16 to 5: Throughput improved from 74.9 tok/s to 82.3 tok/s, a gain of 10%, confirming that reducing verification overhead helps but is insufficient to close the gap.
  4. The accept_len remains the fundamental bottleneck: With accept_len ~2.1, the speculation overhead cannot be fully amortized. The EAGLE-3 paper's scaling curves suggest that accept_len of 3–4 is needed for meaningful speedup, and achieving that requires more training data.
  5. Code generation tasks benefit more from speculation: Prompt 2 (code) achieved 101.8 tok/s, suggesting that the draft model's predictions are more accurate for structured outputs. This is consistent with the intuition that code has more predictable token patterns than free-form text.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible across the preceding messages, follows a clear scientific method: hypothesize, test, measure, analyze, iterate.

The chain of reasoning is:

Conclusion

Message [msg 3640] is a deceptively simple benchmark that encapsulates the outcome of a complex optimization process. It represents the best achievable performance of EAGLE-3 speculative decoding for the Kimi-K2.5 model given the current draft model, hardware configuration, and software stack. The result—82.3 tok/s, 9% below the non-speculative baseline—is both a validation and a limitation. It validates that the hidden state concatenation bug is fixed, that CUDA graphs work correctly with EAGLE-3, and that the draft model's predictions are being accepted at a meaningful rate. But it also demonstrates that the accept_len of ~2.1 is fundamentally insufficient to overcome the overhead of speculative decoding, regardless of how the draft token count or CUDA graph configuration is tuned.

The message thus serves as the empirical foundation for the project's next phase: scaling up the training data by an order of magnitude to improve the draft model's prediction quality. The 82.3 tok/s benchmark is not an endpoint but a diagnostic—it tells the assistant exactly where the bottleneck lies and what must be done to break through it.