The 0.66x Reality Check: When EAGLE-3 Speculative Decoding Makes Things Worse

In the high-stakes world of large language model inference, few moments are as simultaneously triumphant and crushing as the one captured in message 3081 of this opencode session. After days of relentless engineering—building a synthetic data generation pipeline, training an EAGLE-3 drafter from scratch on 10,000 samples of Kimi-K2.5's own reasoning traces, patching vLLM's source code in three separate places to support an architecture it wasn't designed for, and waiting through multiple 25-minute model loading cycles—the assistant finally achieves the goal: vLLM with EAGLE-3 speculative decoding is live, producing correct output with reasoning. The celebration lasts approximately the time it takes to run a five-prompt benchmark. The result: 54.6 tokens per second, compared to a baseline of 82.5 tokens per second without any speculation. A speedup of 0.66x. The speculative drafter is not accelerating the model; it is actively slowing it down.

The Moment of Truth

Message 3081 opens with a deceptively simple declaration: "It works. EAGLE-3 speculative decoding is live and producing correct output with reasoning. Now let me benchmark throughput." The assistant has just confirmed the server is responsive ([msg 3079]), verified that the model returns coherent responses with proper reasoning chains ([msg 3080]), and now moves immediately to the critical question: does it actually help?

This question is the entire point of the exercise. Speculative decoding is not free—it requires running a smaller "draft" model to generate candidate tokens, then having the base model verify them in parallel. The overhead of the draft model, the coordination between draft and base, and the complexity of managing speculative tokens all eat into the theoretical speedup. If the acceptance rate (the fraction of draft tokens the base model accepts) is too low, the overhead dominates and speculation becomes a net loss. The assistant knows this, which is why the benchmark script is the very first thing executed after confirming the server is alive.

The Benchmark Design

The assistant constructs a careful benchmark: five diverse prompts covering physics ("Explain the theory of general relativity in detail"), programming ("Write a Python function to solve the N-queens problem using backtracking"), networking ("What are the main differences between TCP and UDP protocols?"), biology ("Describe the process of photosynthesis step by step"), and mathematics ("Write a proof that there are infinitely many prime numbers"). Each prompt requests 512 completion tokens at temperature 0.6, a standard setting for creative generation. The script measures end-to-end latency per prompt, computes tokens-per-second and milliseconds-per-token, then aggregates across all five prompts for a final average.

The baseline of 82.5 tok/s (12.1 ms/tok) is a known quantity from earlier in the session—established through prior benchmarking of the Kimi-K2.5 INT4 model on the 8x RTX PRO 6000 Blackwell GPU setup without speculative decoding. This gives the assistant a clean apples-to-apples comparison: same model, same hardware, same prompt structure, same token budget. The only variable is whether EAGLE-3 speculation is enabled.

The Results: A Clear Negative Signal

The individual prompt results tell a story of variability with a consistent downward trend:

• Prompt 1 (general relativity): 512 tokens in 10.4s = 49.0 tok/s
• Prompt 2 (N-queens): 512 tokens in 8.9s = 57.5 tok/s
• Prompt 3 (TCP vs UDP): 512 tokens in 9.3s = 55.0 tok/s
• Prompt 4 (photosynthesis): 512 tokens in 10.4s = 49.3 tok/s
• Prompt 5 (prime numbers): 512 tokens in 7.9s = 65.1 tok/s The range spans from 49.0 to 65.1 tok/s, with the math/programming prompts (2 and 5) performing better than the descriptive prompts (1 and 4). This variation likely reflects differences in how predictable each domain is—the drafter may be better at anticipating code and mathematical reasoning than free-form explanation. But even the best case (65.1 tok/s) falls well short of the 82.5 tok/s baseline. The aggregate: 2,560 tokens in 46.9 seconds, averaging 54.6 tok/s at 18.3 ms/tok. The speedup factor of 0.66x means the model is running at two-thirds of its non-speculative speed. The speculative drafter is a net negative.

What This Reveals About the vLLM Integration

This result is not merely disappointing; it is diagnostic. The assistant and the user had previously discussed that the AQ-MedAI baseline drafter (the pre-trained checkpoint from which the fine-tuning began) was reported to achieve approximately 1.8x throughput with SGLang. The fact that the same model architecture achieves 0.66x with vLLM points to a fundamental integration problem, not a training quality issue.

The chunk summary for this segment confirms this interpretation: "Both the newly trained drafter and the pre-trained AQ-MedAI baseline achieved only ~15% acceptance rate, resulting in 0.66x throughput—worse than no speculation. This confirmed a fundamental vLLM integration issue with MLA attention hidden state extraction during decode, not a training quality problem."

The key technical detail here is MLA attention (Multi-head Latent Attention), the attention mechanism used by DeepSeek V3 and Kimi-K2.5. EAGLE-3 works by extracting hidden states from the base model's transformer layers during the decoding process and feeding them into a lightweight drafter head that predicts the next several tokens. If vLLM's implementation of MLA attention does not correctly expose the hidden states in the format EAGLE-3 expects, the drafter receives degraded or misaligned inputs, leading to low-quality draft tokens that the base model frequently rejects. A 15% acceptance rate means that out of every 5 speculative tokens generated, only about 0.75 are accepted—far below the threshold needed to overcome the overhead of running the drafter at all.

The Reasoning Process Visible in the Message

The assistant's thinking is methodical and data-driven. The message contains no hand-waving or speculation about whether the speedup might be acceptable in some scenarios. The benchmark is designed to produce a clear quantitative answer, and the assistant presents that answer without spin. The final line of the output—"Speedup: 0.66x"—is allowed to speak for itself.

However, the assistant also demonstrates an awareness of the broader context. Earlier in the session ([msg 3077]), the assistant had researched SGLang's EAGLE-3 support and noted that "SGLang has first-class EAGLE-3 support—it's their recommended spec decode method" and that "AQ-MedAI/Kimi-K2-Instruct-eagle3 is explicitly tested with SGLang and gets ~1.8x throughput." The user had explicitly suggested considering SGLang ([msg 3063]). So when this benchmark returns 0.66x, the assistant already knows where to look next: the problem is likely in vLLM's integration, not in the drafter itself.

Input Knowledge Required to Understand This Message

To fully grasp what is happening in message 3081, one needs several pieces of context:

1. The baseline throughput: 82.5 tok/s for Kimi-K2.5 INT4 on 8x Blackwell GPUs, established through prior benchmarking in the session.
2. The EAGLE-3 training pipeline: The assistant had just completed a multi-stage process involving synthetic data generation (10K inference runs taking ~5.3 hours), hidden state extraction at 3,165 tok/s producing 828 GB of training data, and a 5-epoch fine-tuning from the AQ-MedAI checkpoint taking 2.6 hours. The drafter being tested is the output of this pipeline.
3. The vLLM patching effort: Three separate source-code patches were required to make vLLM's EAGLE-3 implementation work with DeepSeek V3/Kimi-K2.5: adding the model to a whitelist, handling image token placeholders, and implementing the SupportsEagle3 interface on both the DeepseekV3 and KimiK25ForConditionalGeneration classes. One of these patches introduced a syntax error that had to be fixed (<msg id=3068-3072>).
4. The speculative decoding theory: Understanding that EAGLE-3 uses a lightweight drafter head to predict multiple candidate tokens, which the base model then verifies in parallel. The acceptance rate (fraction of draft tokens accepted) determines whether speculation helps or hurts.
5. The SGLang alternative: The user had suggested SGLang, and the assistant's research confirmed it has first-class EAGLE-3 support with documented speedups on similar model architectures.

Output Knowledge Created by This Message

This message produces several concrete pieces of knowledge:

• Quantified EAGLE-3 performance on vLLM with Kimi-K2.5: 54.6 tok/s average, 0.66x of baseline.
• Confirmation of a vLLM-specific integration problem: The low acceptance rate (~15%) is inconsistent with SGLang's reported ~1.8x speedup for similar models, pointing to an implementation issue rather than a fundamental limitation of the approach.
• A decision point: The assistant now has empirical justification to pivot away from vLLM for EAGLE-3 inference and toward SGLang, as the user had suggested.
• A baseline for future comparisons: Any future speculative decoding attempt on this hardware and model can be compared against the 54.6 tok/s figure.

The Broader Significance

Message 3081 is a textbook example of why rigorous benchmarking matters in machine learning engineering. It would have been easy to declare victory when the server started responding correctly—the model loaded, the patches worked, the output was coherent. But the entire purpose of building an EAGLE-3 drafter was to improve throughput, not just to demonstrate that it could run. The benchmark revealed that the system was not just failing to meet expectations but was actively degrading performance.

This moment also illustrates the asymmetry of effort in speculative decoding: building the training pipeline and getting the model to load took days of work across multiple sub-sessions, while the benchmark that invalidated the entire approach took under a minute to run. The assistant's willingness to run that benchmark immediately, without rationalizing away the likely outcome, is a mark of disciplined engineering practice.

The 0.66x result directly led to the pivot to SGLang documented in the remainder of segment 23. The assistant built sgl-kernel for SM120 (a 48-minute compilation), verified it worked, and began debugging a new set of issues—this time, SGLang server deadlocks on Blackwell GPUs. The journey to accelerate Kimi-K2.5 inference was far from over, but message 3081 provided the critical data point that redirected the entire effort.