The 15% Wall: When EAGLE-3 Speculative Decoding Fails on a 1-Trillion-Parameter Model

Introduction

In the high-stakes world of large language model inference optimization, few techniques promise as much as speculative decoding. The idea is elegant: use a small, fast "draft" model to predict multiple future tokens, then have the large target model verify them in a single forward pass. If the drafts are good, throughput can nearly double. If they are not, the overhead of running the draft model makes everything slower.

Message [msg 3083] captures the exact moment when this promise collided with reality. After days of effort—building an EAGLE-3 training pipeline from scratch, generating 10,000 synthetic training samples, running a multi-hour finetune on 8 Blackwell GPUs, and applying three separate monkey-patches to vLLM just to get the model to load—the assistant computed the acceptance rate and delivered a devastating number: 15.0%. Typical EAGLE-3 deployments achieve 60–80%. At 15%, the speculative decoder was not just failing to accelerate inference; it was actively degrading performance to 0.66× of the baseline.

This message is a masterclass in diagnostic reasoning under uncertainty. It is the turning point where weeks of work on one approach are evaluated, found deficient, and the seeds of a pivot are sown. Let us examine it in detail.

The Context: A Long Road to This Moment

To understand the weight of message [msg 3083], we must appreciate what preceded it. The assistant and user had been working with a Kimi-K2.5 model quantized to INT4—a 1-trillion-parameter monster running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The baseline throughput without speculative decoding was 82.5 tokens per second, already respectable for a model of this scale.

The EAGLE-3 training pipeline had been built from the ground up across multiple sessions (segments 20–23). This involved:

  1. Researching speculative decoding options — ruling out n-gram speculation as too slow, and identifying EAGLE-3 as the most promising approach.
  2. Building a complete training pipeline — writing scripts for hidden state extraction, data preparation, and finetuning, all adapted for the Kimi-K2.5 architecture with its Multi-head Latent Attention (MLA).
  3. Patching the speculators library — resolving API incompatibilities between speculators v0.3.0 and vLLM 0.16, including KV cache config mismatches, scheduler constructor changes, and a unique_reply_rank bug.
  4. Generating synthetic training data — capturing 10,000 real inference outputs from the Kimi-K2.5 server, which took ~5.3 hours and produced 828 GB of hidden state data.
  5. Finetuning the drafter — a 5-epoch training run from the AQ-MedAI K2 checkpoint that completed in 2.6 hours.
  6. Patching vLLM itself — three separate modifications to add SupportsEagle3 to the DeepSeek V2 model class, fix import syntax errors, and handle image token edge cases. By the time we reach message [msg 3081], the vLLM server is finally running with EAGLE-3 enabled. The assistant runs a benchmark: 5 prompts, 512 tokens each. The result is 54.6 tok/s—barely two-thirds of the baseline. The assistant hypothesizes three possible causes: poor draft model quality, TP=8 overhead, or distribution mismatch. Then, in message [msg 3082], the assistant queries the vLLM metrics endpoint and retrieves the raw acceptance data.

The Discovery: Computing the Acceptance Rate

Message [msg 3083] opens with the calculation that changes everything:

Acceptance rate: 1127 accepted / 7530 drafted = 15.0% acceptance rate. That's terrible — typical EAGLE-3 gets 60-80%. This confirms the draft model quality is very poor for this target model.

The numbers are stark. Out of 7,530 draft tokens proposed by the EAGLE-3 drafter across the benchmark run, only 1,127 were accepted by the target model's verification pass. That is a 15% hit rate. In speculative decoding, the acceptance rate is the single most important metric—it directly determines the speedup multiplier. With 5 speculative tokens per step and a 15% acceptance rate, the expected speedup is approximately:

$$E[\text{accepted tokens per step}] = \sum_{k=1}^{5} P(\text{accept} \geq k) \approx 5 \times 0.15 = 0.75$$

When you add the overhead of running the draft model's forward pass on all 8 GPUs, it becomes clear why throughput dropped to 0.66×. The drafter is consuming compute resources but producing predictions that the target model almost always rejects.

What makes this moment particularly striking is the contrast between the effort invested and the result obtained. The assistant had just successfully loaded EAGLE-3 on a model architecture (DeepSeek V2 with MLA) that required extensive patching—a technical achievement in itself. The server was producing correct outputs with reasoning, as verified in message [msg 3080]. Yet the performance was worse than doing nothing at all.

The Diagnostic Analysis: Three Hypotheses

The assistant immediately proposes three possible explanations for the low acceptance rate, each targeting a different layer of the system:

1. Hidden State Mismatch Between Training and Inference

We trained on K2.5 (INT4 quantized) but the hidden states extracted may differ from what the drafter expects during actual decode

This hypothesis targets a subtle but critical issue. During training, hidden states were extracted from the target model by running a forward pass and capturing intermediate activations. During inference with EAGLE-3, the drafter receives hidden states from the verification pass—which may have different numerical properties due to KV cache state, quantization effects, or the presence of speculative tokens in the sequence. If the distribution of hidden states during inference differs from the training distribution, the drafter's predictions will be systematically wrong.

The INT4 quantization adds another layer of complexity. The target model is quantized to 4-bit integers, which introduces non-linearities in the hidden state space. The drafter, trained on float16 hidden states from the quantized model's forward pass, may be learning patterns that don't generalize to the actual inference-time hidden state distribution.

2. Insufficient Finetuning from the Base Checkpoint

The finetune from the AQ-MedAI K2 checkpoint (trained for K2, not K2.5) didn't adapt enough with only 10K samples

The EAGLE-3 drafter was initialized from the AQ-MedAI Kimi-K2-Instruct-eagle3 checkpoint, which was trained for the original Kimi K2 model. Kimi-K2.5 is a different model—it may have different output distributions, different internal representations, or different tokenization behavior. The assistant finetuned for 5 epochs on 10,000 samples, but this may not have been enough to overcome the distribution gap between K2 and K2.5.

In speculative decoding, the drafter must predict tokens that the target model will accept. If the base checkpoint was trained on K2's distribution and the finetuning only partially adapted to K2.5, the drafter may be producing tokens that are reasonable under K2's distribution but unlikely under K2.5's. This would manifest as a low acceptance rate.

3. Vocabulary Mapping Differences

The vocab mapping is different (our t2d/d2t vs what AQ-MedAI used)

This is the most technical hypothesis. EAGLE-3 uses a "token-to-draft" (t2d) mapping that converts the target model's vocabulary to a smaller draft vocabulary, and a "draft-to-token" (d2t) mapping for the reverse direction. If the AQ-MedAI checkpoint used a different vocabulary mapping than what the assistant's pipeline produced, the drafter would be predicting tokens from the wrong distribution entirely.

The Decision: Testing the Untrained Baseline

The most telling part of message [msg 3083] is the final action: the assistant decides to test the untrained AQ-MedAI K2 drafter directly, without any finetuning. This is a critical diagnostic step. If the base drafter also achieves ~15% acceptance, it would suggest that the problem is not with the finetuning quality or the training data, but with a fundamental incompatibility between the EAGLE-3 integration and the Kimi-K2.5 model architecture—perhaps related to how vLLM extracts hidden states during MLA attention.

If the base drafter achieves a higher acceptance rate (closer to the advertised 60-80%), it would indicate that the finetuning corrupted the drafter's predictions, possibly due to overfitting on the 10K training samples or a mismatch in the hidden state extraction pipeline.

The assistant kills the current vLLM server, clears GPU memory, and prepares to launch a new server with the AQ-MedAI checkpoint path. This is methodical debugging: isolate the variable, test it, and use the result to narrow down the root cause.

Assumptions and Their Validity

Several assumptions underpin the assistant's reasoning in this message:

Assumption 1: The acceptance rate metric is accurate. The assistant queries vllm:spec_decode_num_accepted_tokens_total and vllm:spec_decode_num_draft_tokens_total from the vLLM metrics endpoint. This assumes that vLLM's instrumentation correctly counts accepted vs. drafted tokens, which is a reasonable assumption for a production-grade inference engine.

Assumption 2: 60-80% is the expected acceptance rate for EAGLE-3. This figure comes from the SGLang documentation and published benchmarks. However, those benchmarks may use different model architectures (e.g., LLaMA-style models without MLA attention) or different hardware configurations. The assistant implicitly assumes that Kimi-K2.5 with MLA should achieve similar acceptance rates, which may not be valid.

Assumption 3: The 10K sample finetune was correctly executed. The assistant assumes that the training pipeline (hidden state extraction, data preparation, finetuning) was bug-free and that the resulting drafter checkpoint is valid. If there was a silent bug in the training script—such as incorrect loss masking, wrong data shuffling, or a learning rate that caused divergence—the drafter could be producing garbage predictions regardless of the underlying architecture.

Assumption 4: The AQ-MedAI checkpoint is compatible with Kimi-K2.5. The base drafter was trained for Kimi K2, not K2.5. The assistant assumes that the architectural differences between K2 and K2.5 are small enough that the drafter's predictions would transfer. This may be incorrect—K2.5 could have significant differences in its output distribution.

The Broader Significance

Message [msg 3083] represents a classic moment in systems optimization: the point where a theoretically promising technique fails in practice, and the engineer must diagnose why. The 15% acceptance rate is not just a number—it is a signal that propagates through the entire system stack.

For the vLLM integration, it suggests that the EAGLE-3 support for DeepSeek V2 / MLA architectures may have fundamental issues. The three patches required to load the model (model whitelist, image token handling, SupportsEagle3 interface) were heroic efforts, but they may have addressed surface-level compatibility while missing deeper issues in how hidden states are extracted and passed to the drafter during MLA attention.

For the training pipeline, it raises questions about the quality of the synthetic data and the finetuning process. The 828 GB of hidden state data came from a single extraction run—if there was a bug in the extraction script (e.g., incorrect layer targeting, wrong tensor shapes), the entire training dataset would be corrupted.

For the project direction, this message is the pivot point. The user had already suggested SGLang as an alternative ([msg 3063]), and the assistant's research in [msg 3074] confirmed that SGLang has first-class EAGLE-3 support with explicit testing on Kimi-K2 drafters. The 15% acceptance rate in vLLM makes the SGLang pivot not just an option but a necessity.

What Follows

The next messages in the conversation show the assistant testing the AQ-MedAI baseline drafter ([msg 3084]), which achieves a similarly low acceptance rate. This confirms that the problem is not with the finetuning but with the vLLM EAGLE-3 integration for MLA models. The assistant then pivots to SGLang, building the sgl-kernel for SM120 (a 48-minute compilation) and discovering that SGLang loads the model in 22 seconds (vs. 25 minutes in vLLM) but then deadlocks on Blackwell GPUs—a new challenge that requires further debugging.

Conclusion

Message [msg 3083] is a document of intellectual honesty in engineering. It reports a clear failure—15% acceptance rate, 0.66× throughput—without defensiveness or wishful thinking. It proposes three concrete hypotheses, each testable through further experimentation. And it takes immediate action to isolate the root cause by testing the untrained baseline.

In the broader narrative of this coding session, this message marks the end of the vLLM EAGLE-3 experiment and the beginning of the SGLang exploration. It is a reminder that in machine learning systems, the gap between "it loads" and "it works well" can be vast—and that measuring the right metrics is the only way to know which side of that gap you are on.