The EAGLE-3 Verdict: A Todo List That Marked the End of a Speculative Decoding Investigation

Introduction

In the sprawling, multi-day effort to deploy and optimize the Kimi-K2.5 large language model on an 8-GPU Blackwell system, few moments were as decisive as message 3220. On its surface, this message is mundane — a simple todowrite tool call updating a JSON-structured todo list, marking several tasks as completed. But this message is anything but mundane. It represents the formal conclusion of a deep investigation into EAGLE-3 speculative decoding, an investigation that consumed dozens of messages, spanned multiple model architectures, and ultimately produced a counterintuitive result: that speculative decoding — one of the most promising techniques for accelerating LLM inference — provided no benefit whatsoever on this particular hardware and model combination.

The Message Itself

The subject message reads:

[assistant] [todowrite] {"todos":[{"content":"Diagnose SGLang server hang after weight loading","priority":"high","status":"completed"},{"content":"Benchmark SGLang base serving performance","priority":"high","status":"completed"},{"content":"Test SGLang + AQ-MedAI EAGLE-3 drafter (accept ~42%, no speedup)","priority":"high","status":"completed"},{"content":"Test SGLang + custom K2.5 EAGLE-3 drafter (accept 25% = broken)","priority":"high","status":"completed"}]}

Four tasks, all marked completed. The first two — diagnosing the SGLang server hang and benchmarking base performance — were preliminary groundwork. The latter two represent the core of the investigation: testing two different EAGLE-3 draft models against the Kimi-K2.5 INT4 quantized model running on SGLang with 8-way tensor parallelism across Blackwell GPUs.

Why This Message Was Written: The Context and Motivation

To understand why this todo update matters, one must understand the journey that led to it. The assistant had been working for days to deploy Kimi-K2.5, a massive 547GB model quantized to INT4. After resolving a critical SGLang hang issue — which turned out not to be a hang at all, but rather a 5–10 minute model loading time that was mistaken for a deadlock — the assistant achieved impressive baseline performance: 63.6 tokens/second in single-stream mode and a peak throughput of 2,370 tok/s at concurrency 128, significantly outperforming vLLM's peak of 1,536 tok/s.

But the assistant's mandate was not merely to serve the model; it was to optimize it. Speculative decoding, particularly the EAGLE-3 algorithm, promised to accelerate inference by using a lightweight "draft" model to propose multiple tokens per forward pass, which the target model then verifies in parallel. If the draft model is accurate enough — typically requiring a 60–80% acceptance rate — speculative decoding can provide substantial speedups.

The assistant had invested heavily in this approach. It had built a complete EAGLE-3 training pipeline from scratch, patched API incompatibilities between the speculators library and vLLM 0.16, extracted hidden states from the Kimi-K2.5 model, and trained a custom drafter on 10,000 samples of the model's own outputs. It had also identified an existing drafter from AQ-MedAI, originally trained for the earlier Kimi-K2 model, as a potential candidate. Message 3220 is the moment when all that investment was evaluated and, ultimately, found to be misdirected.

The Investigation: Two Drafters, Two Failures

The assistant tested two EAGLE-3 draft models, each representing a different approach:

The AQ-MedAI Drafter (Kimi-K2): This drafter was pre-trained for the earlier Kimi-K2 model, not the newer K2.5. The assistant launched it with SGLang's speculative decoding flags (--speculative-algorithm EAGLE3, --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4). The results were underwhelming: an acceptance rate of approximately 42%, yielding an average acceptance length of 1.65 tokens out of 4 proposed. Single-stream throughput was 62.9 tok/s (essentially identical to the 63.6 tok/s baseline), and peak throughput at concurrency 128 collapsed to 849 tok/s — a 64% regression from the 2,370 tok/s baseline. The culprit was SGLang's automatic max_running_requests=48 limit for speculative mode, which throttled concurrency and negated any potential gains.

The Custom K2.5-Trained Drafter: This was the assistant's own creation, trained on 10,000 samples of Kimi-K2.5's actual outputs using a meticulously constructed pipeline. The results were catastrophic: an acceptance rate of 25% and an acceptance length of exactly 1.00 — meaning the drafter accepted zero draft tokens per step. A 25% acceptance rate with 4 draft tokens is statistically indistinguishable from random guessing (1/4 = 25%). Single-stream throughput dropped to 40.8 tok/s, and peak throughput fell to 597 tok/s. The drafter was, in the assistant's own words, "completely broken."

Root Cause Analysis: The Hidden State Alignment Problem

The assistant's thinking process, visible in the surrounding messages, reveals a sophisticated diagnosis of the failure. The root cause was identified as a hidden state alignment mismatch: the EAGLE-3 drafter was trained on FP16/BF16 hidden states extracted from the model during training, but the serving model was quantized to INT4. Quantization alters the hidden state distributions, and the drafter — which conditions its predictions on these hidden states — could not generalize from the FP16 training distribution to the INT4 inference distribution.

This is a subtle but critical insight. EAGLE-3 works by conditioning the draft model on the target model's hidden states from specific layers. If those hidden states shift due to quantization, the draft model's predictions become no better than random. The AQ-MedAI drafter performed slightly better (42% vs 25%) likely because it was trained on a different model entirely (Kimi-K2) and its predictions were less tightly coupled to the specific hidden state distribution — or perhaps because the K2 and K2.5 models share enough representational structure that some transfer occurred.

Assumptions Made and Lessons Learned

Several assumptions underpinned this investigation, and several were proven wrong:

  1. Assumption: EAGLE-3 would provide meaningful speedup on Blackwell GPUs. This was the central hypothesis, and it was thoroughly falsified. The combination of low acceptance rates, SGLang's concurrency limits for speculative mode, and the overhead of running draft model forward passes meant that speculative decoding was strictly worse than baseline.
  2. Assumption: A drafter trained on K2.5 data would outperform one trained on K2. This seemed intuitive — training on the exact model being served should produce better alignment. In practice, the opposite occurred. The K2.5-trained drafter was worse, likely because it overfit to the FP16 hidden state distribution that diverged from the INT4 serving distribution.
  3. Assumption: The hidden state extraction pipeline produced compatible data. The assistant had built a sophisticated pipeline using SGLang's EAGLE-3 integration to capture hidden states during inference. But the training pipeline used FP16/BF16 representations, while the serving pipeline used INT4. This mismatch was not anticipated and proved fatal.
  4. Assumption: SGLang's speculative decoding implementation was mature. While SGLang's base serving performance was excellent, its speculative decoding mode imposed a severe max_running_requests=48 limit that crippled throughput at high concurrency. This may be a safety measure or a fundamental limitation of the current implementation.

Input and Output Knowledge

To fully understand this message, one needs knowledge of: speculative decoding algorithms (particularly EAGLE-3's approach of conditioning on target model hidden states); the SGLang serving framework and its configuration flags for speculative decoding; the distinction between INT4 quantization and FP16/BF16 training representations; the architecture of Kimi-K2.5 and its DeepSeekV2-derived MLA attention mechanism; and the performance characteristics of Blackwell GPUs with tensor parallelism.

The message creates new knowledge: a definitive empirical result that EAGLE-3 speculative decoding provides no benefit for Kimi-K2.5 on 8x Blackwell GPUs under SGLang. This is a non-obvious finding — speculative decoding is widely reported to provide 1.5x–3x speedups in the literature — and it carries important implications for practitioners deploying quantized models with speculative decoding.

The Thinking Process: A Systematic Evaluation

The assistant's reasoning, visible across the messages leading to this todo update, demonstrates a methodical approach. It established baselines first (base SGLang performance), then tested each drafter independently, collected acceptance rate statistics from the server logs, benchmarked throughput at multiple concurrency levels, and compared results against both the baseline and the vLLM alternative. When the custom drafter failed, the assistant did not simply give up — it checked the acceptance logs, computed the statistical significance (25% = 1/4 = random), and formulated a root cause hypothesis about hidden state alignment. The todo update in message 3220 is the final step in this process: formal acknowledgment that the investigation is complete and the results are conclusive.

Conclusion

Message 3220 is a todo list update, but it is also a tombstone — the marker of a well-executed investigation that produced a negative result. In scientific and engineering work, negative results are as valuable as positive ones. The assistant learned that EAGLE-3 speculative decoding, despite its theoretical promise, does not work on this hardware and model combination due to hidden state alignment issues between INT4 quantization and FP16-trained draft models, compounded by SGLang's concurrency limitations in speculative mode. The path forward, as the subsequent messages reveal, was to abandon speculative decoding and instead focus on tuning SGLang's single-stream performance using NCCL environment variables — a pragmatic pivot that ultimately proved more fruitful.