The Verdict on EAGLE-3: When Speculative Decoding Meets Quantized Reality

Introduction

In the high-stakes world of large language model serving, every millisecond counts. When you're running a 547GB INT4-quantized model like Kimi-K2.5 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a well-tuned serving stack and a mediocre one can mean thousands of tokens per second in throughput—or a frustrating spiral of debugging dead ends. Message 3221 represents a pivotal moment in an extended optimization campaign: the moment when the assistant stops experimenting, synthesizes everything learned, and delivers a clear-eyed verdict on what works, what doesn't, and why.

This message is not a tool call, not a bash command, and not a debugging session. It is a summary—a reflective synthesis of hours of benchmarking, patching, and analysis. It is the assistant stepping back from the tactical trenches to deliver strategic clarity. And what it reveals is both illuminating and sobering: speculative decoding with EAGLE-3, despite being the most sophisticated software-only optimization available, delivers no benefit on this hardware configuration, while the simpler approach of running SGLang without speculation achieves a remarkable 2,370 tokens per second peak throughput—54% better than the already-impressive vLLM baseline.

Why This Message Was Written

The motivation behind message 3221 is rooted in the arc of the preceding conversation. The assistant had spent several rounds (messages 3198–3220) engaged in a multi-pronged investigation. First, it had resolved an apparent SGLang server "hang" on SM120 Blackwell GPUs, discovering that the server was actually running fine—it just took 5–10 minutes to load the massive 547GB model. Then it had patched the kimi_k25.py model file to add three critical delegation methods required for EAGLE-3 speculative decoding. Finally, it had launched and benchmarked four distinct configurations:

1. vLLM baseline (from prior work)
2. SGLang base with CUDA graphs enabled
3. SGLang + AQ-MedAI EAGLE-3 drafter (a pre-trained drafter from a third party)
4. SGLang + Custom K2.5-trained EAGLE-3 drafter (trained earlier in the session on 10K samples) Each configuration required launching a server, waiting for it to load, running benchmarks, analyzing logs, and killing the process. This was a time-intensive, iterative process. By message 3221, the assistant had accumulated enough data to draw definitive conclusions. The summary message serves to: - Document what was learned before moving on - Provide a clear comparison across all tested configurations - Explain why EAGLE-3 failed in this context - Propose actionable next steps for the user to decide - Restore the best-performing configuration (SGLang base) as the running server It is, in essence, a status report and a decision document rolled into one. The assistant is saying: "Here's what we know now. Here's what works. Here's what doesn't. What do you want to do next?"

The Key Revelation: SGLang Was Never Hung

One of the most striking admissions in the summary is the correction of a prior assumption. In earlier messages, the assistant had been diagnosing what appeared to be a server hang after weight loading. The log output showed the server starting, then seemingly stalling. The assistant had even killed and relaunched the server multiple times. But the truth, as revealed in message 3221, was mundane: "The previous 'hang' was simply the 547GB model taking ~5-10 minutes to load."

This is a classic debugging pitfall. When a system takes an unexpectedly long time to initialize, it's easy to assume something is broken—especially when you're working with cutting-edge hardware (Blackwell GPUs) and a nightly build of SGLang. The assistant's willingness to acknowledge this mistake explicitly is important. It demonstrates intellectual honesty and prevents future investigators from chasing the same phantom.

The broader lesson is that model loading time scales linearly with model size and inversely with I/O bandwidth. A 547GB model loaded over what appears to be a shared filesystem (/shared/kimi-k2.5-int4) will naturally take minutes, not seconds. The assistant's subsequent benchmarks confirmed that once loaded, SGLang "works perfectly on SM120 Blackwell with CUDA graphs."

The Patches: Enabling EAGLE-3 on Kimi-K2.5

The message documents three delegation methods added to kimi_k25.py:

• set_eagle3_layers_to_capture() — for EAGLE-3 hidden state capture
• get_embed_and_head() — for sharing embed/lm_head with draft model
• set_embed_and_head() — counterpart setter These methods are required by SGLang's EAGLE-3 worker (eagle_worker.py) to extract the embedding and language model head weights from the target model and share them with the draft model. Without them, the speculative decoding engine cannot function. The patch was non-trivial because the Kimi-K2.5 model has a nested architecture (a KimiK25Model wrapping a DeepseekV2ForCausalLM-style language model), so each method needed to delegate through self.language_model to reach the underlying implementation. The assistant also notes the need for SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 due to the draft model having max_position_embeddings=131072 versus the target's 262144. This environment variable tells SGLang to override the context length mismatch, which would otherwise cause an error during initialization.

The Performance Landscape: A Tale of Two Metrics

The centerpiece of message 3221 is the performance comparison table. It reveals a fascinating split between two metrics:

| Config | Single-stream | C=128 throughput | Accept rate | |--------|:------------:|:----------------:|:-----------:| | vLLM baseline | 82.5 tok/s | 1,536 tok/s | — | | SGLang base (CUDA graphs) | 63.6 tok/s | 2,370 tok/s | — | | SGLang + AQ-MedAI EAGLE-3 | 62.9 tok/s | 849 tok/s | ~42% | | SGLang + Custom K2.5 EAGLE-3 | 40.8 tok/s | 597 tok/s | 25% (broken) |

Single-stream vs. throughput is a classic tension in LLM serving. Single-stream performance (one request at a time, measuring latency) is dominated by how fast the GPU can process a single sequence. Throughput at high concurrency (C=128 requests in flight) is dominated by how efficiently the system can batch multiple requests together, maximizing GPU utilization.

vLLM wins on single-stream (82.5 vs 63.6 tok/s), a 30% advantage. This likely reflects vLLM's more mature CUDA graph capture and scheduling for single-sequence decode. But SGLang dominates at high concurrency (2,370 vs 1,536 tok/s), a 54% advantage. This suggests SGLang's batching and memory management are more efficient when many requests compete for GPU resources.

The EAGLE-3 configurations, however, are disastrous at high concurrency. The AQ-MedAI drafter achieves only 849 tok/s (64% worse than SGLang base), and the custom drafter manages just 597 tok/s (75% worse). Even single-stream, the custom drafter is abysmal at 40.8 tok/s.

Why EAGLE-3 Failed: The Acceptance Rate Problem

The assistant identifies the core issue: acceptance rate. EAGLE-3 speculative decoding works by having a small "draft" model propose multiple future tokens in parallel, then the large "target" model verifies them in a single forward pass. The speedup depends on the acceptance rate—the fraction of draft tokens that the target model accepts.

The AQ-MedAI drafter achieves a ~42% acceptance rate, meaning about 1.65 out of 4 proposed tokens are accepted on average. This is well below the 60–80% typically needed for speculative decoding to break even, considering the overhead of running the draft model and the extra verification step.

The custom K2.5-trained drafter is even worse: a 25% acceptance rate with an accept length of exactly 1.00, meaning zero draft tokens are ever accepted. The assistant bluntly calls this "broken" and "essentially random."

The root cause analysis is sophisticated:

1. INT4 quantization mismatch: The training data for the custom drafter was extracted from the INT4 quantized model using a custom vLLM worker. But SGLang's internal hidden state extraction mechanism may produce different hidden states than vLLM's. Even small differences in quantization numerics can shift the hidden state distribution enough that the drafter's predictions become useless.
2. SGLang's layer offset convention: The assistant suspects SGLang may use a +1 layer offset convention that doesn't match the training data, causing the drafter to receive misaligned hidden states.
3. The AQ-MedAI drafter was trained for Kimi-K2, not K2.5: The pre-trained drafter from AQ-MedAI was designed for the earlier Kimi-K2 model. While Kimi-K2.5 is an evolution, architectural differences likely explain why the acceptance rate is only 42% instead of the expected 70%+.
4. SGLang's max_running_requests=48 limit: When speculative decoding is enabled, SGLang automatically caps the number of concurrent requests at 48 (vs 2048 for base mode). This severely limits throughput at high concurrency, making the speculative overhead even more punishing.

Assumptions and Mistakes

Several assumptions embedded in this work proved incorrect:

• That EAGLE-3 would provide a meaningful speedup: The entire multi-hour effort to train a custom drafter, patch SGLang, and test multiple configurations was predicated on the assumption that speculative decoding would improve throughput. It did not—it made things worse.
• That training on INT4-extracted hidden states would work: The assistant assumed that hidden states extracted from the quantized model during training would align with those produced during inference. This turned out to be false, likely due to subtle differences in the extraction pipeline.
• That the "hang" was a bug: The initial assumption that SGLang was hanging on SM120 was incorrect. It was simply slow to load. This wasted debugging time but ultimately led to a working server.
• That a third-party drafter (AQ-MedAI) would be compatible: While it loaded and ran without errors, its acceptance rate was too low to provide benefit. The drafter was designed for a different model version.

Input Knowledge Required

To fully understand this message, one needs:

• Understanding of speculative decoding: How draft models propose tokens and target models verify them, and the concept of acceptance rate.
• Knowledge of LLM serving frameworks: The difference between vLLM and SGLang, what CUDA graphs are, and how tensor parallelism works across 8 GPUs.
• Familiarity with the Kimi-K2.5 model: Its architecture (DeepseekV2-derived with MLA attention), its quantization to INT4, and its massive 547GB size.
• Understanding of the EAGLE-3 algorithm: How it uses hidden states from the target model to condition the draft model's predictions, and why layer alignment matters.
• Awareness of the hardware context: Blackwell SM120 GPUs, their memory capacity (each RTX PRO 6000 has 96GB), and the implications of PCIe-based multi-GPU communication.

Output Knowledge Created

This message creates valuable knowledge for the project:

1. Definitive performance baselines: The project now has clear numbers for vLLM baseline, SGLang base, and two EAGLE-3 configurations. These can be referenced in future optimization work.
2. Evidence that EAGLE-3 is not viable on this stack: The combination of INT4 quantization, SGLang's serving architecture, and the Kimi-K2.5 model produces acceptance rates too low for speculative decoding to help.
3. Documentation of required patches: The three delegation methods added to kimi_k25.py are documented, along with the SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN workaround. This saves future developers from rediscovering these requirements.
4. A prioritized list of next steps: The assistant proposes five possible directions, from accepting the current solution to debugging hidden state alignment to trying n-gram speculation. This gives the user a menu of options with clear trade-offs.
5. A corrected understanding of SGLang on SM120: The message definitively establishes that SGLang works on Blackwell GPUs, resolving the earlier uncertainty.

The Thinking Process

The reasoning visible in this message reveals a structured analytical mind at work. The assistant:

1. Synthesizes disparate data points into a coherent comparison table, normalizing metrics across configurations that were tested at different times under different conditions.
2. Identifies the root cause hierarchy: The immediate cause of poor EAGLE-3 performance is low acceptance rate, but the deeper causes are quantization mismatch, drafter-model misalignment, and SGLang's concurrency limits.
3. Separates signal from noise: The assistant correctly identifies that the custom drafter's 25% acceptance rate with accept len 1.00 means it's accepting zero draft tokens—a clear sign of a broken system, not merely suboptimal performance.
4. Provides actionable context: Rather than just presenting numbers, the assistant explains why each number matters and what it implies for the project's direction.
5. Leaves the decision to the user: The summary ends with "Possible Next Steps (needs discussion)," acknowledging that the assistant's role is to provide analysis, not to unilaterally decide the project's direction.

Conclusion

Message 3221 is a masterclass in experimental synthesis. After hours of debugging, patching, benchmarking, and iterating, the assistant distills a complex multi-dimensional investigation into a clear, honest, and actionable summary. The central finding—that EAGLE-3 speculative decoding provides no benefit on this hardware and model combination—is delivered without equivocation, supported by concrete data and reasoned analysis.

The message also demonstrates intellectual humility: the assistant admits the earlier "hang" was a false alarm, acknowledges that the custom drafter is "completely broken," and frames the results as input to a human decision rather than as a final verdict. This combination of technical rigor and collaborative framing makes message 3221 a model of how an AI assistant should communicate experimental results in a complex engineering context.

For the project going forward, the path is clear: SGLang base with CUDA graphs is the serving solution, delivering 2,370 tok/s peak throughput. The EAGLE-3 experiment, while ultimately unsuccessful, generated valuable knowledge about the interaction between INT4 quantization, speculative decoding, and SGLang's architecture—knowledge that will inform future optimization efforts, whether they involve n-gram speculation, hidden state alignment debugging, or simply accepting the excellent baseline performance already achieved.