The Moment of Reckoning: When Two EAGLE-3 Drafters Both Failed to Beat the Baseline

Introduction

In the course of a complex machine learning deployment session, a single brief message can crystallize hours of debugging effort and redirect the trajectory of an entire project. Message [msg 3629] is exactly such a moment. It consists of a simple bash command — grepping server logs for acceptance metrics — followed by three lines of output showing the AQ-MedAI EAGLE-3 drafter's performance. On its surface, it appears mundane: a routine check of speculative decoding statistics. But in context, this message represents the final verdict on a weeks-long effort to get EAGLE-3 speculative decoding working with the Kimi-K2.5 model, and it forces a fundamental strategic pivot.

The Full Message

The assistant executed the following command on a remote server:

ssh root@10.1.230.174 'grep "accept len" /data/eagle3/sglang_eagle3_aqmedai.log | tail -10'

Which produced the output:

[2026-02-23 17:41:10 TP0] Decode batch, #running-req: 1, #token: 362, token usage: 0.00, accept len: 2.02, accept rate: 0.13, cuda graph: False, gen throughput (token/s): 50.31, #queue-req: 0
[2026-02-23 17:41:11 TP0] Decode batch, #running-req: 1, #token: 440, token usage: 0.00, accept len: 1.95, accept rate: 0.12, cuda graph: False, gen throughput (token/s): 48.45, #queue-req: 0
[2026-02-23 17:41:13 TP0] Decode batch, #running-req: 1, #token: 516, token usage: 0.00, accept len: 1.90, accept ra...

The Context: A Long Debugging Journey

To understand why this message matters, one must appreciate the debugging saga that preceded it. The team had been working for days to deploy EAGLE-3 speculative decoding — a technique where a lightweight "draft" model generates candidate tokens that a full-sized target model then verifies in parallel, ideally achieving 2-4× speedup over standard autoregressive generation.

The project had hit a devastating bug. Despite training a custom EAGLE-3 drafter on 10,000 samples extracted from the Kimi-K2.5 model, the acceptance rate was effectively zero. Every token the draft model proposed was rejected, meaning the system was operating at a 1.0× accept length — no speculation benefit whatsoever. The draft model's carefully trained weights were completely useless.

After extensive investigation, the root cause was traced to a single flag mismatch. The SGLang server had been launched with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. This seemingly minor difference had catastrophic consequences. The is_eagle3() check in the SGLang codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate hidden states from layers 2, 30, and 58 into a 21,504-dimensional tensor. With the EAGLE flag, the draft model received only the final layer's 7,168-dimensional hidden states. The draft model's fc fusion layer, which expects 21,504-dimensional input, was silently bypassed, and all trained weights were rendered inert.

When the server was restarted with the correct EAGLE3 flag, the debug logs immediately confirmed the fix: hidden states arrived as torch.Size([21, 21504]) — the correct 3-layer concatenation. The custom drafter's acceptance length jumped from 1.0 to approximately 2.1-2.5, and the acceptance rate reached 12-16%. This was a genuine victory — the bug was fixed, the draft model was working, and tokens were being accepted.

The AQ-MedAI Drafter: A Second Opinion

With the custom drafter showing promising but insufficient results, the assistant turned to an alternative: the AQ-MedAI EAGLE-3 drafter. This was a pre-trained draft model from the AQ-MedAI project, originally trained for the Kimi-K2 architecture (not the newer K2.5). The hypothesis was that a drafter trained on more data — even if not specifically aligned with K2.5's token distribution — might achieve higher acceptance rates.

Message [msg 3629] is the moment this hypothesis is tested. The assistant starts the SGLang server with the AQ-MedAI drafter checkpoint, runs the same benchmark suite of five prompts, and then checks the server logs for acceptance metrics. The results are unambiguous.

What the Numbers Reveal

The three log lines in the output tell a consistent story. The AQ-MedAI drafter achieves an accept len of approximately 1.90-2.02 — essentially identical to the custom drafter's ~2.0-2.5. The accept rate hovers around 0.12-0.13, meaning only 12-13% of the 16 draft tokens are accepted per verification step. The throughput is 48-50 tokens per second, slightly worse than the custom drafter's 53-57 tok/s.

These numbers are devastating for the speculative decoding project. Here's why:

  1. Accept length of ~2.0 is insufficient. The EAGLE-3 paper and other speculative decoding literature show that meaningful speedup requires an accept length of at least 3-4 tokens. With only 2 tokens accepted on average, the overhead of running the draft model (forward pass through the EAGLE-3 head) and performing the verification pass (running the full target model on all candidates) exceeds the savings from generating multiple tokens per step.
  2. The non-speculative baseline is 90 tok/s. The team had previously benchmarked the base SGLang server without any speculation at approximately 90 tokens per second single-stream. Both drafters — custom and AQ-MedAI — produce throughput in the 48-57 tok/s range, which is 36-46% slower than the baseline. Speculative decoding is actively hurting performance.
  3. CUDA graphs are disabled. The server was started with --disable-cuda-graph, which means the GPU kernel launch overhead is much higher than it could be. With CUDA graphs enabled, the overhead of the draft model's forward pass would decrease, potentially improving the breakeven point. However, the fundamental problem remains: the draft model's predictions simply aren't good enough.
  4. The AQ-MedAI drafter is no better than the custom drafter. Despite presumably being trained on more data (AQ-MedAI is a well-known project), the general K2 drafter achieves nearly identical acceptance metrics to the custom K2.5-trained drafter. This is both disappointing and informative — it suggests that the limitation is not specific to the training approach but rather a fundamental property of the EAGLE-3 architecture when applied to this model family.

The Implicit Decision: A Strategic Pivot

This message does not contain an explicit decision statement. There is no "therefore we will do X" in the bash command or its output. But the decision is written in the silence between the lines. The assistant does not attempt further hyperparameter tuning. It does not try different numbers of draft tokens, different top-k values, or different numbers of speculative steps. It does not restart the server with CUDA graphs enabled to see if that changes the equation. The acceptance metrics are simply accepted as the final verdict.

The reasoning is visible in the preceding messages. In [msg 3615], the assistant analyzed the custom drafter's performance and concluded: "The issue is: 1. Accept rate ~15% is too low for speedup — speculation overhead costs more than the ~2.3× accept length saves. 2. CUDA graphs are disabled — huge overhead penalty. 3. Only 10K training samples — the EAGLE-3 paper shows clear scaling with more data." By [msg 3624], the assistant had further refined this analysis: "The fundamental problem: accept_len of ~2.1 is too low. With an ideal EAGLE-3 drafter, accept_len should be 3-4+. Our drafter trained on only 10K samples just isn't good enough."

The AQ-MedAI test was the last resort — a check to see if a pre-trained drafter from a reputable source could salvage the approach without further investment. When it too failed, the path forward became clear: more training data. The assistant pivots to scaling up the training dataset by 10×, launching an inference pipeline to generate responses for 83,000 prompts through the Kimi-K2.5 model, capturing the target model's actual token distribution for retraining.

Assumptions and Their Validation

This message tests several implicit assumptions:

Assumption 1: The AQ-MedAI drafter, being trained on more data, would achieve higher acceptance. This was a reasonable hypothesis. The AQ-MedAI project is well-known in the open-source ML community, and their EAGLE-3 drafter was trained on general data. However, the assumption failed because the drafter was trained for Kimi-K2 (the previous generation), not Kimi-K2.5. The token distributions and model behavior differ enough that the drafter's predictions don't align well with K2.5's target distribution.

Assumption 2: The hidden state bug fix would enable both drafters to work correctly. This assumption was validated. Both drafters now receive the correct 21,504-dimensional hidden states and produce non-trivial acceptance rates. The bug fix was successful.

Assumption 3: The acceptance metrics would be informative enough to guide the next decision. This assumption was validated. The metrics clearly show that neither drafter is viable, and the path forward is more training data.

Technical Knowledge Required

To fully understand this message, one needs knowledge of:

The Thinking Process

The assistant's reasoning in this message is economical but precise. The command is targeted — it greps specifically for accept len in the AQ-MedAI log file, showing only the last 10 lines. This indicates the assistant knows exactly what metrics matter and where to find them. The choice of tail -10 suggests the assistant wants to see steady-state performance after the server has been running for a while, not the initial startup phase.

The output is read and interpreted silently. No further commands are issued to investigate the AQ-MedAI drafter. The assistant moves on to the next phase: scaling up training data. This is a mature engineering judgment — recognizing when further tuning is unlikely to yield results and when a fundamentally different approach is needed.

Output Knowledge Created

This message creates several pieces of knowledge:

  1. The AQ-MedAI EAGLE-3 drafter is not suitable for Kimi-K2.5. Its acceptance metrics are nearly identical to the custom 10K-trained drafter, and its throughput is slightly worse.
  2. The hidden state bug fix is confirmed to work correctly for multiple draft models. Both the custom and AQ-MedAI drafters now receive proper hidden states and produce non-trivial acceptance.
  3. The bottleneck is training data quantity, not model architecture or server configuration. Both drafters plateau at ~2.0 accept length, consistent with the EAGLE-3 paper's finding that acceptance scales with training data.
  4. The strategic direction is set. The project will invest in generating 83,000+ training samples and retraining the drafter from scratch, rather than continuing to tune hyperparameters or search for pre-trained alternatives.

Conclusion

Message [msg 3629] is a quiet turning point. It doesn't contain dramatic announcements or complex code changes. It is simply a developer checking whether a hypothesis held up — and finding that it didn't. The AQ-MedAI drafter, the last easy option, failed to improve upon the custom drafter. This forced the project to commit to the harder but more promising path: scaling training data by an order of magnitude.

In the broader narrative of the coding session, this message marks the transition from debugging and tuning to data generation and retraining. The hidden state bug was fixed, two drafters were tested and found inadequate, and the team now knows exactly what they need to do. Sometimes the most important messages are not the ones that announce success, but the ones that honestly report failure and thereby illuminate the correct path forward.