The Acceptance Rate Ceiling: Diagnosing Why EAGLE-3 Speculation Still Underperforms After the Hidden State Fix

Introduction

In the long-running effort to deploy speculative decoding with EAGLE-3 on a Kimi-K2.5 model, message [msg 3624] represents a moment of sobering clarity. After the previous messages had traced and fixed a critical bug—the server was started with --speculative-algorithm EAGLE instead of EAGLE3, causing the auxiliary hidden state capture mechanism to never activate—the assistant now confronts the benchmark results. The fix worked: hidden states are now correctly arriving as 21504-dimensional concatenated tensors instead of the 7168-dim final-layer-only states. The draft model's predictions are being accepted. But the throughput numbers tell a disappointing story: 53.2 tok/s with the custom drafter, still far below the 90 tok/s non-speculative baseline.

This message is the inflection point where the assistant shifts from debugging a technical fault to confronting a fundamental scaling limitation. It is a diagnosis message, not a fix message—and the diagnosis is that the trained drafter simply is not good enough yet.

The Message in Context

To understand message [msg 3624], we need to trace the narrative arc of the preceding messages. The assistant had spent considerable effort debugging why the EAGLE-3 draft model achieved zero acceptance rate despite being trained on 10,000 samples of hidden state data. The root cause, discovered in [msg 3604] and [msg 3605], was a flag mismatch: the SGLang server was launched with --speculative-algorithm EAGLE, but the is_eagle3() function in the codebase is strict—it only returns True for the string "EAGLE3". With "EAGLE", the auxiliary hidden state capture mechanism was never activated, meaning the target model only returned its final-layer hidden states (7168 dimensions) instead of the concatenated intermediate-layer states (21504 dimensions) that the draft model's fc fusion layer expects.

After restarting with the correct flag ([msg 3607]), the assistant confirmed that hidden states were now arriving as torch.Size([21, 21504]) ([msg 3609]), and a quick sanity check verified that the fc.weight tensor was indeed non-zero ([msg 3611]). The draft model was now receiving the correct inputs.

The first benchmark with the corrected setup ([msg 3612]) showed an average throughput of 56.7 tok/s with num_draft_tokens=16 and accept_len around 2.0-2.5. This was an improvement over the effectively zero acceptance before, but still far below the 90 tok/s baseline. The assistant then tried reducing num_draft_tokens to 5 ([msg 3618]), which improved the accept rate to ~0.41-0.45 (because fewer draft tokens means a higher ratio of accepted tokens), but the accept_len remained stubbornly at ~2.0-2.2, and throughput actually dropped slightly to 53.2 tok/s ([msg 3622]).

This is where message [msg 3624] begins.

The Content of the Subject Message

The message opens with a concise summary of the latest benchmark:

With num_draft_tokens=5: accept_rate ~0.41-0.45, accept_len ~2.0-2.2, throughput 53.2 tok/s. Still slower than the 90 tok/s baseline.

The assistant then articulates the core diagnosis:

The fundamental problem: accept_len of ~2.1 is too low. With an ideal EAGLE-3 drafter, accept_len should be 3-4+. Our drafter trained on only 10K samples just isn't good enough. The speculation overhead (running the draft model for each step, doing verification) exceeds the gains from ~2.1× accept length, especially without CUDA graphs.

This is a crucial analytical insight. The assistant correctly identifies that the bottleneck is not a bug but a data scaling problem. The EAGLE-3 paper's scaling curves show that acceptance length improves with more training data, and 10K samples is simply insufficient for a model of this size (7168-dim hidden states, 21504-dim concatenated features).

The message then pivots to a comparative experiment:

Let me also try the AQ-MedAI drafter with EAGLE3 to see if that one does better (it was a general K2 drafter, not K2.5-specific, but it was trained on more data)

The assistant executes a bash command to read the AQ-MedAI drafter's configuration:

ssh root@10.1.230.174 'cat /data/eagle3/aq-medai-k2-drafter/config.json | python3 -c "import sys,json; c=json.load(sys.stdin); print(json.dumps({k:v for k,v in c.items() if \"eagle\" in k.lower() or k==\"hidden_size\" or k==\"draft_vocab_size\" or k==\"architectures\"}, indent=2))"'

The output confirms that the AQ-MedAI drafter uses the same architecture (LlamaForCausalLMEagle3), the same hidden size (7168), the same draft vocab size (32000), and—critically—the same auxiliary hidden state layer IDs [2, 30, 58]. This means it is architecturally compatible with the Kimi-K2.5 target model, even though it was originally trained for DeepSeek-K2 rather than K2.5 specifically.

Reasoning and Decision-Making

The reasoning in this message operates on multiple levels. First, the assistant performs a quantitative analysis: accept_len of ~2.1 with speculation overhead yields net throughput that is ~40% slower than the baseline. The assistant correctly reasons that the overhead of running the draft model's forward pass for each speculation step, combined with the verification pass of the target model over the draft tokens, consumes more time than the ~2.1× effective speedup from accepted tokens provides.

Second, the assistant makes an inference about scaling laws: the EAGLE-3 paper demonstrates that acceptance length scales with training data quantity. The custom drafter was trained on only 10K samples, which is a tiny dataset for a model with 7168-dimensional hidden states and a complex fusion architecture. The assistant implicitly assumes that more training data would improve accept_len, an assumption grounded in the published literature.

Third, the assistant performs a comparative evaluation strategy: by testing the AQ-MedAI drafter (trained on more data, though for a different base model), the assistant can isolate whether the low acceptance is a data quantity issue or a model-specific issue. If AQ-MedAI performs better, it suggests the custom drafter needs more data. If it performs similarly or worse, it might indicate a deeper architectural or training methodology problem.

The decision to read the AQ-MedAI config before launching the server is methodologically sound: the assistant first verifies architectural compatibility (same layer IDs, same hidden size, same vocab size) to avoid wasting time on an incompatible model.

Assumptions Made

Several assumptions underpin the reasoning in this message:

  1. The EAGLE-3 paper's scaling curves apply to this specific setup. The assistant assumes that training on more data will improve accept_len in a predictable way. This is a reasonable assumption given that the paper shows monotonic improvement with data scale, but it's not guaranteed—the Kimi-K2.5 model uses Multi-head Latent Attention (MLA), which differs from the standard attention used in the models tested in the paper.
  2. CUDA graphs would significantly reduce overhead. The assistant mentions "especially without CUDA graphs" as a compounding factor. The assumption is that enabling CUDA graphs would reduce the per-step overhead enough that even an accept_len of 2.1 might yield net speedup. This is tested in subsequent messages ([msg 3632] onwards).
  3. The AQ-MedAI drafter was trained on more data. The assistant states "it was a general K2 drafter, not K2.5-specific, but it was trained on more data." This is an inference—the assistant does not have direct evidence of the training dataset size for AQ-MedAI's model, but reasonably assumes that a publicly released drafter had more resources behind it.
  4. The architectural compatibility is sufficient. The assistant assumes that because the AQ-MedAI drafter uses the same layer IDs [2, 30, 58] and the same hidden size, it will work with Kimi-K2.5. This is correct in terms of tensor shapes, but the drafter's internal representations are optimized for DeepSeek-K2's hidden state distribution, not K2.5's. The subsequent benchmark ([msg 3628]) shows AQ-MedAI achieving 50.5 tok/s, slightly worse than the custom drafter's 53.2 tok/s, confirming this assumption was partially correct—compatible but suboptimal.

Mistakes and Incorrect Assumptions

The message itself does not contain explicit mistakes, but it reveals an implicit assumption that turns out to be incorrect: that the AQ-MedAI drafter would perform better due to more training data. The subsequent benchmark ([msg 3628]) shows it performs slightly worse (50.5 tok/s vs 53.2 tok/s, accept_len ~1.8-2.1 vs ~2.0-2.2). This is analyzed in the follow-up message ([msg 3630]):

AQ-MedAI drafter: 50.5 tok/s, accept_len ~1.8-2.1. Slightly worse than our custom drafter (53.2 tok/s, accept_len ~2.1). This makes sense — AQ-MedAI was trained for DeepSeek-K2 (different model, same architecture) while our custom drafter was specifically trained on K2.5 hidden states.

The assistant correctly interprets this result: training on the correct target model's hidden states matters more than raw data quantity. The custom K2.5-trained drafter, despite its smaller dataset, produces better acceptance because it has learned the specific token distribution of Kimi-K2.5.

Another subtle point: the assistant mentions "without CUDA graphs" as a caveat, but the subsequent attempt to enable CUDA graphs ([msg 3632]) shows that even with CUDA graphs, the throughput improvement is modest. The fundamental limitation is the accept_len, not the graph compilation overhead.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of speculative decoding: The concept of a draft model generating candidate tokens that a target model verifies in parallel. The key metrics are accept_len (how many draft tokens are accepted on average per verification step) and accept_rate (accept_len divided by num_draft_tokens).
  2. Knowledge of EAGLE-3 architecture: Specifically, that EAGLE-3 uses auxiliary hidden states from intermediate layers of the target model, concatenated together, to condition the draft model's predictions. The fc layer projects the concatenated 21504-dim states down to 7168-dim for the draft model's internal processing.
  3. Understanding of the bug context: The previous discovery that --speculative-algorithm EAGLE vs EAGLE3 disabled the auxiliary hidden state capture. Without this context, the benchmark numbers would seem confusing—why would the fix not immediately yield speedup?
  4. Familiarity with SGLang's server flags: --speculative-num-draft-tokens, --speculative-num-steps, --speculative-eagle-topk, --disable-cuda-graph, and how they interact.
  5. Knowledge of the hardware setup: 8× RTX PRO 6000 Blackwell GPUs with tensor parallelism (tp-size 8), running the Kimi-K2.5 INT4 quantized model.

Output Knowledge Created

This message creates several important outputs:

  1. A clear diagnosis: The bottleneck is not a bug but insufficient training data. The accept_len of ~2.1 is fundamentally too low for speculation to be beneficial given the current overhead profile.
  2. A benchmark comparison point: The custom K2.5-trained drafter at 53.2 tok/s with accept_len ~2.0-2.2 serves as the baseline for all future experiments.
  3. A validated architectural compatibility check: The AQ-MedAI drafter's config confirms the correct layer IDs [2, 30, 58] and hidden size 7168, validating that the training pipeline and the SGLang integration are correctly configured.
  4. A decision point: The assistant implicitly decides that the path forward is more training data, not more debugging. This sets the stage for the massive data scaling effort in subsequent messages, where the assistant launches an inference pipeline to generate 83K+ training samples.
  5. A negative result that is valuable: The finding that even a well-trained public drafter (AQ-MedAI) achieves only ~2.0 accept_len on this model confirms that the challenge is fundamental to the model architecture (MLA attention) rather than a training quality issue.

The Thinking Process Visible in the Message

The assistant's reasoning in this message follows a clear diagnostic pattern:

Step 1: State the quantitative result. "With num_draft_tokens=5: accept_rate ~0.41-0.45, accept_len ~2.0-2.2, throughput 53.2 tok/s."

Step 2: Compare to the baseline. "Still slower than the 90 tok/s baseline."

Step 3: Identify the root cause dimension. "The fundamental problem: accept_len of ~2.1 is too low."

Step 4: Reference the ideal target. "With an ideal EAGLE-3 drafter, accept_len should be 3-4+."

Step 5: Explain why the current model falls short. "Our drafter trained on only 10K samples just isn't good enough."

Step 6: Explain the mechanism. "The speculation overhead (running the draft model for each step, doing verification) exceeds the gains from ~2.1× accept length, especially without CUDA graphs."

Step 7: Formulate a comparative hypothesis. "Let me also try the AQ-MedAI drafter with EAGLE3 to see if that one does better."

Step 8: Test the hypothesis by reading the config. The assistant reads the AQ-MedAI config to verify compatibility before committing to a server restart.

This structured reasoning—from observation to diagnosis to hypothesis to verification—is characteristic of effective debugging. The assistant does not jump to conclusions or try random fixes. Instead, it systematically narrows down the possible causes and designs experiments to isolate them.

The Broader Significance

Message [msg 3624] is significant because it marks the transition from the debugging phase to the scaling phase of the project. The hidden state concatenation bug was a hard blocker—without it, the draft model was receiving garbage inputs and could never work. But fixing the bug revealed a softer, more fundamental limitation: the draft model simply needs more training data.

This is a common pattern in machine learning engineering: the hardest bugs are often the ones that silently degrade performance rather than crashing the system. The EAGLE vs EAGLE3 flag mismatch caused the draft model to silently receive the wrong inputs, producing outputs that were technically valid (non-NaN, correct shape) but useless for speculation. It took careful debugging—adding debug prints, checking tensor shapes, verifying weight statistics—to trace the issue.

Once the bug was fixed, the assistant faced the true challenge: the EAGLE-3 paper's results were achieved with hundreds of thousands of training samples, not 10K. The assistant's decision to scale up the training data by 10× (launching an 83K-sample inference pipeline in subsequent messages) follows directly from the diagnosis in this message.

Conclusion

Message [msg 3624] is a masterclass in diagnostic reasoning applied to speculative decoding. The assistant correctly identifies that the low acceptance length (~2.1) is the root cause of the throughput regression, that more training data is the primary lever for improvement, and that even a well-trained public drafter (AQ-MedAI) cannot overcome the fundamental data limitation. The message creates a clear decision point: stop debugging, start scaling. This diagnosis drives the entire subsequent arc of the project, from the 83K-sample inference pipeline to the retraining of the EAGLE-3 drafter from scratch. It is a reminder that in machine learning systems, the most impactful insights often come not from fixing bugs but from understanding the quantitative relationships that govern performance.