The Moment of Discovery: Debugging EAGLE-3's Zero Acceptance Rate

In the complex world of speculative decoding with EAGLE-3, debugging a model that achieves zero acceptance rate despite training to 74.5% step-0 accuracy can feel like chasing a ghost. The subject message — <msg id=3593> — is the pivotal moment in that chase, where a carefully placed debug print finally reveals the fundamental mismatch between training assumptions and inference reality. This single message, containing a bash command and its debug output, crystallizes hours of systematic debugging into one irrefutable piece of evidence.

The Debugging Journey That Led Here

To understand why this message was written, we must trace the path that led to it. The assistant had just completed training an EAGLE-3 draft model for the Kimi-K2.5 architecture — a 1.2B-parameter model designed to predict hidden states that accelerate the main 164K-vocabulary target model. When deployed on SGLang, the draft model showed accept_len ~1.00, accept_rate ~0.20, meaning essentially zero draft tokens were accepted by the target model's verification step. This was identical to the behavior of a previously trained vLLM-based drafter, suggesting a systematic issue rather than a training quality problem.

The assistant had already eliminated several potential causes. First, a weight key name mismatch was discovered and fixed: the speculators training library saves the decoder layer under the key layers.0.*, but SGLang's LlamaForCausalLMEagle3 expects midlayer.*. This meant the trained weights were silently dropped during loading, and the draft model was running with randomly initialized weights. Fixing this was necessary but insufficient — acceptance remained zero.

Second, the d2t (draft-to-target) vocabulary mapping was investigated. The assistant initially thought it was broken, discovering that d2t[:20] showed all zeros while the first 20 target token IDs were [0,1,2,...,19]. This led to a mistaken "fix" that actually broke the mapping further. Upon deeper analysis, the assistant realized the d2t stores offsets (not absolute IDs): target_id = draft_idx + d2t[draft_idx], so zeros for the first 20 entries is correct because draft IDs 0-19 map directly to target IDs 0-19. The original file was correct, and the assistant reverted the damage.

Third, the lm_head loading path was verified. The assistant confirmed that SGLang's EAGLE-3 worker calls set_embed(embed) to replace the embedding with the target model's embedding, but leaves the draft model's own lm_head intact (since load_lm_head_from_target=False). The hot_token_id mapping, computed as d2t + arange(32000), correctly maps draft predictions back to target token space. All these checks passed, yet acceptance remained stubbornly at zero.

The Instrumentation Decision

With all high-level theories exhausted, the assistant made a tactical decision: add a debug print directly into the draft model's forward pass to observe what data it actually receives at inference time. This is a classic debugging technique — when you can't reason about what should happen, instrument the code to see what does happen.

The assistant wrote a Python script (/tmp/add_debug.py) that patched /root/sglang/python/sglang/srt/models/llama_eagle3.py, inserting diagnostic prints right after the line:

hidden_states = forward_batch.spec_info.hidden_states

The debug output would print the shape, dtype, mean, and standard deviation of hidden_states, along with the shape and dtype of embeds (the token embeddings), the first few input_ids, and the weight statistics of the fc fusion layer. This instrumentation was designed to answer a single question: what hidden states does the draft model actually receive?

The assistant then killed any running processes, launched a fresh SGLang server with the EAGLE-3 draft model, waited for it to become healthy, and sent a test request ("What is 2+2?"). After a brief sleep to allow the request to process, it grepped the server log for the EAGLE3-DEBUG prefix.

The Revealing Output

The debug output in <msg id=3593> is devastatingly clear:

[EAGLE3-DEBUG] hidden_states shape=torch.Size([21, 7168]), dtype=torch.bfloat16

The hidden states are 7168-dimensional, not the expected 21504-dimensional concatenation of three auxiliary layer hidden states. The draft model was trained on fused multi-layer features — hidden states from layers 20, 40, and 60 of the target model, concatenated into a 21504-dim vector (3 × 7168), then projected down to 7168-dim by the fc fusion layer. But at inference time, it receives only a single layer's 7168-dim hidden states.

This explains the zero acceptance rate with brutal simplicity. The draft model's fc fusion layer, which expects 21504-dim input and projects to 7168-dim, is never invoked because the shape check in the forward pass:

if hidden_states.shape[-1] != embeds.shape[-1]:
    hidden_states = self.fc(hidden_states)

evaluates to 7168 != 7168 → False. The hidden states pass through untouched, but they're single-layer features, not the fused multi-layer features the draft model was trained on. The draft model receives input that is dimensionally correct but semantically wrong — like feeding a model trained on stereo audio a mono signal that happens to have the same sample rate.

Root Cause: The Missing Auxiliary Hidden State Capture

The fundamental issue is that eagle_use_aux_hidden_state is not properly activated for the KimiK25 model, or equivalently, the target model's capture_aux_hidden_states mechanism is not producing the multi-layer hidden states that the draft model expects. The SGLang server is configured to capture hidden states from the target model's intermediate layers and pass them to the draft model, but this mechanism appears to be capturing only the final layer's hidden states (or the last transformer block's output) rather than the three specific intermediate layers used during training.

This explains why both the old vLLM-trained drafter and the new SGLang-trained drafter exhibit identical zero-acceptance behavior. It's not a training quality issue, a weight loading issue, or a vocabulary mapping issue — it's a feature extraction configuration issue in the inference server. The draft model was trained on rich, multi-layer features, but at inference time it receives impoverished single-layer features. No amount of training data or model capacity can overcome this fundamental mismatch.

The Thinking Process Visible in This Message

The subject message itself is deceptively simple — a bash command and its output — but it represents the culmination of a rigorous debugging process. The assistant's thinking, visible across the preceding messages, follows a clear pattern:

1. Formulate hypothesis: "Maybe the d2t mapping is wrong." Test it. Find it's actually correct (after a brief detour into a mistaken fix).
2. Formulate next hypothesis: "Maybe the weight key names don't match." Test it. Find and fix the mismatch.
3. Formulate next hypothesis: "Maybe the lm_head isn't being loaded correctly." Test it by tracing through the SGLang code. Find it's correct.
4. Formulate next hypothesis: "Maybe the hidden states themselves are wrong." This requires instrumentation because it can't be verified statically.
5. Instrument and observe: Add debug prints, launch server, send request, collect output.
6. Interpret results: The hidden states are 7168-dim instead of 21504-dim. The fc layer is bypassed. The auxiliary capture mechanism isn't working. This is textbook systematic debugging: eliminate the easy possibilities first, then instrument to observe the unobservable. The assistant resists the temptation to jump to conclusions or make premature fixes, instead letting the data guide the investigation.

Input Knowledge Required

To fully understand this message, one needs:

• EAGLE-3 architecture knowledge: Understanding that the draft model receives hidden states from the target model's intermediate layers, concatenated across multiple layers, then projected down via an fc fusion layer. The training pipeline captured hidden states from layers 20, 40, and 60 of the Kimi-K2.5 target model.
• SGLang internals: Understanding that forward_batch.spec_info.hidden_states carries the captured hidden states from the target model to the draft model, and that the shape check hidden_states.shape[-1] != embeds.shape[-1] controls whether the fusion projection is applied.
• The training pipeline: The draft model was trained using the speculators library with aux_hidden_layers=[20, 40, 60], producing a 21504-dim fused feature that the model learned to predict.
• The debugging history: The weight key name fix (renaming layers.0 to midlayer) and the d2t offset correction are necessary context for why the assistant finally resorted to instrumentation.

Output Knowledge Created

This message produces a single, critical piece of knowledge: the hidden states flowing into the draft model are 7168-dimensional, not 21504-dimensional. This knowledge:

1. Explains the zero acceptance rate: The draft model receives single-layer features but was trained on multi-layer features.
2. Identifies the root cause: The auxiliary hidden state capture mechanism is not properly configured for the KimiK25 model architecture.
3. Rules out alternative explanations: It's not a training quality issue, weight loading issue, vocabulary mapping issue, or lm_head configuration issue.
4. Guides the next steps: The fix must involve either (a) properly activating eagle_use_aux_hidden_state for KimiK25 in SGLang's model registry, (b) patching the target model's forward pass to capture and concatenate the correct intermediate layers, or (c) retraining the draft model on single-layer features if multi-layer capture cannot be enabled.

Assumptions and Their Validity

The assistant operated under several assumptions, some validated and some refuted:

• Assumption: The d2t mapping stores absolute token IDs. Refuted — it stores offsets. This led to a brief but instructive detour.
• Assumption: The weight key name mismatch was the sole cause of zero acceptance. Refuted — fixing it didn't help.
• Assumption: The fc fusion layer would be applied if hidden states were 21504-dim. Validated — the code logic is correct, but the condition never triggers because hidden states are 7168-dim.
• Assumption: The auxiliary hidden state capture mechanism is active. Refuted — the debug output proves it's not producing multi-layer features.

Conclusion

Message <msg id=3593> is the turning point in a complex debugging session. It transforms the investigation from speculation to measurement, from "what could be wrong?" to "what is wrong?" The debug output provides an unambiguous answer: the draft model receives single-layer hidden states at inference time, despite being trained on fused multi-layer features. This single observation explains the zero acceptance rate, unifies the behavior of two independently trained drafters (vLLM and SGLang), and points directly to the necessary fix. It's a masterclass in systematic debugging — when static analysis and code reading fail, instrumentation reveals the truth.