The Smoking Gun: Why a Trained EAGLE-3 Draft Model Achieved Zero Acceptance

In the complex world of speculative decoding with EAGLE-3, few moments are as simultaneously frustrating and illuminating as discovering that a carefully trained draft model — one that achieved 74.5% step-0 accuracy on validation data during training — achieves exactly zero token acceptance when deployed in production. This is precisely the situation captured in message 3595 of an extended coding session focused on deploying speculative decoding for the Kimi-K2.5 large language model. The message represents the culmination of a multi-hour debugging odyssey, the moment when a subtle architectural mismatch between training assumptions and inference reality was finally exposed.

The Context: A Long Debugging Arc

The session leading up to message 3595 had been an exercise in systematic elimination. The assistant and user had been working for days to train an EAGLE-3 draft model for the Kimi-K2.5 architecture — a 1.2-billion-parameter auxiliary model designed to predict multiple future tokens in parallel, accelerating inference through speculative execution. The training pipeline had been laboriously constructed: hidden states extracted from the target model via SGLang, a custom training script built on the speculators library, and a draft model trained from scratch with frozen embeddings and a learned language model head operating in a reduced 32K-token draft vocabulary.

When the trained checkpoint was first loaded into SGLang for inference, the results were devastating. The server reported accept_len: 1.00, accept_rate: 0.20 — essentially, the draft model's predictions were being rejected at every step, reducing speculative decoding to a no-op that merely added overhead. The assistant had already eliminated several possible causes. The d2t token mapping (which converts draft token IDs back to target token IDs) was verified to be correct — stored as offsets, not absolute IDs. The weight key names had been fixed: the speculators library saves decoder layers as layers.0.* but SGLang's LlamaForCausalLMEagle3 expected midlayer.*, causing trained weights to be silently dropped during loading. The language model head was confirmed to be loaded correctly from the checkpoint rather than replaced by the target model's head. Each fix brought the team closer to the truth, but acceptance remained stubbornly at zero.

The Message: A Confirmation and a Root Cause

Message 3595 is brief but devastatingly precise:

Confirmed: accept len: 1.00, accept rate: 0.20 — and the root cause is clear: hidden_states are 7168-dim, not 21504-dim. The 3-layer aux hidden states aren't being concatenated.

The message then pivots to investigating why the auxiliary hidden state mechanism isn't activating, checking the draft model's configuration:

The issue is that eagle_use_aux_hidden_state is set based on the draft model's config reading eagle_config. But this reading happens in the eagle_worker, and it reads from the draft model's HF config — which is our config.json. Let me check what our config has: [bash] ssh root@10.1.230.174 "cat /data/eagle3/output_10k_sglang/4/config.json | python3 -m json.tool | grep -A5 eagle" "eagle_config": { "eagle_aux_hidden_state_layer_ids": [ 2, 30, 58 ], "use_aux_hidden_state": true

This message is the moment of truth — the "smoking gun" that the previous message (msg 3594) had identified. The debug instrumentation added to llama_eagle3.py had revealed that the hidden states flowing into the draft model had shape [21, 7168] — a single layer's hidden state dimension — rather than the expected [21, 21504] that would result from concatenating three auxiliary layer hidden states (3 × 7168 = 21504).

The Technical Mechanism: Why Dimensions Matter

To understand why this is catastrophic, one must understand the EAGLE-3 architecture's feature fusion mechanism. EAGLE-3 improves upon earlier speculative decoding approaches by conditioning the draft model on hidden states from multiple layers of the target model, not just the final layer. The intuition is that intermediate layers capture different levels of abstraction — early layers encode syntactic and surface-level features, middle layers capture semantic content, and later layers are most predictive of the next token. By concatenating these multi-scale features, the draft model receives a richer representation of the context, enabling more accurate predictions.

In the Kimi-K2.5 architecture, which has 60 layers, the draft model's configuration specified layers 2, 30, and 58 as auxiliary hidden state sources. The expected pipeline is: the target model's forward pass captures hidden states at these three layers, concatenates them into a 21504-dimensional vector (3 × 7168), and passes this fused representation to the draft model. The draft model's fc (fully connected) layer then projects this 21504-dimensional input down to 7168 dimensions for further processing by the draft transformer.

The debug output revealed that this pipeline was silently broken. The hidden states arriving at the draft model were 7168-dimensional — the final layer's output only, not the concatenated multi-layer features. The condition hidden_states.shape[-1] != embeds.shape[-1] evaluated to 7168 != 7168 → False, causing the fc projection layer to be completely bypassed. The draft model was receiving impoverished single-layer features instead of the rich multi-layer representation it was trained on.

The Root Cause Investigation

The message shows the assistant pivoting to investigate why the auxiliary hidden state mechanism isn't activating. The hypothesis is that eagle_use_aux_hidden_state — a flag that controls whether the target model captures and exposes intermediate layer hidden states — is not being properly set for the KimiK25 model architecture.

The assistant checks the draft model's config.json and finds that use_aux_hidden_state is indeed set to true with the correct layer IDs [2, 30, 58]. But the critical insight is that this configuration is read by the eagle worker from the draft model's HF config. The question becomes: does SGLang's target model implementation for KimiK25 properly implement the capture_aux_hidden_states mechanism? The set_eagle3_layers_to_capture method (visible in the code exploration at msg 3588) is responsible for registering the layer IDs with the target model's forward pass. If this method is not called, or if the KimiK25 model class doesn't support it, the auxiliary hidden states will never be captured, and the draft model will receive only the final layer's output.

This explains a puzzling observation: both the old vLLM-trained drafter and the new SGLang-trained drafter exhibited identical zero-acceptance behavior. Both were trained on fused multi-layer features (21504-dimensional) but both received single-layer features (7168-dimensional) at inference time. The training had taught the draft model to expect a specific structure of concatenated features, but inference provided an entirely different input representation.

Assumptions and Misconceptions

The debugging journey reveals several assumptions that turned out to be incorrect. The assistant initially assumed that the weight key name mismatch was the sole cause of the zero acceptance — a reasonable hypothesis given that the layers.* vs midlayer.* discrepancy would cause the trained decoder weights to be silently dropped. Fixing this improved weight loading but didn't change acceptance rates.

Another assumption was that the token mapping (d2t / hot_token_id) might be the issue. The assistant spent considerable effort verifying that the draft-to-target token mapping was correctly stored as offsets and that SGLang's logits processor was correctly applying the mapping. This was a red herring — the mapping was correct all along.

The most fundamental assumption was that the auxiliary hidden state capture mechanism would "just work" for the KimiK25 model because it was configured in the draft model's config. The assistant discovered that configuration alone is insufficient — the target model's runtime must actually implement the capture, and this implementation may be model-specific. The KimiK25 architecture, being a custom model with its own forward pass logic, may not have the hooks that SGLang's LlamaForCausalLM model uses for auxiliary state capture.

Input and Output Knowledge

To fully understand this message, one needs knowledge of: the EAGLE-3 speculative decoding architecture, particularly its use of multi-layer hidden state fusion; the SGLang inference server's speculative decoding infrastructure, including the eagle_worker.py and llama_eagle3.py components; the concept of auxiliary hidden state capture in transformer models; the Kimi-K2.5 model architecture (a 60-layer DeepSeek-style model with MLA attention); and the speculators library's training pipeline and weight naming conventions.

The message creates critical output knowledge: it definitively establishes that the zero acceptance rate is caused by a dimensional mismatch in the hidden state features passed to the draft model, not by weight loading errors or token mapping issues. It identifies the specific mechanism by which the mismatch occurs (the fc fusion layer is bypassed because the shape check evaluates to false). And it narrows the remaining investigation to a specific question: why is eagle_use_aux_hidden_state not properly activating the auxiliary capture mechanism for the KimiK25 model?

The Broader Significance

This message represents a classic debugging pattern in machine learning systems: a model that trains perfectly but fails at inference due to a subtle mismatch between the training data distribution and the inference-time input distribution. The draft model learned to predict tokens conditioned on 21504-dimensional fused features, but at inference it received 7168-dimensional single-layer features. The model's predictions, while internally consistent, were based on a fundamentally different input representation than what it was trained on, leading to rejection at every step.

The message also highlights the fragility of complex ML pipelines that span multiple frameworks. The training pipeline (built on speculators and Hugging Face) and the inference pipeline (built on SGLang) must agree on dozens of implicit contracts — weight naming conventions, tensor shapes, configuration keys, runtime hooks. A single broken contract can render a trained model completely useless, even when all the individual components appear to be working correctly.

Conclusion

Message 3595 is the turning point in a challenging debugging session. It transforms the problem from "why is our trained model failing?" to "why isn't the auxiliary hidden state capture mechanism activating for KimiK25?" — a much more specific and tractable question. The assistant's systematic approach — eliminating one hypothesis at a time, adding debug instrumentation, checking configurations, and reasoning about the data flow — exemplifies the disciplined debugging that complex ML systems demand. The dimensional mismatch revealed in this message is the key that unlocks the remaining investigation, pointing directly at the target model's runtime implementation as the source of the problem.