The Hidden State Detective: Debugging EAGLE-3's 25% Acceptance Rate on Kimi-K2.5

Introduction

In the high-stakes world of large language model inference optimization, few techniques promise as much as speculative decoding. The idea is elegant: use a small, fast "draft" model to predict the next several tokens, then have the large "target" model verify them in parallel. If the draft model is accurate enough, throughput can double or more. But when speculative decoding fails, it fails silently — the draft model simply predicts garbage, every token gets rejected, and you're left with the overhead of running two models for no benefit.

This article examines a critical sequence of messages from an opencode coding session where an AI assistant is deep in the trenches of exactly such a failure. The assistant had trained a custom EAGLE-3 draft model for Kimi-K2.5 (a massive Mixture-of-Experts model running on 8 NVIDIA Blackwell GPUs), but the drafter achieved only a 25% acceptance rate — effectively zero tokens accepted, meaning the speculative decoding pipeline was completely broken. Across messages 3253–3257, the assistant systematically investigates why, tracing through source code, comparing conventions between vLLM and SGLang, and ultimately pivoting from debugging to a strategic plan for retraining.

The Context: EAGLE-3 on Blackwell GPUs

To understand the significance of this chunk, we need to step back. The session had been running for hours across multiple segments, building an increasingly sophisticated inference pipeline. The team had deployed Kimi-K2.5, a 547GB INT4-quantized Mixture-of-Experts model, on 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang. After extensive profiling, they discovered that the dominant bottleneck was AllReduce communication, consuming 51.5% of decode time. The natural hardware solution — Expert Parallelism — was infeasible due to PCIe bandwidth limitations. So they turned to speculative decoding as a software-only optimization.

The assistant had built a complete EAGLE-3 training pipeline, generating 10,000 synthetic training samples by capturing Kimi-K2.5's actual reasoning outputs, extracting hidden states using the speculators library with vLLM, and fine-tuning a draft model. But when they tested the resulting drafter on SGLang, the acceptance rate was abysmal. A pre-trained drafter from AQ-MedAI (trained on the non-quantized Kimi-K2 using SGLang) achieved 42% acceptance — still not enough for a speedup, but at least functional. The custom drafter, trained on the actual INT4 model the team was using, achieved 25%, which meant it was predicting essentially random tokens.

This was deeply puzzling. Training on the exact same model should produce a drafter that understands its quirks. Something was fundamentally wrong.

The Investigation Begins: Message 3253

The first message in this chunk [1] captures a seemingly simple grep command that targets four specific patterns in the speculators library's custom_worker.py:

grep -n "_setup_hidden_states_capture\|register_forward_hook\|layers_to_capture\|hidden_states.*append" \
  /root/ml-env/lib/python3.12/site-packages/speculators/data_generation/custom_worker.py | head -20

The output reveals three critical lines:

Peering Deeper: Message 3254

Building on the initial grep, the assistant widens the investigation in message 3254 [2] with a targeted sed command to examine lines 78–130 of the same file:

sed -n "78,130p" /root/ml-env/lib/python3.12/site-packages/speculators/data_generation/custom_worker.py

The output reveals the heart of the hidden state capture mechanism:

hidden_states, residual = layer(
    hidden_states=hidden_states, positions=positions, residual=residual
)

absolute_layer_idx = self.start_layer + idx

if absolute_layer_idx in target_layers:
    aux_hidden_states.append((hidden_states + residual).clone())

This code is inside a loop that iterates over the model's transformer layers. At each layer, the forward pass is called with the current hidden states, positions, and residual stream. After the layer computation completes, the code checks whether the current layer's absolute index (accounting for pipeline parallelism via self.start_layer) is in the set of target_layers. If so, it appends (hidden_states + residual).clone() to the aux_hidden_states list.

The critical detail here is when the capture happens: after the layer runs. So if target_layers contains indices [2, 30, 58], the captured states represent the output of those layers — which is the input to layers 3, 31, 59. This is exactly the convention that SGLang's EAGLE-3 integration uses with its +1 offset: SGLang sets layers_to_capture = [val + 1 for val in layer_ids] and captures hidden_states + residual before the layer runs, which gives the same result — the output of layer val.

The output is truncated at hidden_st..., cutting off the rest of the file. This truncation is significant because it means the assistant cannot yet see what happens with the captured hidden states — how they are returned, stored, or processed. The full picture requires seeing lines beyond 130.

The Insight: Message 3255

Message 3255 [3] is where the pieces click together. The assistant synthesizes the findings from the previous two messages and arrives at a critical realization:

Now I see the critical detail: speculators captures (hidden_states + residual) AFTER the layer runs (line 86: hidden_states, residual = layer(...) then aux_hidden_states.append((hidden_states + residual).clone())). This is the output of the target layer.

>

SGLang captures (hidden_states + residual) BEFORE the target layer runs (at layer i+1, capturing input to i+1 = output of i).

>

These are the same values! When speculators extracts at layer [2, 30, 58], it captures the output of layers 2, 30, 58. When SGLang captures at layers [3, 31, 59], it captures the input to layers 3, 31, 59, which is also the output of layers 2, 30, 58.

This is the kind of insight that only comes from reading source code carefully. The assistant had previously assumed that the layer offset difference between vLLM and SGLang was the root cause of the drafter failure. But the two conventions are mathematically equivalent. The +1 offset in SGLang is just an indexing quirk: SGLang captures the input to layer N+1, which equals the output of layer N. The speculators library captures the output of layer N directly. Same values, different indexing.

Having ruled out the simplest explanation, the assistant immediately pivots to deeper investigation:

So the hidden states should match! The issue with our drafter isn't the layer offset — it's something else. Let me check if the vLLM and SGLang DeepSeek models handle the residual connection differently, or if the INT4 dequantization path differs.

This pivot reveals the assistant's mental model of the problem space. There are two remaining hypotheses:

  1. Residual connection handling: The DeepSeekV2 architecture (which Kimi-K2.5 is based on) uses a residual stream where each layer's output is hidden_states + residual. If vLLM and SGLang implement this differently — for example, if one framework normalizes or scales the residual differently — the hidden states would diverge even with the same input tokens.
  2. INT4 dequantization path: This is the more likely culprit. Kimi-K2.5 is loaded in INT4 precision using NVIDIA's MXFP4 format. The dequantization from INT4 back to BF16/FP16 for computation is not a uniquely defined operation — different frameworks may use different dequantization algorithms, different scaling factors, or different numerical precision. If vLLM and SGLang dequantize the weights differently, the hidden states produced by the same model on the same input would be different, and a drafter trained on vLLM's hidden states would fail when given SGLang's hidden states.

Confirming the Code Path: Message 3256

Message 3256 [4] continues the investigation by confirming the exact code path that speculators uses to patch the model. The assistant writes:

I see — the speculators extraction patches base_model.forward with _patched_forward. And for Kimi-K25, it follows the path model.language_model.model (DeepseekV2Model). So the forward pass IS going through the same layers.

>

The real problem might be simpler: the training was done with layer IDs [2, 30, 58, 60] but the speculators extraction captured the output of these layers, while the EAGLE-3 draft model's forward pass in SGLang receives hidden states from layers_to_capture = [3, 31, 59] which are the same values. So the data should match...

>

Unless vLLM's DeepSeek forward path handles hidden_states + residual differently than SGLang's. Let me check vLLM's path.

The assistant confirms that the forward pass goes through the same layers in both frameworks — speculators follows the path model.language_model.model (DeepseekV2Model) for Kimi-K25. This eliminates the hypothesis that the two frameworks are patching different model classes or different points in the computation graph.

But the assistant doesn't stop there. It immediately recognizes that even with the same layer paths, the actual computation of hidden_states + residual might differ. The decision to read vLLM's forward path with another sed command is strategic — the assistant is looking at the specific code that handles the DeepSeek forward pass in vLLM, comparing it to what was already seen in SGLang's deepseek_v2.py.

This message illustrates a fundamental challenge in the ML engineering ecosystem: framework inconsistency. When two frameworks (vLLM and SGLang) implement the same model architecture, they may produce slightly different numerical results due to differences in kernel implementations, operator fusion, memory layout, or quantization paths. These differences are usually negligible for generation quality but can be catastrophic for speculative decoding, where the drafter is trained to predict specific hidden state values.

The Pivot: Message 3257

Message 3257 [5] is the most unusual message in this sequence — it contains nothing but an empty <conversation_data> tag. A blank slate. Yet this seemingly vacuous message triggered one of the most comprehensive status summaries in the entire session.

To understand why this empty message was sent, one must understand the state of the conversation when it arrived. The assistant had been deep in a debugging rabbit hole for several messages, examining the speculators library's custom_worker.py to understand how vLLM extracted hidden states versus how SGLang captured them during EAGLE-3 inference. The assistant had just traced through line 86 of the custom worker, discovering that speculators captures (hidden_states + residual) after each target layer runs, while SGLang captures the same quantity before the next layer runs — confirming they should be equivalent values. The assistant was in the middle of checking whether vLLM's DeepSeek forward path handled the residual differently, running a sed command on the remote machine.

Then the user sent message 3257: empty.

In human conversation, silence or an empty response can carry many meanings: agreement, impatience, confusion, or simply "go on." In the context of an AI-assisted coding session, an empty user message is particularly significant. The user was not asking a question, not issuing a new instruction, not correcting a mistake. They were, in effect, saying: "I've seen enough. Proceed."

This interpretation is supported by what happened next. The assistant did not continue the debugging thread. Instead, the assistant's response was a comprehensive, structured summary of the entire project — a 2,000+ word document that consolidated weeks of work into a coherent plan of action. It began with a clear "Goal" section, listed "Instructions" for the remote environment, catalogued "Discoveries" about hardware and software, presented benchmark results in a table, and laid out "Accomplished" and "Next Steps" sections. This was not a continuation of the layer-capture investigation. It was a reset.

The empty message prompted a zoom-out from analytical debugging to strategic planning. Instead of asking "does vLLM capture hidden states at the same layers as SGLang?", the assistant now asked "what do we need to do to get speculative decoding working?" The answer was pragmatic: re-extract hidden states using SGLang (eliminating any possible mismatch), retrain with 15K samples instead of 10K, and potentially train from scratch instead of fine-tuning from the AQ-MedAI checkpoint.

The Deeper Puzzle: Why the Drafter Failed

Despite the assistant's thorough investigation, the root cause of the 25% acceptance rate remains unresolved at the end of this chunk. The layer offset hypothesis has been ruled out — both frameworks capture the same values at the same logical positions. The remaining hypotheses are:

  1. INT4 dequantization divergence: vLLM and SGLang may use different dequantization kernels for the MXFP4 format, producing numerically different hidden states even with the same input and weights.
  2. Attention backend differences: vLLM uses PagedAttention while SGLang uses flashinfer. These different attention implementations can produce slightly different numerical results, especially at the boundaries of the attention computation.
  3. CUDA graph compilation: SGLang uses CUDA graphs to optimize inference, which can change the order of operations and introduce numerical differences compared to vLLM's eager execution.
  4. Residual connection handling: Even though both frameworks capture hidden_states + residual, the internal computation of hidden_states and residual might differ due to different normalization placement or different handling of the residual stream. The pragmatic response — re-extract using SGLang — bypasses the mystery entirely. By using the same framework for both training data extraction and inference, any cross-framework inconsistencies are eliminated. This is the engineering equivalent of "if you can't fix it, work around it."

Lessons for Practitioners

This chunk of the conversation offers several important lessons for engineers working with speculative decoding:

1. Training data must match inference conditions exactly. The hidden states produced by a model depend not just on the model weights, but on the exact software implementation used to run inference. Different serving frameworks, different attention backends, and different quantization paths can all produce numerically different hidden states. A drafter trained on one framework's hidden states may fail when deployed on another.

2. Source code reading is essential. The assistant's key insight came from reading the actual source code of both the speculators library and SGLang's model implementation. Documentation and high-level APIs can obscure critical details about when and how hidden states are captured. When debugging speculative decoding failures, there is no substitute for tracing through the code.

3. Layer indexing conventions matter. A seemingly trivial detail — whether to capture hidden states before or after a layer runs, and whether to index from 0 or 1 — can determine the success or failure of a multi-day training effort. The assistant spent significant effort verifying that the +1 offset convention in SGLang was equivalent to the direct indexing in the speculators library.

4. Know when to pivot. The empty message from the user served as a signal to stop debugging and start executing. The assistant had been going down a rabbit hole investigating layer capture conventions, but the practical solution was simpler: re-extract using SGLang and retrain. Recognizing when further analysis has diminishing returns is a critical skill.

5. Cross-framework transfer is risky. The EAGLE-3 training pipeline used vLLM for hidden state extraction and SGLang for inference. This cross-framework approach introduced an unnecessary source of variability. The assistant's plan to switch to SGLang-based extraction aligns with the principle of minimizing differences between training and deployment environments.

Conclusion

Messages 3253–3257 capture a pivotal sequence in a complex debugging session. The assistant systematically investigates why a custom EAGLE-3 drafter achieves only 25% acceptance rate, tracing through source code, comparing conventions between vLLM and SGLang, and ruling out the layer offset hypothesis. The investigation reveals that both frameworks capture equivalent hidden state values — the output of layers 2, 30, and 58 — just indexed differently. This forces the assistant to confront deeper questions about numerical consistency between frameworks.

The empty message at index 3257 serves as a turning point, prompting the assistant to zoom out from debugging to strategic planning. The resulting plan — re-extract hidden states using SGLang, retrain with 15K samples, and potentially train from scratch — represents a pragmatic response to a complex problem. Rather than continuing to chase the root cause of the cross-framework mismatch, the assistant chooses to eliminate the mismatch entirely by using the same framework for both training and inference.

This chunk demonstrates the kind of systematic, evidence-based debugging that characterizes effective engineering in large-scale ML systems. It also highlights the fragility of speculative decoding pipelines, where seemingly minor differences between frameworks can silently corrupt the training signal. For practitioners building similar systems, the lesson is clear: ensure that your training data extraction and inference deployment use the same framework, or be prepared for mysterious failures that resist all but the most determined debugging.