The Hidden State Detective: Debugging EAGLE-3's 25% Acceptance Rate on Kimi-K2.5

Introduction

In the high-stakes world of large language model inference optimization, few techniques promise as much as speculative decoding. The idea is elegant: use a small, fast "draft" model to predict the next several tokens, then have the large "target" model verify them in parallel. If the draft model is accurate enough, throughput can double or more. But when speculative decoding fails, it fails silently — the draft model simply predicts garbage, every token gets rejected, and you're left with the overhead of running two models for no benefit.

This article examines a single message ([msg 3255]) from an opencode coding session where an AI assistant is deep in the trenches of exactly such a failure. The assistant had trained a custom EAGLE-3 draft model for Kimi-K2.5 (a massive Mixture-of-Experts model running on 8 NVIDIA Blackwell GPUs), but the drafter achieved only a 25% acceptance rate — effectively zero tokens accepted, meaning the speculative decoding pipeline was completely broken. The message captures a pivotal moment of debugging: the assistant realizes that the hidden state extraction convention between vLLM (used for training data generation) and SGLang (used for inference) might actually be correct after all, and pivots to deeper investigation.

The Context: EAGLE-3 on Blackwell GPUs

To understand the significance of this message, we need to step back. The session had been running for hours across multiple segments, building an increasingly sophisticated inference pipeline. The team had deployed Kimi-K2.5, a 547GB INT4-quantized Mixture-of-Experts model, on 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang. After extensive profiling, they discovered that the dominant bottleneck was AllReduce communication, consuming 51.5% of decode time. The natural hardware solution — Expert Parallelism — was infeasible due to PCIe bandwidth limitations. So they turned to speculative decoding as a software-only optimization.

The assistant had built a complete EAGLE-3 training pipeline, generating 10,000 synthetic training samples by capturing Kimi-K2.5's actual reasoning outputs, extracting hidden states using the speculators library with vLLM, and fine-tuning a draft model. But when they tested the resulting drafter on SGLang, the acceptance rate was abysmal. A pre-trained drafter from AQ-MedAI (trained on the non-quantized Kimi-K2 using SGLang) achieved 42% acceptance — still not enough for a speedup, but at least functional. The custom drafter, trained on the actual INT4 model the team was using, achieved 25%, which meant it was predicting essentially random tokens.

This was deeply puzzling. Training on the exact same model should produce a drafter that understands its quirks. Something was fundamentally wrong.

The Message: A Detective's Reasoning

The subject message begins with a critical realization. The assistant has been reading the speculators library source code and comparing it with SGLang's EAGLE-3 integration code. Let's quote the key passage:

Now I see the critical detail: speculators captures (hidden_states + residual) AFTER the layer runs (line 86: hidden_states, residual = layer(...) then aux_hidden_states.append((hidden_states + residual).clone())). This is the output of the target layer.

>

SGLang captures (hidden_states + residual) BEFORE the target layer runs (at layer i+1, capturing input to i+1 = output of i).

>

These are the same values! When speculators extracts at layer [2, 30, 58], it captures the output of layers 2, 30, 58. When SGLang captures at layers [3, 31, 59], it captures the input to layers 3, 31, 59, which is also the output of layers 2, 30, 58.

This is the kind of insight that only comes from reading source code carefully. The assistant had previously assumed (in [msg 3250]) that the layer offset difference between vLLM and SGLang was the root cause of the drafter failure. The EAGLE-3 architecture captures hidden states at three intermediate layers — typically layers 2, 30, and 58 out of 60 — and uses these as conditioning features for the draft model. If the training data captured hidden states from different layers than those used during inference, the draft model would receive completely unfamiliar inputs and fail.

But the assistant now realizes the two conventions are mathematically equivalent. The +1 offset in SGLang is just an indexing quirk: SGLang captures the input to layer N+1, which equals the output of layer N. The speculators library captures the output of layer N directly. Same values, different indexing.

The Pivot: Deeper Investigation

Having ruled out the simplest explanation, the assistant immediately pivots to deeper investigation:

So the hidden states should match! The issue with our drafter isn't the layer offset — it's something else. Let me check if the vLLM and SGLang DeepSeek models handle the residual connection differently, or if the INT4 dequantization path differs.

This pivot reveals the assistant's mental model of the problem space. There are two remaining hypotheses:

  1. Residual connection handling: The DeepSeekV2 architecture (which Kimi-K2.5 is based on) uses a residual stream where each layer's output is hidden_states + residual. If vLLM and SGLang implement this differently — for example, if one framework normalizes or scales the residual differently — the hidden states would diverge even with the same input tokens.
  2. INT4 dequantization path: This is the more likely culprit. Kimi-K2.5 is loaded in INT4 precision using NVIDIA's MXFP4 format. The dequantization from INT4 back to BF16/FP16 for computation is not a uniquely defined operation — different frameworks may use different dequantization algorithms, different scaling factors, or different numerical precision. If vLLM and SGLang dequantize the weights differently, the hidden states produced by the same model on the same input would be different, and a drafter trained on vLLM's hidden states would fail when given SGLang's hidden states. The assistant then begins investigating the speculators custom worker code to understand exactly how it patches the model forward pass:
Let me check: during vLLM extraction, did we use the same model class (KimiK25) or did we use a different path?

This is a crucial question. The speculators library needs to patch the forward pass of the specific model class to capture hidden states. If it's patching a different class or a wrapper (like KimiK25ForConditionalGeneration) instead of the actual DeepseekV2Model, the hidden states might come from a different point in the computation graph.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of context:

EAGLE-3 architecture: EAGLE-3 is a speculative decoding framework that trains a lightweight Transformer "draft" model to predict the target model's next tokens. The draft model is conditioned on hidden states from intermediate layers of the target model — specifically, it receives the hidden states from layers 2, 30, and 58 (for a 60-layer model). The draft model learns to predict what the target model would generate, and the target model then verifies these predictions in parallel.

DeepSeekV2 residual stream: The DeepSeekV2 architecture (and its Kimi-K2.5 derivative) uses a residual stream where each transformer layer computes an update to the hidden states, and the output is hidden_states + residual. This is different from the standard Pre-LN Transformer where the residual is implicit in the layer output. The hidden_states + residual value is what gets passed to the next layer, and it's what EAGLE-3 captures.

The +1 offset convention: In SGLang's EAGLE-3 implementation, the layers_to_capture parameter uses a +1 offset convention. The configuration layers_to_capture = [3, 31, 59] means "capture the input to layers 3, 31, and 59." Since the input to layer N is the output of layer N-1, this is equivalent to capturing the output of layers 2, 30, and 58.

INT4 quantization: Kimi-K2.5 is loaded in INT4 precision using NVIDIA's MXFP4 format. This is a lossy compression that reduces model size by ~4x but introduces quantization error. Different inference frameworks may implement INT4 dequantization differently, leading to different numerical values in the hidden states.

The two drafters: The team had two EAGLE-3 draft models to test. The AQ-MedAI drafter was trained on the non-quantized Kimi-K2 model using SGLang by a third party. The custom drafter was trained on the INT4-quantized Kimi-K2.5 using vLLM-extracted hidden states from the team's own inference runs.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. The layer offset hypothesis is ruled out: The assistant definitively establishes that the +1 offset in SGLang's convention does not cause a mismatch with speculators' convention. Both capture the same values — the output of layers 2, 30, and 58 — just indexed differently.
  2. A refined problem statement: The real problem is narrowed to either residual connection handling differences or INT4 dequantization path differences between vLLM and SGLang. This is a much more specific and actionable diagnosis.
  3. A debugging methodology: The assistant demonstrates a systematic approach to debugging speculative decoding failures: read the source code of both frameworks, compare the exact points in the computation graph where hidden states are captured, verify mathematical equivalence, and only then move on to deeper hypotheses.
  4. The importance of framework alignment: The message implicitly highlights a critical lesson for speculative decoding: the training data generation pipeline must use the same inference framework as the deployment pipeline. Hidden states are not framework-agnostic — they depend on the exact numerical implementation of the model.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message:

That the hidden states are truly equivalent: The assistant concludes that speculators' [2, 30, 58] and SGLang's [3, 31, 59] produce the same values. This is mathematically correct if both frameworks capture hidden_states + residual at the same point in the computation graph. However, there's a subtlety: speculators captures the value after the layer computes, while SGLang captures it before the layer computes. These are the same value only if no transformation happens between the output of layer N and the input to layer N+1. In standard Transformer architectures, this is true — the output of layer N is directly the input to layer N+1. But if there's any normalization, scaling, or routing between layers (which some architectures have), the equivalence breaks.

That the model class path is correct: The assistant assumes that speculators is patching the right model class. The follow-up message ([msg 3256]) confirms that speculators follows the path model.language_model.model to reach the DeepseekV2Model. But if there's any wrapper or adapter in between that modifies the hidden states, the patching might capture values at a different point than expected.

That the INT4 dequantization is the likely culprit: The assistant leans toward the INT4 dequantization path as the explanation. This is a reasonable hypothesis, but it's not the only possibility. The difference could also come from different KV cache implementations, different attention kernels (flashinfer vs. triton vs. trtllm_mla), or different tensor parallelism implementations.

The Thinking Process

What makes this message particularly interesting is the visible reasoning process. The assistant doesn't just present a conclusion — it walks through the evidence step by step:

  1. Observation: speculators captures hidden states AFTER the layer runs.
  2. Comparison: SGLang captures hidden states BEFORE the next layer runs.
  3. Analysis: These are mathematically equivalent — the output of layer N equals the input to layer N+1.
  4. Conclusion: The layer offset is not the problem.
  5. Pivot: The real problem must be elsewhere — either residual connection handling or INT4 dequantization.
  6. Next step: Investigate the speculators custom worker code to verify the model class being patched. This is textbook debugging methodology: formulate a hypothesis, test it against the evidence, rule it out if it doesn't fit, and move to the next hypothesis. The assistant is effectively acting as a senior engineer tracing through a complex system failure. The message also shows the assistant's ability to hold multiple hypotheses simultaneously. It doesn't commit to either the residual-connection hypothesis or the dequantization hypothesis — it identifies both as possibilities and prepares to investigate both.

Broader Implications

This message, while focused on a specific technical detail, illustrates several broader principles about AI systems engineering:

The reproducibility crisis in ML: Hidden states are not unique. Two different inference frameworks, even running the same model with the same weights, can produce different hidden states due to numerical differences in implementation. This makes cross-framework transfer of training data unreliable.

The importance of source code reading: The assistant's insight came from reading the actual source code of both speculators and SGLang. In a field where practitioners often rely on documentation or high-level APIs, this message demonstrates the value of going to the source.

Debugging speculative decoding: Speculative decoding is notoriously difficult to debug because failures are silent — the system still produces output, just slower. The assistant's systematic approach to understanding why the drafter failed is a model for how to debug these systems.

Conclusion

Message [msg 3255] captures a pivotal moment in a complex debugging session. The assistant rules out a plausible hypothesis (layer offset mismatch) and pivots to deeper investigation of the actual root cause. The reasoning is clear, systematic, and grounded in source code evidence. While the message doesn't resolve the problem — that will require further investigation into the INT4 dequantization paths — it demonstrates the kind of careful, evidence-based thinking that characterizes effective debugging of complex ML systems.

The lesson for practitioners is clear: when building speculative decoding pipelines, ensure that the training data generation and inference deployment use the same framework, or at least validate that hidden states are numerically identical across frameworks. Otherwise, you may find yourself with a 25% acceptance rate and no idea why.