The Hidden State Wiring Bug: Debugging EAGLE-3 Speculative Decoding from 56.8 tok/s to Root Cause

Introduction

In the high-stakes world of large language model inference, every token per second matters. When deploying speculative decoding — a technique where a small "draft" model predicts multiple tokens cheaply and a large "target" model verifies them in parallel — the promise is transformative throughput gains. But when the numbers don't add up, the debugging journey can reveal deep, unexpected flaws in the pipeline.

This article chronicles a multi-hour debugging session spanning dozens of messages in an opencode coding session, where an AI assistant and user worked to deploy an EAGLE-3 draft model for the Kimi-K2.5 1-trillion-parameter Mixture-of-Experts model on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The draft model had achieved 74.7% validation accuracy during training, yet when deployed with SGLang's speculative decoding, it delivered a paltry 56.8 tokens per second — far below the 90 tok/s non-speculative baseline. What followed was a systematic investigation that uncovered not one but two critical bugs, culminating in the discovery of a silent hidden state wiring mismatch between training and inference that explained the catastrophic performance gap.

The First Bug: A Silent Configuration Override

The initial benchmark at message [msg 4356] told a sobering story: 56.8 tok/s with EAGLE-3 and 16 draft tokens configured. The server logs revealed an accept length of only ~1.6 tokens — meaning the draft model's predictions were being rejected almost immediately. With 16 draft tokens generated and mostly discarded, the speculation overhead was destroying performance.

The assistant's first breakthrough came from investigating SGLang's internal parameter handling. A subagent task (message [msg 4360]) traced through the server argument code and discovered a critical constraint: when --speculative-eagle-topk 1 is set (the only sensible value for EAGLE-3's chain-based speculation), SGLang silently forces num_draft_tokens = num_steps + 1. The server had been started with --speculative-num-steps 1 (the default), which meant num_draft_tokens was clamped to just 2 — not the 16 the user had explicitly configured. The draft model was only ever proposing one token per step, making the entire speculation mechanism nearly pointless.

The fix was straightforward: set --speculative-num-steps 15 to match the 16 draft tokens. But when the assistant restarted the server and re-ran the benchmark at message [msg 4370], the result was even worse: 46.7 tok/s. The accept length barely budged, hovering around 1.9 tokens. This was the moment of crisis — the hypothesis that the num_steps override was the primary bottleneck had been decisively falsified. Something deeper was wrong with how the draft model was being used.

The Pivot: Testing on Training Data

At this critical juncture, the user suggested a diagnostic experiment that would reframe the entire investigation. At message [msg 4374], the user proposed: "Try with a training sample and see if accept rate looks correct. Possible we didn't wire in the model correctly."

This was a classic debugging heuristic: if a model performs well on held-out validation data during training but fails in deployment, test it on training data to distinguish between a generalization problem and an integration problem. If the model also fails on training data, the issue is in how it's loaded or invoked; if it succeeds on training data but fails on new data, the issue is generalization.

The assistant immediately recognized the value of this approach. At message [msg 4375], the assistant articulated the diagnostic framework: "If accept rate is high on training data but low on new data, it's a generalization issue. If it's also low on training data, we have a wiring/loading bug." This binary framework would guide the next several hours of work.

Building the Standalone Test

The assistant's next step was to build a standalone test script that bypassed SGLang entirely. At message [msg 4379], the assistant wrote test_drafter_standalone.py — a Python script that loaded the draft model weights from safetensors, loaded a training sample's hidden states from disk, manually ran the EAGLE-3 forward pass, and compared the predicted next token against the ground truth.

The script was SCP'd to the remote server and executed at message [msg 4388]. The initial run showed the model weights loading correctly — all expected keys present with the correct shapes and dtypes. But the critical discovery came in the following message ([msg 4389]): 0.0% accuracy on training data. The draft model was completely unable to predict the next token on data it had been explicitly trained on.

This was the smoking gun. The problem was definitively a wiring or loading bug, not a generalization issue. But what kind of wiring bug could cause a model to achieve 0% accuracy on its own training data?

The Hidden State Revelation

The assistant's investigation of the training data format revealed a crucial detail. The hidden states directory contained 4 tensors per sample, not the 3 the script initially expected. At message [msg 4385], the assistant examined the tensor norms and discovered a striking pattern:

Tracing Through the Training Pipeline

The answer came from reading the speculators library's data preprocessing code. At message [msg 4462], the assistant read the standardize_data_v1 function in the training library's data.py:

def standardize_data_v1(data: dict[str, Any]) -> dict[str, Any]:
    return {
        "hidden_states": torch.cat(data["hidden_states"][:-1], dim=-1),
        "input_ids": data["input_ids"],
        "verifier_last_hidden_states": data["hidden_states"][-1],
        ...
    }

The critical line was data["hidden_states"][:-1] — Python slice notation that takes all elements except the last one. Since the hidden states list contained [embed_output, layer3_output, layer31_output, layer59_output], the [:-1] slice selected [embed_output, layer3_output, layer31_output]. These three tensors were concatenated along the feature dimension to form the 21,504-dimensional input to the fc layer.

The last hidden state (layer59_output) was stored separately as verifier_last_hidden_states, used for a different purpose in the training loop.

The Root Cause: A Silent Wiring Mismatch

At message [msg 4463], the assistant synthesized the discovery:

Training hidden_states = cat([embed, layer3, layer31], dim=-1) = cat(data["hidden_states"][:-1]) → [seq_len, 3*7168] But our HS dump has 4 entries: [embed, layer3, layer31, layer59] Training uses the first 3 (embed + 2 aux layers), NOT the last 3 (3 aux layers) This means the fc layer was trained on cat([embed_output, layer3, layer31]) but we're feeding it cat([layer3, layer31, layer59]). That's the bug!

The draft model's fc layer had been trained on a specific combination of hidden states: the embedding output (which carries information about the previous token) plus two auxiliary hidden states from layers 3 and 31. But SGLang, configured with eagle_aux_hidden_state_layer_ids = [2, 30, 58], was capturing only the three auxiliary hidden states (layers 3, 31, and 59) and concatenating them — completely omitting the embedding output.

The embedding output is qualitatively different from the layer outputs. It has a norm of ~1 compared to norms of 37–174 for the layer outputs, meaning the fc layer had learned to expect a specific pattern of magnitudes in its input. When it received three large-magnitude layer outputs instead of one small-magnitude embedding plus two large-magnitude layer outputs, the projections were completely wrong.

The Fix and Its Aftermath

The fix required two changes. First, the SGLang model code (deepseek_v2.py) needed to be modified to capture the embedding output when layer_id=-1 is specified in the layer IDs configuration. Second, the draft model config needed to be updated from [2, 30, 58] to [-1, 2, 30] so that the embedding output was included as the first auxiliary hidden state.

The assistant implemented these changes at messages <msg id=4464-4470>, restarted the server, and re-ran the benchmark. The standalone test confirmed the fix: with the correct input format (cat([embed, layer3, layer31])), accuracy jumped from 34.1% to 76.9% — matching the training metrics. The draft model itself was fine; it had just been receiving the wrong inputs.

However, the production benchmark after the fix showed only modest improvement: 54.8 tok/s with an accept length of ~1.8 out of 6 draft tokens, still far below the 90 tok/s baseline. This suggested that while the hidden state wiring mismatch was real and had been corrected, additional issues remained in the inference pipeline — possibly related to tensor parallelism, KV cache management, or the interaction between the draft model and the target model's verification step.

Lessons Learned

This debugging session illustrates several fundamental principles for deploying speculative decoding systems:

1. Training-inference data format alignment is critical. The most subtle bugs are often not in the model weights or the training algorithm, but in the silent assumptions that bridge the gap between training and deployment. A single Python slice ([:-1]) was the root cause of a catastrophic performance collapse.

2. Standalone tests are invaluable. By isolating the draft model from SGLang's complex inference pipeline, the assistant could test the model directly against training data and definitively distinguish between a generalization problem and a wiring problem. This test was the key to finding the bug.

3. Configuration parameters can silently override each other. The --speculative-num-steps 1 flag silently overriding --speculative-num-draft-tokens 16 was a hidden dependency that wasted significant debugging time. Understanding the full parameter interaction graph is essential.

4. Embedding outputs are not the same as layer outputs. The embedding layer produces hidden states with fundamentally different statistical properties (norm ~1 vs norms of 37-174 for transformer layers). Confusing them with layer outputs can silently corrupt the input to downstream components.

5. The debugging process is iterative. Each fix revealed new layers of the problem. The num_steps fix exposed the wiring bug, and the wiring fix exposed remaining performance issues. Effective debugging requires the flexibility to abandon hypotheses and pivot when the data doesn't match expectations.

Conclusion

The EAGLE-3 debugging journey chronicled in this chunk is a masterclass in systematic ML engineering debugging. From the initial disappointing benchmark of 56.8 tok/s, through the discovery of the silent num_steps override, to the pivotal standalone test that revealed a 0% accuracy on training data, and finally to the root cause — a hidden state wiring mismatch caused by a single [:-1] slice in the training data pipeline — each step built on the previous one.

The fix improved performance to 54.8 tok/s, still below the 90 tok/s baseline, indicating that additional challenges remain. But the core discovery — that the draft model's fc layer was trained on cat([embed, layer3, layer31]) while SGLang was passing cat([layer3, layer31, layer59]) — was the essential piece that no amount of hyperparameter tuning or model retraining could have solved. It required tracing the data flow through four separate codebases: the training data extraction pipeline, the speculators library's data preprocessing, the draft model configuration, and SGLang's inference engine.

In the frontier of LLM inference optimization, the most elusive bugs are often not in the model weights or the training algorithm, but in the silent assumptions that bridge the gap between training and deployment. This session stands as a testament to the power of systematic debugging, standalone testing, and the willingness to question every assumption when the numbers don't add up.## The Architecture Investigation: Tracing Through the Speculators Library

Before the standalone test could reveal the wiring mismatch, the assistant had to understand the draft model's architecture in detail. This required tracing through the speculators library's core.py and model_definitions.py files to reconstruct the exact forward pass.

At message [msg 4403], the assistant synthesized the forward pass into five steps:

  1. hidden_states = self.fc(hidden_states) — maps 3×hidden_size to hidden_size
  2. input_embeds = self.embed_tokens(input_ids) — embed the previous token
  3. hidden_states = torch.cat([input_embeds, hidden_states], dim=-1) — concatenate to 2×hidden_size
  4. Pass through decoder layers with RoPE
  5. logits = self.lm_head(self.norm(hidden_states)) — project to vocabulary But this raised an immediate question: the concatenation in step 3 produces a 2×hidden_size tensor, yet the RMSNorm in step 5 expects hidden_size. How does the decoder layer reduce the dimension? The answer came from reading the decoder layer source code. The first decoder layer is special — it splits the 2×hidden_size input at the midpoint, applies separate LayerNorms to each half, recombines them for attention, and produces a hidden_size output. This "split-and-recombine" pattern is a non-trivial architectural choice that has profound implications for how hidden states must be prepared before being fed into the draft model. At message [msg 4408], the assistant wrote: "Now I understand the full architecture. The first layer is special." This was followed by a seven-step description of the forward pass that captured the split-and-recombine logic. The assistant then identified the critical next question: "Now the key question: what does SGLang's llama_eagle3.py do vs speculators?" This question drove the comparison that would eventually reveal the hidden state format mismatch. The assistant read the full llama_eagle3.py file from the SGLang source tree and compared it line by line against the speculators implementation. The comparison revealed that SGLang's fc projection only runs if the hidden state dimension doesn't match the embedding dimension — if the hidden states are already 7168-dimensional, the fc layer is skipped entirely. This was a clue that the hidden states arriving at the draft model were not in the expected format.

The Vocabulary Mapping Red Herring

Along the way, the assistant investigated and ruled out several potential causes. One significant red herring was the d2t vocabulary mapping tensor. The draft model uses a reduced vocabulary of 32,000 tokens (down from the target model's 163,840), and the d2t tensor maps draft token IDs to target token IDs using an offset format: target_id = draft_id + d2t[draft_id].

The assistant initially suspected that SGLang might misinterpret this offset mapping. At messages <msg id=4390-4396>, the assistant traced through SGLang's llama_eagle3.py and discovered that it correctly converts offsets to direct mappings via hot_token_id = loaded_weight + torch.arange(loaded_weight.shape[0]). The standalone test confirmed this: hot_token_id[:10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], showing that the first 10 draft tokens map to the first 10 target tokens as expected.

This investigation, while ultimately a dead end, was valuable because it eliminated a whole class of potential bugs and narrowed the search to the hidden state input format.

Tracing the Hidden State Pipeline in SGLang

The assistant also traced how SGLang captures hidden states from the target model. At message [msg 4419], the assistant grepped the DeepSeek V2 model code for layers_to_capture and discovered the mechanism:

self.model.layers_to_capture = [val + 1 for val in layer_ids]

This off-by-one shift means that a layer_ids value of [2, 30, 58] becomes [3, 31, 59] in the actual capture list. The assistant also discovered that the default layer selection formula was [2, num_layers // 2, num_layers - 3] — early, middle, and late layers.

But the critical insight was that SGLang's capture mechanism only captured transformer layer outputs, not the embedding output. There was no mechanism to capture the embedding layer because it's not a transformer layer — it's a separate embedding table that runs before any transformer layers. This is why the training pipeline had to be patched to capture the embedding output separately, and why SGLang's default configuration silently omitted it.

The Verification Step

After identifying the root cause, the assistant verified the fix by modifying the standalone test to use the correct input format (cat([embed, layer3, layer31]) instead of cat([layer3, layer31, layer59])). The accuracy jumped from 34.1% to 76.9%, matching the training metrics. This verification was critical because it proved that the draft model itself was fine — the weights learned during training were correct. The problem was entirely in how SGLang fed inputs to the model at inference time.

The production fix required two changes: modifying deepseek_v2.py to capture the embedding output when layer_id=-1 is specified, and updating the draft model config from [2, 30, 58] to [-1, 2, 30]. After these changes, the server was restarted and re-benchmarked, achieving 54.8 tok/s — an improvement from the 46.7 tok/s after the num_steps fix, but still far below the 90 tok/s baseline.

Broader Implications

This debugging session illustrates a fundamental truth about deploying speculative decoding systems: the training data preparation pipeline and the inference engine's hidden state capture mechanism are often developed independently, and their assumptions about data format may not align. The training pipeline captured four hidden states and used the first three for the fc input. The inference engine, configured with the same layer IDs, captured only the three transformer layer outputs and omitted the embedding. Neither was wrong in isolation, but together they produced a silent wiring mismatch that destroyed performance.

The session also demonstrates the importance of building standalone tests that bypass the serving infrastructure. Without the standalone test, the assistant might have spent days tuning hyperparameters, adjusting server configurations, or retraining the model — all while the real bug was a simple slice operation in the data preprocessing code. The standalone test isolated the draft model from SGLang and revealed that the model itself was correct, forcing the investigation to focus on the interface between training and inference.

In the end, the fix improved performance but did not fully solve the problem. The remaining gap — 54.8 tok/s vs 90 tok/s baseline — suggests that additional issues remain in the inference pipeline beyond the input format mismatch. These might include tensor parallelism interactions, KV cache management during speculative decoding, or the overhead of the verification step itself. But the core discovery — the hidden state wiring mismatch — was the essential piece that no amount of parameter tuning could have addressed.## References

[1] "The Art of the State Dump: How an AI Assistant Preserves Context Across a Multi-Hour Machine Learning Deployment" — Message 4348 analysis covering the project state summary before benchmarking began.

[9] "The Moment of Truth: Benchmarking EAGLE-3 Speculative Decoding at 56.8 tok/s" — Message 4356 analysis of the initial disappointing benchmark.

[13] "The Silent Override: Debugging SGLang's Speculative Decoding Configuration" — Message 4361 analysis of the --speculative-num-steps override bug.

[22] "The Moment a Hypothesis Crumbles: Debugging EAGLE-3 Speculative Decoding at 46.7 tok/s" — Message 4370 analysis of the worsened performance after fixing the num-steps parameter.

[27] "The Diagnostic Pivot: Testing EAGLE-3 with Training Data to Uncover a Wiring Bug" — Message 4375 analysis of the pivot to testing on training data.

[31] "Isolating the Signal: How a Standalone Test Uncovered the EAGLE-3 Hidden State Wiring Mismatch" — Message 4379 analysis of the standalone test creation.

[40] "The Standalone Test That Exposed EAGLE-3's Hidden State Wiring Bug" — Message 4388 analysis of the first standalone test execution.

[52] "The Shape of Silence: Debugging a Hidden State Mismatch in EAGLE-3 Speculative Decoding" — Message 4400 analysis of the forward pass shape mismatch.

[55] "The Architecture Detective: Tracing EAGLE-3's Hidden State Dimensions Through Source Code" — Message 4403 analysis of the forward pass architecture synthesis.

[60] "The Architecture Revealed: A Pivotal Debugging Moment in EAGLE-3 Speculative Decoding" — Message 4408 analysis of the full architecture understanding.

[71] "Tracing the Hidden State Pipeline: Debugging EAGLE-3's layers_to_capture in DeepSeek V2" — Message 4419 analysis of the hidden state capture mechanism.

[112] "The Verification That Confirmed a Hidden Bug: Reading the Training Script" — Message 4460 analysis of the training script verification.

[114] "The Smoking Gun: Tracing an EAGLE-3 Hidden State Mismatch to Its Source" — Message 4462 analysis of the standardize_data_v1 function discovery.

[115] "The Eureka Moment: Uncovering the Hidden State Mismatch in EAGLE-3 Speculative Decoding" — Message 4463 analysis of the root cause synthesis.