Tracing the Hidden State Pipeline: Debugging EAGLE-3's layers_to_capture in DeepSeek V2

The Message

In the midst of a deep debugging session on EAGLE-3 speculative decoding performance, the assistant issued the following command:

[assistant] Now let me check how the DeepSeek V2 model (which KimiK25 delegates to) handles `layers_to_capture`:
[bash] ssh root@10.1.230.174 'grep -n "layers_to_capture\|captured_hidden\|set_eagle3" ~/sglang/python/sglang/srt/models/deepseek_v2.py | head -20'
2631:        self.layers_to_capture = []
2721:                if i in self.layers_to_capture:
2963:    def set_eagle3_layers_to_capture(self, layer_ids: Optional[List[int]] = None):
2970:            self.model.layers_to_capture = [2, num_layers // 2, num_layers - 3]
2975:            self.model.layers_to_capture = [val + 1 for val in layer_ids]

This short message — a single grep command with a brief explanatory comment — sits at a pivotal moment in a multi-hour investigation. To the casual observer, it looks like a trivial code search. But in context, it represents a critical fork in the debugging road: the moment when the assistant pivots from examining the draft model's internals to examining the target model's hidden state capture mechanism. Understanding why this grep was issued, what assumptions it tested, and what knowledge it produced reveals the layered reasoning behind one of the most subtle bugs in speculative decoding.

Context: The EAGLE-3 Performance Crisis

To understand this message, one must first understand the crisis that precipitated it. The assistant and user had spent the better part of a day training an EAGLE-3 draft model for KimiK25, a large language model that internally delegates to a DeepSeek V2 architecture. The training pipeline had produced a checkpoint with a respectable 74.7% validation accuracy — seemingly a strong result. Yet when deployed with SGLang's speculative decoding engine, the drafter achieved a paltry 54.8 tokens per second against a 90.0 tok/s baseline, with an accept length of barely 1.8 out of 6 draft tokens. The draft model was essentially being rejected by the target model at nearly every step.

The assistant had already ruled out several obvious suspects. The --speculative-num-steps 1 flag had been silently overriding --speculative-num-draft-tokens 16 to produce only 2 draft tokens — a configuration bug that wasted the draft model's capacity. Fixing that improved nothing. A standalone test of the draft model (isolated from SGLang's inference pipeline) showed 76.9% accuracy, matching the training metrics. The model could predict well. So why was SGLang's speculative engine getting such poor results?

The answer, the assistant suspected, lay in the wiring between the target model's hidden state outputs and the draft model's inputs. In EAGLE-3, the draft model does not receive raw token embeddings. Instead, it receives a concatenation of hidden states extracted from intermediate layers of the target model — so-called "auxiliary hidden states." These hidden states encode richer contextual information than the final-layer logits, allowing the draft model to make more informed predictions. But this only works if the right hidden states are captured, in the right order, with the right dimensionality.

Why This Message Was Written

The assistant wrote this message to answer a specific question: How does the DeepSeek V2 model (the backbone of KimiK25) implement the layers_to_capture mechanism that EAGLE-3 depends on?

The preceding messages in the conversation show the assistant tracing through SGLang's eagle_worker.py and llama_eagle3.py files, trying to understand how hidden states flow from the target model's forward pass into the draft model's fc (fully-connected) projection layer. The assistant had already discovered that the draft model's fc layer expects a 3 * hidden_size input (21504 for KimiK25's 7168-dimensional hidden states), but was receiving only hidden_size (7168). This meant the fc projection was being skipped entirely — the condition if hidden_states.shape[-1] != embeds.shape[-1] was evaluating to False because both were 7168.

The grep in this message was the logical next step: if the hidden states arriving at the draft model were already 7168-dimensional instead of 21504-dimensional, then somewhere upstream, the auxiliary hidden state capture mechanism was failing. The layers_to_capture list in the target model's forward pass is the mechanism that collects intermediate hidden states. By examining how DeepSeek V2 implements this, the assistant hoped to find either a configuration error (wrong layer IDs) or a code path that silently skipped the capture.

Input Knowledge Required

To understand the significance of this message, the reader needs to grasp several layers of technical context:

First, the architecture of EAGLE-3 speculative decoding. Unlike earlier draft models that predict tokens from scratch, EAGLE-3 uses a "feature-level" approach: it takes hidden states from intermediate layers of the target model (typically 3 layers), concatenates them into a 3 * hidden_size vector, projects this down to hidden_size via an fc layer, concatenates the result with token embeddings, and passes the combined representation through a lightweight transformer decoder. The draft model's predictions are conditioned on the target model's own internal representations, making it more accurate than a standalone language model.

Second, the KimiK25 model architecture. KimiK25 is a wrapper model that delegates to DeepSeek V2 (or more precisely, DeepSeek V3) for its core computation. The KimiK25 class has a set_eagle3_layers_to_capture method that forwards to self.language_model.set_eagle3_layers_to_capture, which in turn is the DeepSeek V2 method. Understanding this delegation chain is essential — a bug in any link breaks the chain.

Third, the SGLang speculative decoding infrastructure. SGLang's eagle_worker.py orchestrates the interaction between the target model (which runs the full forward pass) and the draft model (which runs a cheaper forward pass with speculation). The worker sets capture_hidden_mode = CaptureHiddenMode.FULL during target model extension, which triggers the collection of hidden states at layers specified by layers_to_capture. The collected states are then passed to forward_draft_extend as the hidden_states argument.

Fourth, the layer indexing convention. The assistant had previously discovered that SGLang applies an off-by-one shift: self.model.layers_to_capture = [val + 1 for val in layer_ids]. This means a layer_ids value of [2, 30, 58] becomes [3, 31, 59] in the actual capture list. The draft model's configuration stored the original layer IDs (matching the training pipeline's convention), but SGLang's internal convention added 1. This indexing mismatch was one of the bugs the assistant was tracking.

The Reasoning Process Visible in This Message

The assistant's reasoning in this message is compact but revealing. The phrase "Now let me check how the DeepSeek V2 model (which KimiK25 delegates to) handles layers_to_capture" shows a deliberate narrowing of scope. The assistant had been examining the draft model's forward pass, the eagle_worker.py orchestration, and the llama_eagle3.py model implementation. Each of these investigations had produced partial answers but not the root cause. The decision to look at DeepSeek V2 specifically reflects a hypothesis: that the target model's hidden state capture code might have a bug or a missing feature that prevents the auxiliary states from being collected.

The grep itself is targeted. The assistant searches for three patterns: layers_to_capture (the mechanism), captured_hidden (an alternative naming convention), and set_eagle3 (the configuration method). The head -20 limit shows the assistant wants a quick overview, not a deep read — this is a reconnaissance grep to find the relevant code locations before diving into details.

The results confirm the existence of the mechanism:

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message. The first is that the layers_to_capture mechanism is actually being triggered during inference. The grep shows the code exists, but it doesn't confirm that set_eagle3_layers_to_capture is being called with the right arguments, or that the KimiK25 wrapper properly delegates the call. The assistant addresses this in subsequent messages by checking the KimiK25 wrapper and the eagle_worker.py call sites.

The second assumption is that the layers_to_capture mechanism is the only path for auxiliary hidden state capture. In reality, the DeepSeek V2 forward pass has a complex structure with "normal" layers, "first_k_dense_replace" layers, and expert layers. The layers_to_capture check happens only in the normal layer loop, and only if the layer index falls within normal_start_layer to normal_end_layer. If the captured layers fall outside this range, they would be silently skipped.

The third assumption — implicit in the decision to grep DeepSeek V2 — is that the bug is in the target model's code rather than in the eagle_worker.py orchestration. This turns out to be partially correct: the target model's code works, but the worker's configuration of which layers to capture was using the wrong layer IDs. The assistant had previously set the draft model config to [2, 30, 58] (matching training), but the SGLang worker was using a different convention.

A potential mistake in this message is the narrow search scope. By searching only for layers_to_capture, captured_hidden, and set_eagle3, the assistant might miss alternative hidden state capture mechanisms. For instance, the DeepSeek V2 model also has a return_hidden_states mode and a CaptureHiddenMode.LAST mode used during verification. The grep doesn't cover these paths. However, given the assistant's goal — to understand the specific EAGLE-3 capture path — the focused search is appropriate.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Confirmation that layers_to_capture exists in DeepSeek V2. The mechanism is present at lines 2631, 2721, and 2963-2975. This means the target model can capture intermediate hidden states for EAGLE-3.
  2. The default layer selection formula. When set_eagle3_layers_to_capture() is called without arguments, it selects layers [2, num_layers // 2, num_layers - 3]. For a 60-layer model like KimiK25, this would be [2, 30, 57] (before the +1 shift, making it [3, 31, 58]). This differs from the draft model's configured [2, 30, 58] (which becomes [3, 31, 59] after shifting).
  3. The off-by-one indexing convention. The line self.model.layers_to_capture = [val + 1 for val in layer_ids] at line 2975 confirms that SGLang adds 1 to the layer IDs provided by the draft model config. This is because the draft model's layer numbering starts from 0 (the embedding layer), while the target model's internal layer numbering starts from 1 (the first transformer layer). This convention must be accounted for when configuring the draft model.
  4. A clear target for further investigation. With the mechanism confirmed to exist, the assistant can now focus on why it's not producing the expected output. The next step is to check whether set_eagle3_layers_to_capture is being called at all, and whether the KimiK25 wrapper properly delegates the call.

The Broader Debugging Arc

This message is message 4419 in a conversation that spans thousands of exchanges. It sits at the transition point between two debugging phases. In the preceding phase (messages ~4393-4418), the assistant focused on the draft model's internals — the fc projection, the hot_token_id mapping, the standalone test that confirmed the model could predict well. In the following phase (messages ~4420-4450), the assistant would trace the call chain from eagle_worker.py through model_runner.py to DeepseekV2.set_eagle3_layers_to_capture, eventually discovering that the KimiK25 wrapper was correctly delegating the call, but that the eagle_worker.py was using a hardcoded default that didn't match the draft model's configuration.

The specific insight that flows from this message is the val + 1 shift. In the next chunk (chunk 0 of segment 31), the assistant would discover that the training pipeline used cat([embed_output, layer3, layer31]) — taking the embedding output plus layers 3 and 31 — while SGLang was passing cat([layer3, layer31, layer59]) — three auxiliary hidden states without the embedding. This mismatch in the composition of the hidden state vector, not just the layer indices, was the root cause of the poor performance. The assistant would eventually modify deepseek_v2.py to capture the embedding output when layer_id=-1 is specified, and update the draft model config from [2, 30, 58] to [-1, 2, 30].

Conclusion

Message 4419 is a deceptively simple grep command that represents a critical pivot in a complex debugging session. It demonstrates a key skill in systems debugging: knowing when to stop examining the component that's failing and start examining the component that's supplying the failing component. The assistant had spent considerable effort understanding the draft model's forward pass, only to realize that the draft model was receiving the wrong inputs. By tracing the hidden state pipeline upstream to the target model's layers_to_capture mechanism, the assistant set the stage for discovering the fundamental wiring mismatch between training and inference.

The message also illustrates the importance of understanding indexing conventions, delegation chains, and default behaviors in large codebases. A single off-by-one shift, a missing delegation call, or a hardcoded default can silently corrupt the data flow between components, producing a system that runs without errors but delivers catastrophically wrong results. The assistant's systematic tracing — from draft model to worker to target model — is a textbook example of how to debug such failures.