The 34.1% Mystery: Debugging an EAGLE-3 Draft Model's Silent Wiring Failure

In the middle of a complex debugging session spanning speculative decoding performance, a single message captures the precise moment when a developer realizes something fundamental is wrong. Message [msg 4454] in this opencode session is a turning point — a diagnostic pivot that transforms a vague suspicion ("our draft model isn't predicting well") into a concrete, testable hypothesis about a wiring mismatch between training and inference.

The message itself is deceptively simple: the assistant has just run a standalone test of the EAGLE-3 draft model against training data and received an accuracy of 34.1%, far below the 74.7% reported during training. The assistant's response is a burst of analytical reasoning, followed by a diagnostic bash command to inspect the hidden state tensors. But what makes this message significant is not the data it produces — it's the thinking process it reveals, the assumptions it questions, and the investigative path it opens.

The Context: A Speculative Decoding Pipeline That Should Work

To understand why this message matters, we need to understand the broader context. The developer has been building an EAGLE-3 speculative decoding system for the Kimi-K2.5 large language model. Speculative decoding is a technique where a small "draft" model proposes candidate tokens that a larger "target" model then verifies in parallel, potentially accelerating inference without degrading quality. EAGLE-3 is a particular architecture where the draft model takes as input not just the target model's final hidden states, but intermediate hidden states from multiple layers — auxiliary hidden states that provide richer contextual information.

The pipeline had been carefully constructed. Training data was generated by running the Kimi-K2.5 target model on 100K prompts, capturing both the input tokens and the hidden states at specific layers. The draft model was trained on this data and achieved a validation accuracy of 74.7% — a promising result. The model was then deployed into SGLang, a high-performance inference engine, configured for speculative decoding with the EAGLE-3 algorithm.

Yet when the developer benchmarked the deployed system, the results were baffling: the speculative decoding server achieved only ~56.8 tokens per second, compared to a baseline of 90.0 tok/s without speculation. The acceptance length — the average number of draft tokens accepted per step — was around 1.6 out of 16 proposed tokens. This was catastrophically low. A well-functioning draft model should achieve acceptance rates of 70-90%, not 10%.

The Initial Investigation: Configuration Errors

The developer's first hypothesis was a configuration problem. Earlier investigation (documented in the chunk summary) had revealed that --speculative-num-steps 1 was silently overriding --speculative-num-draft-tokens 16, limiting the system to just 2 draft tokens due to an internal SGLang constraint when topk=1. Fixing this to --speculative-num-steps 15 actually made performance worse, dropping to 46.7 tok/s with an accept length of only ~1.9. This was a crucial clue: even with more draft tokens available, the draft model was barely predicting any of them correctly.

This led to the second hypothesis: the draft model itself was not producing good predictions. But the training accuracy of 74.7% contradicted this. How could a model that predicts the next token correctly 74.7% of the time on validation data achieve an acceptance rate of only ~10% in deployment?

The Standalone Test: Isolating the Draft Model

To answer this question, the developer wrote a standalone test script (test_drafter_standalone.py) that loaded the draft model weights and training data directly, bypassing SGLang entirely. This was a critical methodological decision: by eliminating the entire inference engine from the equation, any discrepancy between training accuracy and test accuracy would have to be caused by either (a) a bug in the standalone test's forward pass implementation, or (b) a mismatch between the data format used during training and the data format used during testing.

The standalone test ran and produced the shocking result: 34.1% accuracy. This was the moment captured in message [msg 4454].

The Reasoning: Three Hypotheses

The assistant's reasoning in this message is a textbook example of diagnostic thinking. It immediately identifies the gap — 34.1% vs 74.7% — and proposes three possible explanations:

  1. The manual forward pass doesn't match the training forward pass exactly. This is the most obvious suspect: perhaps the standalone test implements some component differently from the training code. The RoPE (Rotary Position Embedding) implementation is a particularly likely culprit, as the assistant had already encountered a RoPE dimension mismatch in an earlier iteration of the test (see [msg 4452]).
  2. The RoPE implementation differs between training and the manual test. The training code uses the speculators library, which internally uses HuggingFace's LlamaDecoderLayer. The standalone test uses a manual forward pass. If the RoPE computation differs even slightly — different frequency computation, different application logic — the predictions would diverge.
  3. The hidden states concatenation order is wrong. This is the most subtle and interesting hypothesis. The EAGLE-3 draft model takes as input a concatenation of multiple hidden state tensors from different layers of the target model. If the order in which these tensors are concatenated differs between training and inference, the model would receive its inputs in a completely different arrangement than what it was trained on, causing catastrophic performance degradation. The assistant's phrasing is telling: "Let me check if the order matters — maybe we need to verify which hidden state corresponds to which layer." This reveals an emerging suspicion that the problem is not in how the model computes, but in what data it receives.

The Diagnostic Command: Inspecting Hidden State Norms

The assistant then executes a bash command that loads a training data sample and prints the norms of each hidden state tensor. This is a clever diagnostic technique: by examining the statistical properties of each hidden state, the developer can identify which tensor corresponds to which layer.

The output is revelatory:

hs[0]: shape=torch.Size([1281, 7168]), norm_mean=1.0383, norm_std=0.2234
  first tok norm: 1.6597
  last tok norm:  0.9806
hs[1]: shape=torch.Size([1281, 7168]), norm_mean=37.1650, norm_std=92.0252
  first tok norm: 3321.8455
  last tok norm:  29.9057
hs[2]: shape=torch.Size([1281, 7168]), norm_mean=174.3977, norm_std=99.4064
  first tok norm: 2599.4795
  last tok norm:  121.7965
hs[3]: shape=torch.Size([1281, 7168]), norm_mean=111.3359, norm_std=7.9431
  first tok norm: 61.5139
  last tok norm:...

The first hidden state (hs[0]) has a mean norm of approximately 1.0, with low variance — this is characteristic of an embedding output, where tokens are mapped to a learned embedding space with unit-like norms. The subsequent hidden states have much larger norms (37, 174, 111), which is characteristic of transformer layer outputs where activations can grow through the network.

This is the crucial insight. The training data contains 4 hidden states: the embedding output plus the outputs of three intermediate layers (layers 3, 31, and 59 of the 60-layer Kimi-K2.5 model). The first tensor is clearly the embedding; the others are layer outputs.

What the Assistant Didn't Yet Know

At this point in the conversation, the assistant has not yet made the full connection. The message ends with the norm output truncated. But the following messages ([msg 4455] through [msg 4463]) complete the picture.

The critical discovery comes when the assistant examines the training data pipeline. In message [msg 4458], the assistant reads the standardize_data_v1 function in the speculators training library:

"hidden_states": torch.cat(data["hidden_states"][:-1], dim=-1),  # all but last
"verifier_last_hidden_states": data["hidden_states"][-1],         # last one

This means the training pipeline takes the first 3 of the 4 hidden states — [embed_output, layer3, layer31] — concatenates them along the last dimension to form a 21504-dimensional input, and feeds this to the draft model's fully-connected (fc) layer. The last hidden state (layer59) is stored separately as verifier_last_hidden_states for the verification step.

But SGLang's inference pipeline was passing the last 3 hidden states — [layer3, layer31, layer59] — to the draft model. This is a completely different input. The fc layer was trained to expect the embedding output as its first third of the input, but at inference time it was receiving a layer output instead.

This explains the 34.1% accuracy perfectly: the model is being fed data in a format it was never trained on. The weights learned to map cat([embed, layer3, layer31]) to the next token, but at inference it's receiving cat([layer3, layer31, layer59]) — a different distribution entirely.

Assumptions and Mistakes

This debugging episode reveals several assumptions that turned out to be incorrect:

  1. The assumption that training and inference use the same data format. The developer (reasonably) assumed that since the same "EAGLE-3" algorithm was specified in both training and inference, the data pipeline would be consistent. But the training library and the inference engine (SGLang) had independently implemented the data format, and they diverged in which hidden states were included.
  2. The assumption that 4 hidden states meant 3 auxiliary states. The EAGLE-3 architecture typically captures hidden states from 3 intermediate layers. The training data contained 4 tensors, which the developer initially interpreted as 3 auxiliary states plus something else. In reality, the 4 tensors were [embed, aux1, aux2, aux3], and the training code used [:-1] to select the first 3.
  3. The assumption that the 74.7% training accuracy was trustworthy. While the training accuracy was technically correct for the training data format, it was misleading because it didn't validate that the inference pipeline would provide data in the same format.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message produces several pieces of knowledge:

  1. Empirical evidence that the draft model performs at 34.1% in the standalone test, confirming that the problem is not in SGLang's inference engine but in the data pipeline or model wiring.
  2. Statistical characterization of the hidden state tensors, showing that hs[0] has norms ~1.0 (embedding output) while hs[1] through hs[3] have much larger norms (layer outputs).
  3. A refined hypothesis space: The three proposed explanations (forward pass mismatch, RoPE difference, concatenation order) narrow the investigation to specific components.
  4. The insight that hidden state identity can be verified through norm analysis, providing a methodology for future debugging.

The Thinking Process

What's most striking about this message is the assistant's disciplined reasoning process. Rather than jumping to conclusions or making ad-hoc fixes, it:

  1. Quantifies the gap: 34.1% vs 74.7% — a precise measurement of the discrepancy.
  2. Generates multiple hypotheses: Three distinct possible explanations, each testable.
  3. Prioritizes the most informative diagnostic: Checking hidden state norms can simultaneously validate or eliminate hypothesis #3 (concatenation order) and provide clues about hypotheses #1 and #2 (forward pass correctness).
  4. Executes the diagnostic efficiently: A single bash command that loads a data sample and prints norms. This is the hallmark of effective debugging: not just fixing bugs, but systematically narrowing the space of possible causes until the root cause is isolated.

The Broader Lesson

The bug ultimately discovered — a mismatch between training and inference data formats — is a classic failure mode in machine learning systems. It's particularly insidious because both the training accuracy (74.7%) and the inference pipeline appear correct in isolation. The error only manifests when the two are compared, and even then, the symptoms (low throughput) are easily attributed to other causes (configuration errors, model quality, hardware limitations).

The lesson is that in complex ML systems, data format consistency between training and inference is not guaranteed by architectural alignment alone. The same algorithm name ("EAGLE-3") can mean different things in different codebases. The only way to catch these mismatches is through direct, end-to-end validation — exactly what the standalone test provided.

Message [msg 4454] captures the moment when the developer realizes that the numbers don't add up and begins the methodical process of finding out why. It's a small but crucial step in a debugging journey that would ultimately reveal a fundamental wiring failure — and demonstrate the power of systematic diagnostic thinking.