The Zero Percent Accuracy Moment: Debugging a Silent Vocab Mapping Mismatch in EAGLE-3 Speculative Decoding

In the high-stakes world of large language model deployment, few moments are as jarring as seeing 0.0% accuracy on training data. This is exactly what confronted the assistant in message [msg 4389] of an extended debugging session focused on deploying an EAGLE-3 draft model for the Kimi-K2.5 architecture using SGLang's speculative decoding engine. The message captures a pivotal investigative turn: the assistant has just run a standalone test that bypasses the entire SGLang inference stack, loading the draft model directly and feeding it hidden states from the training pipeline, only to discover that the model predicts nothing correctly — even on the very data it was trained on.

The Road to Zero

To understand why this message matters, we must first appreciate the debugging journey that led to it. The team had trained an EAGLE-3 draft model on 100,000 samples, achieving a respectable 74.7% validation accuracy during training. Yet when deployed within SGLang's speculative decoding framework, the results were bafflingly poor: throughput of approximately 46–56 tokens per second against a baseline of 90 tok/s without speculation, and an acceptance length of merely 1.6–1.9 tokens out of 16 drafted tokens. The draft model appeared to be failing in production despite its strong training metrics.

The assistant had already identified and fixed one significant issue: the --speculative-num-steps 1 flag was silently overriding --speculative-num-draft-tokens 16 to just 2 draft tokens due to a SGLang constraint when topk=1. After correcting this to --speculative-num-steps 15, performance actually worsened to 46.7 tok/s, revealing that the draft model itself was the bottleneck — the overhead of 15 sequential draft model forwards was not being offset by meaningful token acceptance.

At the user's suggestion ("Try with a training sample and see if accept rate looks correct. Possible we didn't wire in the model correctly," [msg 4374]), the assistant pivoted to writing a standalone test script. This script would load the draft model's weights directly from their safetensors file, bypass SGLang entirely, and run the EAGLE-3 forward pass using actual hidden states captured from the training data. If the draft model performed well on training data in isolation, the problem was in SGLang's integration. If it performed poorly, the model weights themselves were suspect.

The Moment of Discovery

Message [msg 4389] opens with the result of that standalone test: 0.0% accuracy. The assistant's reasoning text captures the shock and immediate analytical pivot:

0.0% accuracy — even on training data, without the transformer layer. This is a critical finding. But wait — this is without the transformer layer. However, the embedding norm is 0.60 while fc_out norm is 74.39, so the embedding contribution is negligible. The fc projection itself should still give reasonable logits if the weights are correct.

This passage reveals the assistant's rapid mental model-building. The test was run "without the transformer layer" — meaning only the EAGLE-3 feature compression (fc) layer and the lm_head were used, skipping the midlayer transformer that normally refines the draft hidden states. Even so, the fc projection alone should produce reasonable logit distributions if the weights are correctly trained. The norms suggest the fc output dominates (norm 74.39 vs embedding norm 0.60), so the embedding contribution is negligible — yet the accuracy is still zero. Something deeper is wrong.

The assistant then notices a suspicious pattern in the d2t (draft-to-target) vocabulary mapping tensor:

But the d2t mapping looks wrong — d2t[:10] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] means the first 10 draft tokens ALL map to target token 0. That's clearly broken.

This is the critical insight that drives the rest of the message. The d2t tensor is supposed to map draft vocabulary IDs to target vocabulary IDs. If the first 10 entries are all zero, then draft tokens 0 through 9 would all map to target token 0 — a clearly degenerate mapping that would explain 0% accuracy. The assistant immediately decides to investigate by comparing the d2t tensor stored in the model's safetensors file against the original vocab mapping used during training.

The Investigation Unfolds

The assistant dispatches a bash command that loads both the model's safetensors and the original vocab mapping files, printing detailed diagnostics:

w = load_file("/data/eagle3/output_100k_sglang/4/model.safetensors")
d2t_model = w["d2t"]
t2d_model = w["t2d"]
# ... then compare with original
d2t_orig = torch.load("/data/eagle3/synth_10k_sglang/vocab_mapping/d2t.pt")
t2d_orig = torch.load("/data/eagle3/synth_10k_sglang/vocab_mapping/t2d.pt")

The output reveals that the model's d2t tensor has shape [32000] (matching the draft vocabulary size), with the first 20 entries all zero, the last 20 entries containing large values like 131239 through 131608, and a minimum of 0 and maximum of 131608. The t2d tensor has shape [163840] (matching the target vocabulary size) with exactly 32,000 True entries — one for each draft token.

Crucially, the output also shows the original vocab mapping's d2t tensor, though the output is truncated in the message. The key finding at this stage is that the model's d2t and the original d2t appear to be identical tensors — the mapping was preserved correctly through training and export. The problem is not data corruption or a training bug.

But the assistant hasn't yet resolved the interpretation question. The first 20 entries being zero could mean that draft tokens 0–19 all map to target token 0 (direct mapping interpretation), or it could mean something else entirely. The message ends with the investigation in progress — the data has been gathered, but the conclusion is pending.

The Hidden Assumption

The most interesting aspect of this message is the assumption embedded in the assistant's reasoning: that d2t represents a direct mapping from draft IDs to target IDs. Under this assumption, d2t[i] = target_id for draft token i. If the first 10 entries are 0, then draft tokens 0–9 all map to target token 0, which is indeed "clearly broken."

However, as the very next message ([msg 4390]) reveals, this assumption is incorrect. The d2t tensor actually stores offsets, not direct mappings. The correct formula is target_id = draft_id + d2t[draft_id]. Under this interpretation, d2t[0] = 0 means draft token 0 maps to target token 0 (0 + 0), d2t[1] = 0 means draft token 1 maps to target token 1 (1 + 0), and so on. The first 10 entries being zero simply means the first 10 draft tokens map to the first 10 target tokens with no offset — a perfectly reasonable mapping for the beginning of the vocabulary.

This is a classic debugging pitfall: an assumption about data format that seems obvious but is wrong. The assistant's instinct to verify the mapping by comparing against the original was sound, but the interpretation of the comparison was still colored by the initial assumption. The resolution comes only in the follow-up message, where additional diagnostics confirm the offset interpretation.

Input and Output Knowledge

To fully understand this message, one needs significant context about the EAGLE-3 architecture and the specific deployment setup. The EAGLE-3 draft model uses a compressed vocabulary — 32,000 tokens instead of the target model's 163,840 — requiring a mapping between the two vocabularies. The d2t and t2d tensors encode this mapping. The training pipeline captures hidden states from four layers of the target model (embedding output, layer 3, layer 31, and layer 59) and uses these as conditioning input for the draft model's predictions.

The message also assumes familiarity with SGLang's speculative decoding infrastructure, including how draft tokens are generated and verified against the target model. The standalone test script referenced in the message (written in [msg 4379] and [msg 4386]) loads the draft model's weights, constructs the EAGLE-3 forward pass manually, and compares predicted tokens against ground truth from the training data.

The output knowledge created by this message is nuanced. On one hand, the assistant has confirmed that the d2t tensor in the deployed model matches the original vocab mapping — ruling out a class of data corruption or export bugs. On the other hand, the 0.0% accuracy result remains unexplained, and the d2t interpretation question is left open. The message advances the debugging process by narrowing the search space: the problem is not a corrupted mapping, but rather either (a) the mapping is being interpreted incorrectly (as turns out to be the case), or (b) there is a deeper issue with how the draft model's forward pass is constructed in the standalone test.

The Broader Significance

This message exemplifies a critical pattern in ML engineering debugging: the moment when a surprising experimental result forces a fundamental re-examination of assumptions. The 0.0% accuracy on training data was a "red alert" signal that contradicted everything the team believed about their model. Rather than panicking or jumping to conclusions, the assistant methodically gathered evidence — comparing model weights against originals, examining tensor shapes and values, and reasoning about the architecture.

The message also illustrates the value of the standalone test as a debugging technique. By isolating the draft model from the complex SGLang inference stack, the assistant eliminated an entire class of potential integration bugs. The problem was narrowed to either the model weights themselves or the standalone test's implementation. As it turned out, the standalone test was using the wrong hidden state inputs — a different issue that would be discovered in subsequent messages ([chunk 31.0]). But the d2t investigation in this message was a necessary detour that ruled out one hypothesis and built understanding of the vocab mapping format.

In the end, the d2t mapping was not the culprit. The real issue was a hidden state input format mismatch: the training pipeline concatenated [embed_output, layer3, layer31] as input to the fc layer, while SGLang was passing [layer3, layer31, layer59] — the three auxiliary hidden states without the embedding output. This message's investigation into the d2t mapping, while ultimately a red herring, was an essential part of the systematic debugging process that led to the real fix.