The 252-in-32000 Problem: Diagnosing a Catastrophic Vocab Mapping Mismatch in EAGLE-3 Fine-Tuning

Introduction

In the high-stakes world of speculative decoding for large language models, fine-tuning an existing drafter to a new base model is supposed to be a shortcut to better performance. The reasoning is straightforward: if someone has already trained a draft model for a similar architecture, adapting it to your variant should be faster and cheaper than training from scratch. But this assumption rests on a critical precondition — that the two models speak the same token language. When they don't, the result is not merely suboptimal; it is catastrophic.

Message 4980 in this opencode session captures the exact moment when that precondition was tested and found to be spectacularly violated. The assistant, responding to the user's astute observation that fine-tuning loss was essentially random despite the drafter working in inference, executed a diagnostic script that revealed a stark truth: only 252 out of 32,000 draft-to-target token positions matched between the AQ-MedAI K2 drafter and the Kimi-K2.5 model's vocabulary mapping. This article examines that message in depth — the reasoning that motivated it, the diagnostic decisions it embodied, the assumptions it challenged, and the knowledge it produced.

The Context: A Fine-Tuning Mystery

To understand message 4980, we must first understand the puzzle that preceded it. The user had been engaged in a multi-session effort to improve Kimi-K2.5 inference throughput using EAGLE-3 speculative decoding. After training a from-scratch drafter that achieved 74.7% validation accuracy and ~60 tok/s, the user sought to leverage the AQ-MedAI K2 EAGLE-3 drafter — a publicly available checkpoint trained on the earlier Kimi-K2 model. The hope was that fine-tuning this existing drafter on K2.5 data would yield better performance than the from-scratch model.

Phase 0, a direct probe of the AQ-MedAI drafter on K2.5 without any fine-tuning, showed promising results: an accept length of approximately 1.5 tokens and 52 tok/s throughput. While worse than the from-scratch drafter (accept_len ~2.0, 60 tok/s), it was above 1.0, meaning the drafter was genuinely predicting tokens that the target model accepted. This suggested the K2 representations shared meaningful structure with K2.5 — a positive signal for fine-tuning.

Phase 1 launched the fine-tuning run. But the initial results were baffling. The loss was approximately 18-20 (cross-entropy over a 32,000-token vocabulary), and accuracy was 2-7% — essentially random performance. This made no sense: if the same drafter achieved accept_len ~1.5 in inference (meaning it was predicting above chance), why was its training loss indistinguishable from a randomly initialized model?

The user identified this contradiction in message 4975, asking directly: "Why is the loss/accuracy basically zero if the model was predicting ok in sglang? Are we passing it correct layers/hidden states etc correctly?" This question was the catalyst for message 4980. It forced a re-examination of the fundamental assumption underlying the fine-tuning approach.

The Diagnostic Hypothesis: Vocab Mapping Mismatch

The assistant's response in message 4976 showed the reasoning process. Three possible causes were identified:

  1. Vocab mapping mismatch: The AQ-MedAI drafter was trained with its own d2t (draft-to-target) and t2d (target-to-draft) mappings. The training data used the K2.5 mappings for target labels. If draft token ID 42 mapped to target token X in the AQ-MedAI system but target token Y in the K2.5 system, the loss computation would be fundamentally wrong — the model would be penalized for predicting the correct token under the wrong labeling scheme.
  2. Hidden state format differences: Perhaps the training pipeline fed hidden states in a different format than what SGLang provided at inference.
  3. Verifier lm_head masking: The training loss used the K2.5 verifier lm_head with the K2.5 t2d mask, but the AQ-MedAI lm_head was trained to match a different mask. The assistant correctly identified the vocab mapping mismatch as the most likely culprit and designed a diagnostic script to test it. This decision reflected a deep understanding of how EAGLE-3 training works: the draft model's lm_head produces logits over the draft vocabulary (a subset of the full target vocabulary), and the d2t mapping converts draft token IDs to target token IDs for loss computation. If the mappings differ between training and inference, the loss computation becomes misaligned with what the model actually learned.

Message 4980: The Diagnostic Execution

Message 4980 is the execution of that diagnostic. The assistant SCPs a Python script to the remote machine and runs it. The script loads both the AQ-MedAI and K2.5 vocab mappings, computes the "hot token IDs" (the target token that each draft position maps to), and compares them.

The output is devastatingly clear:

AQ-MedAI hot_token_ids (first 20): [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Our hot_token_ids (first 20): [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

At first glance, the first 20 entries match perfectly — both mappings start with a sequential mapping from draft position 0 to target token 0, draft position 1 to target token 1, and so on. This is the identity mapping for the first 20 positions, which is typical when the draft vocabulary is a prefix of the full vocabulary sorted by token ID.

But the deeper comparison reveals the catastrophe:

Positional match (same draft_id -> same target_id): 252 / 32000

Only 252 out of 32,000 draft positions map to the same target token in both mappings. That is 0.79% agreement. The remaining 31,748 positions are effectively scrambled — draft position 42 in the AQ-MedAI system maps to a completely different target token than draft position 42 in the K2.5 system.

The overlap analysis provides more detail:

AQ-MedAI covers 32000 target tokens
Ours covers 32000 target tokens
Overlap: 25750 target tokens

AQ-MedAI draft tokens that exist in our draft vocab: 25750
AQ-MedAI draft tokens NOT in our draft vocab: 6250

So 25,750 target tokens are covered by both mappings (they appear somewhere in both draft vocabularies), but only 252 of them are at the same position. The remaining 25,498 overlapping tokens are at different positions. And 6,250 target tokens covered by the AQ-MedAI drafter don't appear in the K2.5 draft vocabulary at all.

Why This Matters: The Training Catastrophe Explained

This mismatch explains the random training loss perfectly. Consider what happens during fine-tuning:

  1. The training script loads the AQ-MedAI drafter weights, including the lm_head layer.
  2. The lm_head has 32,000 output neurons, each corresponding to a draft vocabulary position.
  3. During training, the model produces logits over these 32,000 positions.
  4. The loss is computed by comparing these logits against the target token ID of the next token, converted to a draft position using the K2.5 d2t mapping.
  5. But the lm_head was trained so that neuron 42 predicts the target token that AQ-MedAI's draft position 42 maps to — not the target token that K2.5's draft position 42 maps to. So when the training loss says "neuron 42 should fire for target token X" (where X is determined by K2.5's d2t), the model's lm_head was actually trained to fire neuron 42 for target token Y (where Y is determined by AQ-MedAI's d2t). Since X and Y are different for 31,748 out of 32,000 positions, the gradient signals are essentially random noise. The model cannot learn because the targets are misaligned with its output structure. This is fundamentally different from the inference scenario. In SGLang, the drafter receives hidden states from the K2.5 base model and produces logits over its own 32,000-position draft vocabulary. The SGLang speculative decoding engine then uses the drafter's own d2t mapping to convert draft tokens to target tokens for verification. The drafter never needs to know about the K2.5 d2t mapping during inference — it only needs to produce good logits over its own vocabulary, and the engine handles the translation. This is why accept_len ~1.5 was achievable: the drafter's internal representations and output distribution were coherent within its own vocabulary system. But during training, the loss computation uses the K2.5 d2t mapping to create labels, breaking the coherence. The drafter is being asked to predict tokens under a labeling scheme it was never trained on.

Assumptions and Their Violations

This episode reveals several assumptions that were made and subsequently violated:

Assumption 1: Vocabulary mappings are compatible across model variants. The AQ-MedAI K2 drafter and the Kimi-K2.5 model both use a 32,000-token draft vocabulary, and both claim to cover the full 163,840-token vocabulary of the DeepSeek-based architecture. But "covering the same tokens" does not mean "assigning the same draft positions to the same tokens." The two mappings are essentially independent permutations of the token space, with only 252 positions accidentally coinciding (likely the first 252 tokens that are common prefixes).

Assumption 2: Fine-tuning from related weights is always beneficial. The conventional wisdom in transfer learning is that starting from related weights provides a better initialization than random. But this assumes the weight structure is compatible with the new task's output space. When the output space is permuted, the initialization becomes actively harmful — the model must unlearn its old output mapping before it can learn the new one.

Assumption 3: Inference success implies training compatibility. The accept_len ~1.5 result was interpreted as evidence that the K2 representations were "correlated" with K2.5. While true at the hidden state level, this correlation did not extend to the output vocabulary mapping. The drafter's internal representations of linguistic structure were useful, but its output layer was wired to a different labeling convention.

Input Knowledge Required

To fully understand message 4980, several pieces of domain knowledge are necessary:

EAGLE-3 Architecture: EAGLE-3 is a speculative decoding framework where a lightweight "draft" model predicts multiple future tokens in parallel, which are then verified by the full "target" model. The draft model has its own vocabulary (a subset of the target model's vocabulary) and uses d2t (draft-to-target) and t2d (target-to-draft) mappings to translate between the two token spaces.

Vocab Mappings: In EAGLE-3, the draft vocabulary is a selection of the most frequent or important tokens from the target vocabulary. The d2t mapping is a tensor where position i contains the target token ID that draft token i represents. The t2d mapping is a boolean mask over the target vocabulary indicating which tokens are included in the draft vocabulary. These mappings are model-specific and depend on the token frequency distribution of the training data.

Fine-tuning Mechanics: When fine-tuning a drafter, the training script loads the existing weights and continues training on new data. The loss is computed by comparing the drafter's logits (over its draft vocabulary) against the ground-truth next token (converted to a draft position via the d2t mapping). If the d2t mapping used during training differs from the one used during the original training, the weight-to-position correspondence is broken.

The DeepSeek/Kimi Token System: The Kimi models use a large vocabulary (163,840 tokens) derived from the DeepSeek architecture. The draft vocabulary of 32,000 tokens is a subset. Different model variants (K2 vs K2.5) may select different subsets or assign different positions to the same tokens based on their training data distributions.

Output Knowledge Created

Message 4980 produced several concrete pieces of knowledge:

  1. Quantified mismatch severity: The exact degree of positional mismatch (252/32,000) was measured, providing a precise answer to why fine-tuning failed.
  2. Overlap statistics: 25,750 target tokens are covered by both mappings, with 6,250 tokens unique to AQ-MedAI. This means the draft vocabularies are ~80% overlapping in terms of token coverage, but the position assignment within that overlap is nearly random.
  3. Root cause identification: The diagnostic confirmed that the vocab mapping mismatch, not hidden state format or other issues, was the primary cause of the training failure.
  4. Incompatibility proof: The result proved that fine-tuning the AQ-MedAI K2 drafter on K2.5 data, as configured, was fundamentally impossible without either (a) remapping the lm_head weights to align with the K2.5 d2t mapping, or (b) retraining the lm_head from scratch while freezing the earlier layers.

The Thinking Process

The assistant's thinking process in the messages leading up to 4980 is worth examining. When the user questioned the random loss, the assistant did not dismiss the observation or attribute it to insufficient training time. Instead, it immediately recognized the contradiction and formulated three specific hypotheses. The choice to investigate vocab mapping first was strategic — it was the hypothesis that, if true, would explain the entire phenomenon without invoking more complex issues like hidden state format differences.

The diagnostic script itself reveals careful thinking. It doesn't just compare the d2t tensors directly (which would show they're different but not explain why). Instead, it computes the "hot token IDs" — the actual target tokens that each draft position maps to — and compares them positionally. This directly answers the question: "If the model outputs logit at position 42, what target token is it being trained to predict, and does that match what it was originally trained to predict?"

The script also computes the overlap in target token coverage and the reverse mapping (target token → our draft position), which provides a complete picture of the mismatch. This thoroughness suggests the assistant anticipated follow-up questions and wanted to provide all relevant statistics in one shot.

Conclusion

Message 4980 is a textbook example of diagnostic debugging in machine learning systems. It identifies a silent but catastrophic assumption — that vocabulary mappings are interchangeable across model variants — and quantifies its impact with precision. The 252-in-32,000 mismatch is not just a number; it is the explanation for why a seemingly promising fine-tuning approach produced random results, and it is the foundation for all subsequent decisions about whether to proceed with the AQ-MedAI drafter or abandon the approach.

For the broader narrative of this coding session, message 4980 represents a turning point. It closed the door on the "quick fine-tuning win" approach and forced a more fundamental reconsideration of strategy. The knowledge produced here — that vocab mapping compatibility cannot be assumed even between closely related model variants — is a lesson that extends well beyond this specific session, serving as a cautionary tale for anyone attempting to transfer speculative decoding drafters between model checkpoints.