The 252-in-32,000 Problem: How a Vocab Mapping Mismatch Nearly Derailed EAGLE-3 Fine-Tuning

In the high-stakes world of large language model inference optimization, few techniques offer as much promise—and as many hidden pitfalls—as speculative decoding. The idea is elegant: use a small "draft" model to predict tokens cheaply, then have the large "target" model verify them in parallel, achieving throughput gains without sacrificing quality. But as this session demonstrates, the gap between theoretical elegance and practical deployment is filled with subtle, data-level mismatches that can turn promising results into random noise.

This article examines a single message in an opencode coding session—message 4981—where an AI assistant diagnosed a critical vocab mapping mismatch that was causing a fine-tuning run to produce essentially random loss. The discovery, that only 252 out of 32,000 draft-to-target token positions matched between two related models, explains why a drafter that worked reasonably well in inference (accept_len ~1.5) appeared to be learning nothing during training (loss ~18-20, accuracy ~2-7%).

The Scene: Fine-Tuning a Pre-Trained Drafter

The session's broader goal was to improve inference throughput for the Kimi-K2.5 language model using EAGLE-3 speculative decoding. The team had already trained a from-scratch EAGLE-3 drafter that achieved respectable results, but they were exploring whether a pre-trained drafter from AQ-MedAI (trained on the earlier Kimi-K2 model) could be fine-tuned to K2.5 more efficiently than starting from scratch.

Phase 0 had been encouraging: when the AQ-MedAI K2 drafter was deployed directly with K2.5 in SGLang, it achieved an accept_len of approximately 1.5 and 52 tok/s. While this was worse than the from-scratch drafter (accept_len ~2.0, 60 tok/s) and significantly worse than the baseline without speculation (82 tok/s), it was still above 1.0—meaning the drafter was genuinely predicting tokens that the target model accepted. The representations shared meaningful structure across model versions.

This positive signal justified Phase 1: fine-tuning the AQ-MedAI weights on the existing 37K K2.5 training samples. The assistant launched training with a conservative learning rate (5e-5, lower than the 3e-5 used from scratch) to gently adapt the K2 representations without destroying them.

The Alarm: Random Loss from a Working Model

When the assistant checked the training logs after a few minutes, the numbers were alarming:

loss_0: 17.8, full_acc_0: 0.06, loss_1: 19.6, full_acc_1: 0.0

A cross-entropy loss of ~18-20 over a 32,000-token vocabulary, with accuracy hovering near zero, is essentially random performance. A model that predicts uniformly would achieve a loss of approximately ln(32000) ≈ 10.4, so a loss of 18-20 is actually worse than random—the model is confidently wrong.

The user immediately spotted the contradiction ([msg 4975]):

Why is the loss/accuracy basically zero if the model was predicting ok in sglang? Are we passing it correct layers/hidden states etc correctly?

This was the right question. If the same drafter weights achieved accept_len ~1.5 in SGLang inference, they were clearly capable of above-chance prediction. Something in the training pipeline was presenting fundamentally different data to the model.

The Investigation: Three Hypotheses

The assistant responded by killing the training job and launching a systematic investigation ([msg 4976]). Three possible causes were identified:

  1. Vocab mapping mismatch: The training data used the user's t2d/d2t mappings for target labels, but the AQ-MedAI lm_head was trained with a different mapping. Draft token 42 might map to target token X in one system and target token Y in the other.
  2. Hidden state format: The standardize_data_v1 function in training might feed hidden states differently than SGLang does at inference.
  3. Verifier lm_head masking: The training loss is computed using a verifier_lm_head derived from K2.5's masked lm_head with the user's t2d mask, but AQ-MedAI's lm_head was trained against a different mask. The assistant prioritized hypothesis #1 as the most likely culprit and wrote a diagnostic script to compare the vocab mappings directly.

The Script and the Shell Escape

The first attempt to run the diagnostic script failed due to a classic Unix problem: shell escaping. The assistant tried to inline a Python script within a complex SSH command, but the nested quoting tripped over zsh's parser ([msg 4978]):

zsh:32: parse error near `)'

This is a familiar frustration for anyone who works with remote ML infrastructure. The assistant pivoted gracefully, writing the script to a file on the local machine and SCP'ing it to the remote server ([msg 4979]). This pattern—attempting an inline command, hitting shell escaping issues, then falling back to a file-based approach—is a recurring theme in infrastructure-heavy ML work.

The Discovery: 252 Out of 32,000

The script ran successfully on the remote machine and produced the output that forms the backdrop to message 4981 ([msg 4980]):

AQ-MedAI hot_token_ids (first 20): [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Our hot_token_ids (first 20): [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

AQ-MedAI covers 32000 target tokens
Ours covers 32000 target tokens
Overlap: 25750 target tokens

Positional match (same draft_id -> same target_id): 252 / 32000

The first 20 positions matched—these are likely special tokens (BOS, EOS, padding) and byte-level tokens that are universal across tokenizers. But after that, the orderings diverged completely. Only 252 out of 32,000 positions had the same draft-to-target mapping.

This is the critical insight: both systems use a "draft vocabulary" of 32,000 positions, where each position corresponds to a specific target token ID. But the assignment of which target token goes to which draft position is different. When the fine-tuning script loads AQ-MedAI's lm_head weights, it places them at positions 0-31,999. The training labels, however, are computed using the user's mapping—so position 500 in the lm_head predicts target token X (AQ-MedAI's mapping), but the label says position 500 should predict target token Y (the user's mapping). The result is pure label scrambling.

Message 4981: The Diagnosis

This brings us to the subject message itself. The assistant writes:

Found it. Only 252 out of 32,000 draft vocab positions match between AQ-MedAI's mapping and ours. The first ~50 are shared (low token IDs, likely special tokens / bytes), but after that the orderings diverge completely.

>

The problem: When we fine-tune from AQ-MedAI weights, we're loading their lm_head which was trained to output logits for their draft-to-target mapping, but the training labels are computed using our t2d/d2t. The lm_head predicts "token A" at position 500 (AQ-MedAI's meaning), but the label says position 500 means something else (our meaning). Pure label scrambling → random loss.

>

The fix: We need to either: - Option A: Remap AQ-MedAI's lm_head weights to our vocab mapping (permute rows to align) - Option B: Use AQ-MedAI's t2d/d2t during training (but then our from-scratch drafter wouldn't be compatible for comparison)

>

Option A is cleaner. Let me build the permutation matrix and remap the lm_head.

The message is concise but packs a significant analytical payload. Let's unpack what it reveals.

The Reasoning Process

The assistant's thinking demonstrates several hallmarks of effective ML debugging:

Contradiction-driven investigation: The core insight was that the drafter worked in inference (accept_len > 1.0) but failed in training (random loss). This contradiction meant the problem couldn't be with the weights themselves—it had to be in how the training pipeline presented data versus how SGLang presented data at inference.

Hypothesis prioritization: Three possible causes were identified, but the assistant correctly prioritized the vocab mapping mismatch as the most likely. This was a good judgment call: hidden state format differences would typically cause a different kind of failure (e.g., NaN loss or poor convergence from the start), while the verifier lm_head masking would affect both training and inference equally. The vocab mapping mismatch specifically explains why training fails while inference succeeds.

Quantitative confirmation: Rather than speculating, the assistant wrote a concrete diagnostic script that computed exact overlap statistics. The number 252 out of 32,000 is unambiguous—it's not a "maybe" problem, it's a definitive structural mismatch.

Clean solution design: Option A (remapping the lm_head via a permutation matrix) is the right engineering choice. It preserves the user's existing training pipeline and data format while correcting the weight placement. Option B would require changing the data pipeline, which would break comparability with the from-scratch drafter and introduce more moving parts.

Assumptions and Their Validity

The assistant made several assumptions in this analysis:

  1. That the lm_head is the primary source of the mismatch: This is almost certainly correct. The lm_head is the final linear layer that maps draft representations to vocabulary logits. If the position-to-token mapping is wrong, the loss will be random regardless of how well the transformer layers are adapted.
  2. That the transformer layers (midlayer weights) don't need remapping: This is a reasonable assumption. The transformer layers operate on hidden states, which are in the same representational space regardless of which vocab mapping is used downstream. The lm_head is the only layer that directly interacts with token IDs.
  3. That Option A is cleaner than Option B: This depends on the implementation complexity of building the permutation matrix. If some target tokens in AQ-MedAI's mapping don't exist in the user's mapping (6,250 were missing), those lm_head rows would need to be handled specially—either zeroed out or randomly reinitialized.
  4. That the first ~50 shared positions are special tokens: This is a reasonable inference but wasn't verified. It's possible that some early vocabulary positions happen to align by coincidence, though the pattern of the first 20 matching exactly and then diverging strongly suggests special/control tokens.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. The quantitative mismatch statistic: 252/32,000 positional matches, 25,750/32,000 target token overlap (meaning 6,250 target tokens in AQ-MedAI's mapping don't exist in the user's mapping).
  2. The root cause diagnosis: Random loss during fine-tuning is caused by label scrambling due to mismatched vocab mappings, not by poor weight initialization or incompatible architectures.
  3. The fix strategy: A permutation matrix can remap the lm_head rows to align with the user's vocab mapping.
  4. A reusable debugging pattern: When a model works in inference but fails in training, compare the data pipelines systematically—the mismatch is almost certainly in how data is presented, not in the weights themselves.

Broader Significance

This message illustrates a class of bugs that are increasingly common as the ML ecosystem matures: cross-model compatibility issues. When models are trained by different teams, on different data, with different tokenizers or vocabulary mappings, their weights cannot be directly transferred without careful alignment. The assumption that "it's all the same architecture" is dangerously incomplete—the architecture defines the tensor shapes, but the semantics of each position in those tensors depends on the training data and preprocessing pipeline.

The 252-in-32,000 problem is also a cautionary tale about the gap between inference and training. A model can appear to work in inference (producing non-random outputs that pass verification) while being fundamentally incompatible with the training pipeline. Inference only requires that the model's outputs be correlated with the correct distribution; training requires that each output position be exactly aligned with the correct label.

The Path Forward

After this message, the assistant proceeded to implement Option A: building a permutation matrix to remap the AQ-MedAI lm_head weights to align with the user's vocab mapping. This fix would be applied to the training script before relaunching the fine-tuning run. The expectation was that with correct label alignment, the loss would drop significantly (from ~18 to something more reasonable like ~2-3) and accuracy would rise, allowing the fine-tuning to actually adapt the K2 representations to K2.5.

Whether that fine-tuning would ultimately succeed—whether the K2 weights would prove to be a good initialization for K2.5—remained an open question. But at least the training would now be measuring the right thing.