The 32K Vocab Question: A Moment of Critical Reflection in EAGLE-3 Training

"Why do we train a model with smaller vocab if we train from scratech"

This single sentence, typed by a user in the midst of a complex EAGLE-3 speculative decoding pipeline, represents one of the most valuable moments in any engineering collaboration: the moment someone stops the machinery and asks a fundamental "why" question. The typo ("scratech" instead of "scratch") betrays the casual, real-time nature of the exchange, but the substance of the question is anything but casual. It cuts directly to an architectural assumption that had been silently carried forward across multiple training runs, and it forced a deliberate re-examination of a design decision that might otherwise have remained unexamined.

The Context: A Pipeline Built on Inherited Assumptions

To understand why this question matters, we must understand the history that led to it. The assistant and user had been building an EAGLE-3 speculative decoding system for Kimi-K2.5, a massive language model with a vocabulary of 163,840 tokens. The EAGLE-3 architecture requires a "drafter" model — a smaller network that predicts likely next tokens, which the main "verifier" model then checks in parallel. This drafter needs its own language modeling head (lm_head) to produce token probabilities.

Earlier in the project, the drafter had been built by finetuning from an existing checkpoint: the AQ-MedAI EAGLE-3 drafter, which used a reduced 32,000-token vocabulary. When finetuning from a pretrained checkpoint, the vocabulary size is locked — you cannot change it without breaking the weight matrices. The 32K vocab was therefore a hard constraint inherited from the pretrained source.

But the project had since pivoted. After discovering that the finetuned drafter achieved only a ~15% acceptance rate — meaning the speculative decoding was actually slower than running the verifier alone — the team decided to train a new drafter from scratch using freshly extracted hidden states from SGLang. The assistant had just completed extracting 10,000 samples of hidden states (17.3 million tokens, 924 GB of data) and was launching the new training run.

The Moment of Questioning

When the assistant launched the training command in message [msg 3426], it included --vocab-mapping-dir /data/eagle3/synth_10k_sglang/vocab_mapping. This flag tells the training script to use the same 32K vocabulary mapping that was created for the finetuning approach. The vocab mapping files — t2d.npy (token-to-draft mapping) and d2t.npy (draft-to-token mapping) — had been copied from the previous run in message [msg 3419], where the assistant described them as "model-independent, just maps Kimi-K2.5's 163K vocab to 32K draft vocab."

The user saw this and asked the obvious question: if we're training from scratch, why are we constraining ourselves to a smaller vocabulary? The 32K vocab was a necessity when finetuning from AQ-MedAI's checkpoint, but with a fresh start, shouldn't we use the full 163,840-token vocabulary?

This question reveals several things about the user's understanding. First, they grasp the fundamental difference between finetuning (where architecture is frozen) and training from scratch (where it is not). Second, they recognize that a reduced vocabulary is a tradeoff — it limits what tokens the drafter can propose, potentially capping the maximum theoretical acceptance rate. Third, they are paying close enough attention to the pipeline to notice when an assumption from one phase is being silently carried into another.

The Assumptions at Play

The assistant had made a reasonable but unexamined assumption: that the 32K vocab was the right choice regardless of training regime. This assumption had several layers:

  1. Efficiency carryover: The 32K vocab reduces the lm_head from 1.17 billion parameters (163,840 × 7,168) to 230 million parameters (32,000 × 7,168). This is a ~940M parameter saving. In speculative decoding, the drafter runs multiple forward passes per verification step, so lm_head size directly impacts latency. The assistant had internalized this efficiency argument without re-evaluating it for the from-scratch case.
  2. Infrastructure inertia: The vocab mapping files already existed, the training script already supported them, and the data pipeline was already configured. Changing to full vocab would require code changes, data reformatting, and re-validation. There is always pressure to keep moving forward.
  3. The 98.3% coverage heuristic: The 32K most frequent tokens in Kimi-K2.5's vocabulary cover approximately 98.3% of actual token usage. The assistant implicitly assumed this was "good enough" — that the 1.7% of tokens outside the reduced vocab were rare enough to not matter. But this assumption had never been explicitly tested for the speculative decoding use case.
  4. Training cost: A larger lm_head means more trainable parameters, longer training time, and more GPU memory. The assistant's training log later shows 1,190.9M trainable params with 32K vocab; with full vocab this would jump to approximately 2.1B, nearly doubling training cost.

The Assistant's Analysis

The assistant's response in message [msg 3430] is a model of how to handle such a question. Rather than defending the existing choice or immediately switching, the assistant lays out the tradeoffs clearly:

"The 32K draft vocab was carried over from the AQ-MedAI K2 drafter architecture, which used a reduced vocab for efficiency. When finetuning from that checkpoint, the smaller vocab was mandatory to match the pretrained weights. But training from scratch, there's no such constraint."

This acknowledgment is crucial. It separates the historical reason (mandatory for finetuning) from the current situation (optional for from-scratch). The assistant then enumerates both options with their concrete implications:

Option 1 (32K draft vocab): Smaller lm_head (230M params), faster softmax and sampling during speculative decoding, but requires token mapping and cannot propose ~2% of tokens (which get mapped to UNK).

Option 2 (Full 163,840 vocab): No mapping needed, drafter can propose any token, but lm_head is much larger (1.17B params), adding ~940M parameters and slowing each draft forward step.

The assistant also provides a key insight about the speculative decoding use case: "the draft model runs multiple times per verify step. The lm_head cost matters." This contextualizes the efficiency argument — in speculative decoding, the drafter's speed directly determines how many draft tokens can be generated per verification cycle, which in turn determines the potential speedup.

The Resolution

The assistant concludes that the 98.3% coverage of the 32K vocab should be "fine in practice" because the missed tokens are rare, and the smaller lm_head enables faster draft steps, creating "more speculation opportunities." The user accepts this reasoning in message [msg 3432] with a simple "Let's keep 32k then."

This resolution is satisfying because it represents genuine collaboration. The user raised a legitimate concern, the assistant engaged with it seriously, provided a structured analysis of the tradeoffs, and the user made an informed decision to proceed. The conversation didn't need to be long — the question, the analysis, and the decision all happened within a few messages — but the intellectual work was real.

The Deeper Lesson

This exchange illustrates a pattern that recurs throughout complex engineering projects: the silent propagation of assumptions across phase boundaries. When a project pivots from "finetune an existing checkpoint" to "train from scratch," many architectural decisions that were previously constrained become unconstrained. But unless someone explicitly re-examines them, they persist by inertia.

The user's question — "Why do we train a model with smaller vocab if we train from scratch?" — is the kind of question that separates deep engagement from surface-level participation. It required understanding not just what the pipeline was doing, but why each component existed and whether its original justification still held. In a project spanning thousands of lines of code, dozens of configuration parameters, and multiple training runs across different frameworks (vLLM, SGLang, speculators), such moments of critical reflection are invaluable.

The 32K vocab decision, once examined, turned out to be defensible on its own merits even for the from-scratch case. But it was important that it was examined, rather than remaining an unspoken assumption. The user's typo-ridden question, for all its casual presentation, performed a genuinely useful function: it forced the team to articulate a design rationale that had previously only existed implicitly in the code and configuration files.

Input and Output Knowledge

To understand this exchange, a reader needs to know: what EAGLE-3 speculative decoding is and why it needs a drafter model; the difference between finetuning and training from scratch; how vocabulary size affects language model head parameters; and the specific history of this project (that the previous drafter was finetuned from AQ-MedAI's checkpoint with a 32K vocab). The exchange also assumes familiarity with the concept of token mapping (t2d/d2t) and the idea that a reduced vocabulary can cover most but not all actual tokens.

The output knowledge created by this exchange is a documented design decision: the team consciously chose to retain the 32K vocabulary for the from-scratch drafter, with a clear rationale based on the tradeoff between coverage and speed. This decision is now explicit rather than implicit, which means it can be revisited later if empirical results show that the 1.7% of uncovered tokens are causing problems.

Conclusion

The question "Why do we train a model with smaller vocab if we train from scratch?" is, on its face, a simple clarification request. But in the context of this complex engineering project, it represents something more: a refusal to let assumptions propagate unexamined across phase boundaries. The assistant's thoughtful analysis of the tradeoffs, and the user's informed acceptance of the recommendation, demonstrate collaborative engineering at its best. The 32K vocab stayed, but it stayed because it was chosen, not because it was inherited.