The Critical Question: Why Train a Smaller Vocabulary When Starting from Scratch?

In the middle of a complex machine learning engineering session spanning days of work on speculative decoding for the Kimi-K2.5 model, the user interjected with a deceptively simple question at message 3429:

Why do we train a model with smaller vocab if we train from scratch?

This single sentence, posed at a pivotal moment in the conversation, reveals a deep understanding of the architectural decisions underpinning the EAGLE-3 training pipeline. To appreciate why this question matters, we must trace the context that led to it.

The Backstory: A Long Journey to EAGLE-3 Training

The assistant had been working for hours—across multiple segments of the conversation—to build a working EAGLE-3 speculative decoding system for Kimi-K2.5, a massive Mixture-of-Experts language model with 163,840 vocabulary tokens. Speculative decoding works by having a smaller "draft" model generate candidate tokens quickly, which a larger "verifier" model then checks in parallel. The key insight is that if the draft model is fast enough and accurate enough, overall throughput improves.

Earlier in the session, the assistant had been finetuning an EAGLE-3 draft model starting from a pretrained checkpoint called AQ-MedAI K2. That checkpoint used a reduced vocabulary of 32,000 tokens—roughly one-fifth the size of Kimi-K2.5's full vocabulary. When finetuning from a pretrained checkpoint, the architecture must match exactly: the embedding layer and language model head (lm_head) dimensions are baked into the weights. The 32K vocab was therefore mandatory during the finetuning phase.

But the finetuned draft model performed poorly, achieving only about 15% acceptance rate during speculative decoding—far too low to be useful. After extensive debugging, the assistant concluded that the AQ-MedAI checkpoint was trained on a different model family and was fundamentally incompatible with Kimi-K2.5's architecture. The pivot was decisive: train a new EAGLE-3 drafter from scratch, with no pretrained checkpoint as a starting point.

The Moment of Insight

The assistant had just completed a massive data extraction pipeline. Using a custom-patched SGLang server, it had extracted hidden states from 10,000 inference samples, producing 17.3 million tokens of training data occupying 924 GB of disk space. The extraction ran for nearly two hours at 1.5 samples per second, with zero errors. After verifying the data format was correct, the assistant killed the extraction server, restored the original model code, copied over the vocabulary mapping files from the previous run, and launched the training command.

The training command included --vocab-mapping-dir /data/eagle3/synth_10k_sglang/vocab_mapping, pointing to a directory containing t2d.npy (token-to-draft mapping) and d2t.npy (draft-to-token mapping) files. These mapping files translate between Kimi-K2.5's 163,840-token vocabulary and the draft model's 32,000-token vocabulary. The assistant had copied these files from the earlier finetuning pipeline, assuming they were reusable.

The user caught this assumption immediately.

What the Question Reveals About the User's Understanding

The question "Why do we train a model with smaller vocab if we train from scratch?" demonstrates several layers of comprehension:

First, the user understands the fundamental difference between finetuning and training from scratch. When finetuning, the model architecture is constrained by the pretrained weights—you cannot change the vocabulary size without breaking the weight matrices. But when training from scratch, all weights are randomly initialized, and the vocabulary size is a free architectural parameter. The user recognized that the constraint that justified the 32K vocab (matching the AQ-MedAI checkpoint) no longer applied.

Second, the user grasps the implications of vocabulary size on model capability. A smaller vocabulary means the draft model can only propose tokens that exist within those 32,000 entries. Any token outside this set gets mapped to an "unknown" token (UNK), meaning the drafter can never propose roughly 2% of the verifier's vocabulary. The user implicitly questioned whether this limitation was still acceptable given that the constraint had been lifted.

Third, the user is thinking holistically about the system. They're not just asking "can we change it?" but "should we keep it?"—recognizing that the smaller vocabulary was a design choice inherited from a previous approach, not a deliberate decision for the new pipeline.

The Assistant's Response and the Trade-Off Analysis

The assistant's response (msg 3430) laid out the trade-off clearly. A 32K draft vocabulary produces an lm_head weight matrix of size 32,000 × 7,168 = approximately 230 million parameters. A full 163,840 vocabulary would produce an lm_head of size 163,840 × 7,168 = approximately 1.17 billion parameters—nearly a billion extra parameters. In speculative decoding, the draft model runs multiple forward passes for each verification step (typically 3–5 steps). Each forward pass must compute the lm_head projection and a softmax over the vocabulary. The larger the vocabulary, the more expensive each draft step becomes.

The assistant noted that the 32K vocabulary covers 98.3% of tokens seen in practice—the missed tokens are rare. And the smaller lm_head means faster draft steps, which translates to more speculation opportunities per unit time. This is the classic speculative decoding trade-off: draft speed versus draft coverage.

However, the assistant also acknowledged a subtle point: the 163K vocabulary with a head dimension of 7,168 is not unusually large by modern standards—it's the same size as the verifier's own lm_head. The verifier computes this projection for every token anyway. The question is whether the draft model's smaller size (roughly 2.6 billion parameters) can afford the computational cost of a full-vocabulary projection.

The Deeper Architectural Assumption

The assistant's decision to reuse the 32K vocabulary mapping reveals an implicit assumption: that the mapping infrastructure built for finetuning was a permanent fixture of the pipeline, not a temporary workaround. This is a common pattern in complex ML engineering—infrastructure built to solve one problem (matching a pretrained checkpoint) becomes assumed as necessary for the next problem (training from scratch). The user's question forced a re-examination of this assumption.

The vocabulary mapping files themselves (t2d.npy, d2t.npy, and their PyTorch equivalents) encode a specific selection of 32,000 tokens from the full 163,840-token vocabulary. These tokens were presumably chosen as the most frequent or most important tokens. But if training from scratch, one could choose a different set of 32,000 tokens, or use the full vocabulary, or even use a completely different vocabulary size like 64,000. The mapping files represent a frozen design decision that the user correctly identified as potentially outdated.

Why This Question Matters Beyond This Session

This moment illustrates a critical dynamic in AI-assisted engineering: the assistant, operating with vast context but limited reasoning about long-term design decisions, can carry forward assumptions that deserve re-examination. The user, with a fresh perspective and a clear understanding of the architectural trade-offs, spotted the inconsistency.

The question also highlights the importance of understanding why infrastructure exists, not just what it does. The vocabulary mapping directory was copied because it was "model-independent" and "reusable"—but reusability depends on whether the design decisions encoded in those files remain valid. The user's question forced the assistant to articulate the trade-off explicitly, turning a silent assumption into a conscious design choice.

In the end, the assistant checked the training logs and found that training had already started with the 32K vocab configuration. The decision to continue or restart would depend on whether the 2% token coverage gap was acceptable and whether the speed benefits of the smaller lm_head outweighed the capability loss. But the question itself had already done its most important work: it transformed an implicit assumption into an explicit engineering decision that could be evaluated on its merits rather than inherited by default.