Building the Vocabulary Bridge: The Critical Mapping Step in EAGLE-3 Training

Introduction

In the complex pipeline of training an EAGLE-3 speculative decoding drafter for a 1-trillion-parameter language model, most attention naturally falls on the dramatic steps: generating 10,000 synthetic reasoning samples over five hours, extracting hidden states from a live inference server, or running the multi-hour finetuning loop. Yet between these marquee operations lies a quiet but essential bridge — the vocabulary mapping step. This article examines a single message ([msg 2954]) from an opencode coding session in which the assistant executes a vocabulary mapping script that connects the target model's vast 163,840-token vocabulary to the draft model's compact 32,000-token vocabulary, achieving 98.3% token frequency coverage in the process.

The Message

The message consists of a single bash command executed on a remote container, followed by its output:

ssh root@10.1.230.174 '/root/ml-env/bin/python3 /root/eagle3-train/03_build_vocab_mapping.py 
  --token-freq /data/eagle3/synth_10k/prepared/token_freq.pt 
  --output-dir /data/eagle3/synth_10k/vocab_mapping 
  --target-vocab-size 163840 
  --draft-vocab-size 32000'

The output reports loading the token frequency tensor (shape [163840] with 72,928 non-zero entries), building the mapping from 163,840 to 32,000 tokens, and saving both PyTorch and NumPy format files. The critical statistic is that the draft vocabulary covers 98.3% of token frequency from the training data.

Why This Message Was Written: The Reasoning and Motivation

This message was written because the assistant had just completed the 10,000-sample synthetic data generation step ([msg 2951]) — a 5.3-hour inference run that produced 21 million tokens of training data with 100% reasoning capture. The assistant's todo list explicitly shows the pipeline: after inference comes vocab mapping, then hidden state extraction, then training, then testing. The assistant is executing this plan methodically, wasting no time between steps.

The deeper motivation is architectural. EAGLE-3 is a speculative decoding framework that uses a lightweight "draft" model to predict multiple tokens in parallel, which the target model then verifies. For efficiency, the draft model operates in a reduced vocabulary space — typically 32,000 tokens instead of the full 163,840. But the draft model must still be able to propose tokens that map back to the target model's vocabulary. The vocabulary mapping is the translation table that makes this possible: it defines which target tokens the draft model is allowed to predict and how to convert between the two vocabularies.

Without this mapping, the entire EAGLE-3 pipeline would fail at the training step. The draft model would have no way to know which tokens it should learn to predict, and the verification step would have no way to interpret the draft's output. This message represents the moment where the assistant ensures that the two models — the massive Kimi-K2.5 and the compact EAGLE-3 drafter — can speak the same language.

Decisions Made and Their Significance

Several deliberate choices are encoded in this single command:

Target vocabulary size of 163,840: This matches Kimi-K2.5's full vocabulary, which is typical for DeepSeek-derived architectures. The assistant did not need to discover this — it was already known from the model configuration.

Draft vocabulary size of 32,000: This is the standard EAGLE-3 draft vocabulary size, chosen for efficiency. A smaller vocabulary means the draft model's prediction head is smaller (32,000 logits vs 163,840), which directly translates to faster inference and lower memory usage. The trade-off is coverage: some rare tokens from the target vocabulary will be unmappable, but as the output shows, 98.3% coverage is achieved.

Using token frequency data: The script uses token_freq.pt — a frequency tensor built during the tokenization step. This is a smart design choice: instead of naively mapping the first 32,000 tokens, the frequency-based approach ensures that the most commonly used tokens are prioritized. The 98.3% coverage statistic confirms that the long tail of rare tokens accounts for only 1.7% of actual usage.

Saving both PyTorch and NumPy formats: The output creates t2d.pt, d2t.pt (PyTorch tensors) and t2d.npy, d2t.npy (NumPy arrays). This dual-format approach is pragmatic — PyTorch tensors are needed for GPU-based training, while NumPy arrays are useful for CPU-based preprocessing and debugging. It also provides a fallback if one format has compatibility issues.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

EAGLE-3 architecture: Understanding that speculative decoding uses a draft model with a reduced vocabulary, and that the mapping between vocabularies is a core component of the training pipeline.

Vocabulary mechanics in LLMs: Knowing that language model tokenizers map between text and integer token IDs, and that different models can have different vocabulary sizes. The target model (Kimi-K2.5) uses 163,840 tokens; the draft model uses 32,000.

Tensor operations: The output mentions t2d (target-to-draft) as a boolean mask of shape [163840] and d2t (draft-to-target) as an integer index tensor of shape [32000]. Understanding that t2d indicates which target tokens are representable in the draft vocabulary, and d2t maps each draft token ID to its corresponding target token ID.

The broader pipeline context: This step sits between data generation and hidden state extraction. The assistant is following a predetermined sequence where each step produces inputs for the next.

Output Knowledge Created

This message produces several important outputs:

  1. The mapping files themselves: Four files (t2d.pt, d2t.pt, t2d.npy, d2t.npy) that encode the vocabulary translation. These will be consumed by the hidden state extraction script and the training script.
  2. The 98.3% coverage metric: This is a quality signal. If coverage were significantly lower (say, below 90%), it would indicate a problem — perhaps the draft vocabulary is too small, or the token frequency distribution is pathological. 98.3% is excellent and validates the design choices.
  3. Confirmation of data integrity: The fact that 72,928 non-zero tokens exist out of 163,840 (and that these account for 19.7 million token occurrences) confirms that the synthetic data generation produced a diverse and representative dataset.
  4. A checkpoint in the pipeline: The assistant can now proceed to the next step (hidden state extraction) with confidence that the vocabulary bridge is solid.

Assumptions and Potential Issues

The assistant makes several assumptions that deserve scrutiny:

The draft vocabulary size of 32,000 is appropriate: This is the standard EAGLE-3 default, but it's worth verifying that the coverage is sufficient. 98.3% is good, but the 1.7% of unmapped tokens could include important tokens for certain tasks. If the drafter never learns to predict rare but critical tokens (like special formatting tokens or domain-specific terms), speculation quality could suffer.

The frequency-based mapping is optimal: The script selects the most frequent tokens for the draft vocabulary. This is reasonable for general use, but it might not be optimal for all scenarios. For example, if the drafter needs to predict specific tokens that are rare in the training data but common in deployment, a different selection strategy might be better.

The mapping is static: Once built, this mapping is fixed for the entire training and deployment lifecycle. If the model's usage patterns change (e.g., new tasks with different token distributions), the mapping might become suboptimal.

The script runs correctly: The assistant did not verify the script's logic before running it — it simply read the file, copied it to the container, and executed it. This trust is justified by the clean output, but it's worth noting that no validation step confirms the mapping is correct beyond the summary statistics.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, reveals a methodical approach. After the 10K inference completes ([msg 2951]), the assistant immediately checks the output files ([msg 2952]), reads the vocab mapping script to understand its interface, copies it to the container ([msg 2953]), and then executes it ([msg 2954]). There is no hesitation, no debugging, no trial-and-error — the assistant has clearly done this before or is following a well-understood pattern.

The todo list in [msg 2951] shows the assistant's mental model of the pipeline: "Run vocab mapping (03_build_vocab_mapping.py)" is listed as the next step after inference completes. The assistant is executing a pre-planned sequence, and this message represents the clean execution of one step in that sequence.

Conclusion

The vocabulary mapping message ([msg 2954]) may appear mundane — a single command with a clean output — but it represents a critical juncture in the EAGLE-3 training pipeline. It is the moment where the assistant bridges two different vocabulary spaces, ensuring that the draft model can communicate with the target model. The 98.3% coverage statistic validates the design choices and provides confidence for the subsequent training steps. In the broader narrative of deploying speculative decoding for a 1-trillion-parameter model, this quiet message is where theory meets practice — where the abstract architecture of EAGLE-3 is grounded in concrete tensor operations that will determine whether the drafter learns to predict tokens effectively.