The Critical Vocab Check: Phase 0 of the K2-to-K2.5 EAGLE-3 Drafter Probe

Introduction

In the high-stakes world of large language model inference optimization, every millisecond counts. When a team has invested weeks training a custom EAGLE-3 speculative decoding drafter only to see it hurt performance (60 tok/s vs. 82 tok/s baseline), the temptation to chase radical solutions is strong. Message [msg 4944] captures the exact moment when a months-long optimization journey pivots from planning to execution—the first concrete action of "Phase 0," a quick probe to determine whether a pre-trained drafter from a different model variant can be repurposed with minimal effort.

This message, sent by the AI assistant in an opencode coding session, is deceptively simple on its surface: three parallel bash commands, a brief explanatory sentence, and a set of results. But beneath this surface lies a carefully reasoned strategy, a deep understanding of the system's bottlenecks, and a series of assumptions that will determine the success or failure of the entire fine-tuning approach. This article unpacks that single message in detail, examining the reasoning, decisions, assumptions, and knowledge that make it a pivotal moment in the conversation.

The Message in Full

The assistant writes:

Good. Now let me clean up GPU handles and check the AQ-MedAI safetensors for t2d/d2t vocab mappings, and also check our existing drafter's config for comparison.

It then dispatches three bash commands in parallel:

  1. GPU cleanup: fuser -k /dev/nvidia* followed by nvidia-smi to verify all GPU memory is freed
  2. Config comparison: Reading the config.json from the team's existing K2.5 drafter at /data/eagle3/output_100k_sglang/4/config.json
  3. Vocab mapping inspection: A Python script that opens the AQ-MedAI safetensors file and prints every key with its shape and dtype The results confirm all eight GPUs are clean (0 MiB used), the K2.5 drafter config matches the AQ-MedAI architecture, and the AQ-MedAI safetensors contain the expected weight keys including the critical d2t (draft-to-target) vocabulary mapping.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the narrative arc that led to it. The session's broader context is a months-long effort to deploy the Kimi-K2.5-INT4 model—a 1-trillion-parameter Mixture-of-Experts language model—on a machine with 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe (no NVLink). The baseline throughput of 82 tok/s is respectable, but the team wants more.

The previous major effort was training a custom EAGLE-3 speculative decoding drafter from scratch using 37,000 samples of K2.5 hidden state data. That drafter achieved ~75% validation accuracy and was deployed, but it reduced throughput to 60 tok/s—27% worse than baseline. The root cause was identified: the verify step (where the target model checks the draft tokens) takes ~30ms per cycle, consuming 97% of the total cycle time. The drafter's accept rate of ~2.0 tokens per step was below the break-even threshold of 2.46.

With the from-scratch drafter underperforming, the team discovered the AQ-MedAI Kimi-K2-Instruct-eagle3 model on HuggingFace—a pre-trained EAGLE-3 drafter for the Kimi-K2 model (the predecessor to K2.5). This drafter was trained on 1.4 million samples and achieves an accept_len of 3.2–3.5 on K2. The critical question: could it be fine-tuned for K2.5?

The assistant spent several messages ([msg 4923] through [msg 4931]) analyzing the AQ-MedAI drafter: downloading it, comparing every weight shape against the K2.5 drafter, and confirming architectural compatibility. The conclusion was that the architectures are "identical in every way"—same hidden_size (7168), same intermediate_size (18432), same attention heads, same layer structure. The only difference was that AQ-MedAI's safetensors file excludes embed_tokens.weight (loaded from the base model at runtime).

The user then directed the assistant to write a comprehensive game plan document ([msg 4932]–[msg 4938]), which was expanded into a full project summary ([msg 4939]). When the user gave the green light to proceed ([msg 4940]), the assistant read the game plan, created a todo list, killed existing server processes, and checked the AQ-MedAI directory ([msg 4941]–[msg 4943]).

Message [msg 4944] is the next step in that execution chain: the first concrete action of Phase 0. The assistant has finished planning and is now executing the probe that will determine whether the entire fine-tuning approach is viable.

The Three Parallel Actions: A Study in Efficient Execution

The assistant dispatches three independent checks simultaneously, demonstrating a sophisticated understanding of parallelism. Each check serves a distinct purpose:

1. GPU Cleanup: Ensuring a Clean Slate

The command fuser -k /dev/nvidia* is more aggressive than simply killing Python processes. The fuser utility identifies and terminates any process—regardless of its name—that has a file handle open on any NVIDIA device file. This catches zombie processes, orphaned CUDA contexts, and any other residual GPU state that could interfere with a fresh server launch.

The follow-up nvidia-smi query confirms that all eight GPUs show 0 MiB of memory used. This is a critical precondition: if any GPU had residual allocations, the SGLang server might fail to initialize or experience memory fragmentation.

The assistant's choice to run this command before launching the probe, rather than as part of the earlier process kill ([msg 4943]), shows careful staging. The earlier kill targeted Python processes specifically; this command is a more thorough cleanup that catches anything the earlier pass missed.

2. Config Comparison: Verifying Architecture Compatibility

Reading the existing K2.5 drafter's config.json serves as a reference point. The assistant already compared these configs in [msg 4925]–[msg 4926] and found them identical, but re-reading the config here serves two purposes: it confirms the file is still accessible (the directory structure hasn't changed), and it provides the config content in the same message as the AQ-MedAI inspection, making it easy for the user (or a future reader) to verify compatibility.

The config reveals the critical parameters: draft_vocab_size: 32000, vocab_size: 163840, eagle_aux_hidden_state_layer_ids: [2, 30, 58], and the standard DeepSeek V3 / MLA architecture parameters. These must match exactly for the AQ-MedAI weights to be loadable.

3. Vocab Mapping Inspection: The Hidden Pitfall

The most technically significant check is the Python script that inspects the AQ-MedAI safetensors. The assistant specifically mentions checking for "t2d/d2t vocab mappings"—this is the crux of the compatibility question.

In EAGLE-3, the drafter predicts tokens in a draft vocabulary (typically 32,000 tokens), which must be mapped to the target model's vocabulary (typically 163,840 tokens for DeepSeek-based models). The d2t (draft-to-target) mapping is a 32,000-element integer tensor that maps each draft token ID to its corresponding target token ID. The t2d (target-to-draft) mapping is a 163,840-element boolean tensor that indicates which target tokens have a corresponding draft token.

If the AQ-MedAI drafter's d2t mapping differs from what K2.5 expects, the drafter would predict tokens that map to entirely different meanings in the target model's vocabulary—effectively producing gibberish. The assistant's inspection confirms that AQ-MedAI has a d2t tensor of shape [32000] with dtype int64, matching the K2.5 drafter's d2t shape.

The output is partially truncated in the message, but the presence of d2t is confirmed. The t2d tensor may be absent from the safetensors (it's a large boolean tensor that can be derived from d2t at runtime), or it may appear later in the truncated output.

Assumptions and Reasoning

Every action in this message rests on a foundation of assumptions, some explicit and some implicit:

Assumption 1: Architecture compatibility implies functional compatibility. The assistant has confirmed that every weight shape matches between the AQ-MedAI K2 drafter and the K2.5 drafter. But shape compatibility does not guarantee representation compatibility—the hidden states at layers [2, 30, 58] might encode different features in K2 vs. K2.5, even though the tensor dimensions are the same. This is precisely what Phase 0 is designed to test.

Assumption 2: The d2t mapping is compatible. The assistant checks that AQ-MedAI has a d2t tensor, but does not verify that the values match what K2.5 expects. If K2 and K2.5 use different tokenizer vocabularies, the draft-to-target mappings could be completely different even though the tensor shapes are the same. This is a critical risk that the probe will reveal.

Assumption 3: GPU cleanup was successful. The nvidia-smi output shows 0 MiB on all GPUs, but this only reports memory usage, not GPU state. There could be lingering CUDA contexts or driver state that causes issues on the next launch. The assistant implicitly trusts that a clean memory report means a clean launch environment.

Assumption 4: The existing drafter directory is valid. The assistant reads /data/eagle3/output_100k_sglang/4/config.json without verifying that the corresponding model.safetensors file is intact. If the weights were corrupted or the directory was modified, the config alone wouldn't reveal this.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. EAGLE-3 speculative decoding architecture: Knowledge that EAGLE-3 uses a small "draft" model to predict multiple tokens per step, which are then verified by the large "target" model. The draft model operates on hidden states from specific layers of the target model.
  2. The K2→K2.5 model relationship: Understanding that Kimi-K2 and Kimi-K2.5 are two versions of the same base architecture (DeepSeek V3 / MLA with 61 layers, 7168 hidden size), but may have different trained weights and potentially different tokenizer vocabularies.
  3. Vocabulary mapping in EAGLE-3: The concept of draft vocabulary (typically 32K tokens) vs. target vocabulary (163,840 tokens for DeepSeek models), and the d2t/t2d mappings that bridge them.
  4. The PCIe bottleneck context: Understanding that the verify step's ~30ms latency is dominated by NCCL all-reduce operations across PCIe-connected GPUs, and that improving the accept rate is the primary lever for making speculative decoding viable.
  5. The session's optimization history: The failed from-scratch drafter, the NCCL tuning efforts, the profiling work, and the decision to pivot to fine-tuning the AQ-MedAI drafter.

Output Knowledge Created

This message produces several concrete outputs:

  1. Confirmed clean GPU state: All eight GPUs are ready for a fresh server launch, with no residual memory allocations.
  2. Confirmed config compatibility: The K2.5 drafter's config.json is accessible and contains the expected architecture parameters, matching the AQ-MedAI config.
  3. Confirmed AQ-MedAI weight structure: The safetensors file contains the expected keys with matching shapes: d2t ([32000] int64), fc.weight ([7168, 21504] bf16), lm_head.weight ([32000, 7168] bf16), and the midlayer transformer weights.
  4. Verified d2t presence: The critical draft-to-target vocabulary mapping exists in the AQ-MedAI safetensors, confirming that the drafter can map its predictions to the target model's vocabulary.
  5. Established a clean baseline for Phase 0: With GPUs clean and files verified, the assistant can proceed to the actual probe: launching SGLang with the AQ-MedAI drafter and measuring the accept rate on K2.5.

The Thinking Process: What the Message Reveals

The assistant's reasoning is visible in several aspects of this message:

Prioritization of risk: The assistant leads with the vocab mapping check, not the config comparison or GPU cleanup. This reveals that the assistant considers vocabulary compatibility to be the highest-risk item—if the mappings don't match, the entire approach fails regardless of architecture compatibility.

Parallel execution strategy: By running all three checks in a single round, the assistant minimizes latency. Each check is independent (GPU state doesn't affect file reads, and file reads don't affect each other), so there's no reason to serialize them. This is a hallmark of efficient agentic reasoning.

Explicit intent communication: The assistant states "let me clean up GPU handles and check the AQ-MedAI safetensors for t2d/d2t vocab mappings" before executing the commands. This verbalizes the reasoning for the user, making the decision process transparent.

Building on prior work: The message references earlier discoveries (the config comparison from [msg 4925]–[msg 4926], the safetensors inspection from [msg 4930]) without rehashing them. The assistant assumes the user is following the narrative and doesn't need redundant explanation.

Cautious optimism: The assistant's tone is matter-of-fact, not triumphal. It doesn't declare victory after finding matching shapes—it simply records the findings and moves to the next step. This reflects an understanding that shape compatibility is necessary but not sufficient for success.

Mistakes and Incorrect Assumptions

While the message itself is well-executed, several potential issues are worth noting:

The truncated output is a minor flaw: The Python script's output is cut off mid-key (ending with midlayer.mlp.up_proj.weight: torch.Size([18432, 7168]) torch...). This means the full set of AQ-MedAI keys is not visible in the message. While the most critical keys (d2t, fc.weight, lm_head.weight) are confirmed present, the truncation obscures whether t2d exists or whether all midlayer attention weights are present. A more careful script would print all keys without truncation.

No verification of d2t values: The assistant checks that d2t exists with the right shape, but doesn't inspect its values. If K2 and K2.5 use different tokenizer vocabularies, the d2t mapping could map draft token 0 to target token 42 in K2 but to target token 99 in K2.5—a catastrophic mismatch that shape alone cannot detect.

Over-reliance on nvidia-smi for GPU state: The nvidia-smi memory check only reports current memory usage. It doesn't detect GPU compute mode errors, driver issues, or ECC errors that could surface during model loading. A more thorough check might include nvidia-smi -q -d COMPUTE or a quick CUDA sample.

Assumption that the existing config is authoritative: The config at /data/eagle3/output_100k_sglang/4/config.json might be stale if the training script modified it after saving. The assistant doesn't cross-reference against the training script's parameters.

Significance in the Larger Narrative

Message [msg 4944] occupies a pivotal position in the session's narrative arc. It is the bridge between analysis and action—the moment when the team stops studying the AQ-MedAI drafter and starts using it. The three checks performed here are the final prerequisites before launching the actual probe server.

The message also embodies the session's core tension: the gap between architectural compatibility and functional compatibility. The team knows that the K2 and K2.5 models share the same architecture, but they don't yet know whether the hidden state representations are similar enough for the K2 drafter to be a useful initialization for K2.5 fine-tuning. This message's checks address the easy questions (shape compatibility, vocab mapping presence) to clear the way for the hard question (representation similarity), which will be answered by the actual Phase 0 probe.

In the broader context of LLM inference optimization, this message illustrates a fundamental principle: speculative decoding optimization is a systems engineering problem as much as a machine learning problem. The assistant must manage GPU state, file paths, vocabulary mappings, and configuration parameters alongside the ML concerns of model architecture and training data. Each of these layers can introduce failure modes that derail the entire effort.

Conclusion

Message [msg 4944] is a masterclass in efficient, risk-aware execution. In a single round of parallel tool calls, the assistant cleans up GPU state, verifies architectural compatibility, and inspects the critical vocabulary mapping that could make or break the entire fine-tuning approach. The reasoning is transparent, the assumptions are reasonable, and the execution is clean.

The message's true significance lies not in what it accomplishes (which is modest—a few file reads and a GPU cleanup) but in what it enables: the Phase 0 probe that will determine whether the AQ-MedAI drafter can be adapted for K2.5. Every subsequent action in the session—the server launch, the accept rate measurement, the fine-tuning attempt, and ultimately the pivot to system-level optimization—traces back to the foundation laid in this message.

For anyone studying LLM inference optimization, this message offers a valuable case study in how to approach the critical transition from planning to execution: verify your assumptions, clean your state, check your mappings, and then proceed with eyes open to the risks that remain.