Verifying the EAGLE-3 Checkpoint: A Critical Format Compatibility Check

In message [msg 2782] of this opencode coding session, the assistant performs a seemingly simple but technically significant verification step: inspecting the contents of a freshly trained EAGLE-3 draft model checkpoint to determine whether its format is compatible with vLLM's speculative decoding loading mechanism. The message captures a moment where months of infrastructure work—building CUDA toolkits, resolving flash-attn compilation issues, deploying multi-GPU inference servers, and constructing a custom EAGLE-3 training pipeline—converges on a single, focused question: will vLLM be able to load this checkpoint?

The Context: An EAGLE-3 Training Pipeline Comes Online

The assistant has just completed the first successful end-to-end training run of an EAGLE-3 draft model for Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts language model running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. EAGLE-3 is a speculative decoding framework that trains a lightweight "draft" model to predict the next several tokens the main "verifier" model would generate, enabling faster inference by processing multiple draft tokens in parallel. The training pipeline—built on top of the speculators library—had been painstakingly debugged over the preceding messages: API incompatibilities between speculators v0.3.0 and vLLM 0.16 were patched, a custom worker for the DeepSeekV2-derived architecture was written, hidden state extraction was validated on test samples, and finally a 10-sample training run completed in about one minute on a single GPU.

But a successful training run is only half the battle. The checkpoint must be loadable by vLLM, the inference engine that will actually use the draft model for speculative decoding. This is where message [msg 2782] enters.

The Question: Separate Files or Embedded Tensors?

The assistant's reasoning, stated explicitly in the message, reveals a precise concern:

"Now let me also check what vLLM expects for loading an EAGLE-3 speculator — does it need t2d.pt and d2t.pt in the model directory, or are they saved inside the model.safetensors?"

This question arises from knowledge of how EAGLE-3 works architecturally. The draft model maintains a vocabulary mapping between the draft model's tokenizer and the target (verifier) model's tokenizer. Two mappings are needed: d2t (draft-to-target) maps draft token IDs to target token IDs, and t2d (target-to-draft) maps target token IDs to draft token IDs. These mappings are essential because the draft and target models may use different tokenizers with different vocabulary sizes. In some implementations, these mappings are stored as separate PyTorch files (d2t.pt and t2d.pt) alongside the model weights. In others, they are embedded directly in the safetensors weight file. The assistant needs to determine which convention the speculators training pipeline follows, because vLLM's EAGLE-3 loader will look for these mappings in a specific location.

The assistant's choice to verify this before attempting to load the checkpoint into vLLM demonstrates a disciplined engineering approach: test the artifact independently rather than debugging a cryptic vLLM loading error later.

The Execution: Inspecting the Safetensors

To answer this question, the assistant executes a bash command that opens the freshly created model.safetensors file using the safetensors library's safe_open function, iterates over all keys, and prints each tensor's name and shape. This is a lightweight, non-destructive inspection that requires no GPU memory and completes in seconds.

The output reveals 16 keys in total. Among them is a tensor named d2t with shape torch.Size([32000])—the draft-to-target vocabulary mapping. Critically, there is no corresponding t2d tensor. This tells the assistant that the speculators training pipeline embeds the d2t mapping inside the safetensors file but does not save the reverse t2d mapping. The remaining keys include the expected model components: embed_tokens.weight (a large embedding table at [163840, 7168]—note the unusual first dimension, which is 32000 * 5.12 rounded, reflecting the EAGLE-3 multi-token prediction head), fc.weight (the fusion layer), and the decoder layer's sub-components including layernorms, MLP projections, and self-attention projections.

What This Reveals About the Checkpoint's Health

The presence of d2t confirms that the vocabulary mapping was correctly extracted from the verifier model and saved during training. The shape [32000] matches the draft vocabulary size configured in the training script. The embedding table's first dimension of 163840 is particularly telling: EAGLE-3 predicts multiple future tokens simultaneously, and this dimension represents draft_vocab_size * (num_pred_tokens + 1)—in this case, 32000 * 5.12, where the fractional multiplier arises from the specific EAGLE-3 architecture variant being used.

The decoder layer keys (layers.0.self_attn.k_proj, layers.0.self_attn.v_proj, layers.0.self_attn.q_proj, layers.0.self_attn.o_proj) confirm that the single-layer transformer decoder was properly initialized and trained. The MLP projections (gate_proj, up_proj, down_proj) with dimensions [18432, 7168] and [7168, 18432] match the intermediate size configured for the draft model.

Assumptions and Knowledge Required

To fully understand this message, the reader needs background knowledge in several areas. First, an understanding of speculative decoding and the EAGLE-3 architecture: that it uses a lightweight draft model to predict multiple future tokens, requiring vocabulary mappings between draft and target tokenizers. Second, familiarity with HuggingFace's safetensors format and how model weights are stored and organized. Third, knowledge of the speculators library and its training pipeline, which the assistant had been exploring and patching over the preceding messages. Fourth, awareness of vLLM's speculative decoding API and its expectations for checkpoint layout.

The assistant makes a key assumption: that vLLM's EAGLE-3 loader will look for d2t and t2d tensors inside the safetensors file (or as separate files in the model directory). This assumption is reasonable given the assistant's prior exploration of the vLLM source code for EAGLE-3 loading, but it is not verified in this message—that verification will come in a subsequent step when the checkpoint is actually loaded into vLLM.

The Missing t2d Mapping: A Deliberate Design Choice or an Oversight?

One of the most interesting findings from this inspection is the absence of a t2d tensor. The assistant does not comment on this omission in the message itself, but it is a significant detail. In the EAGLE-3 architecture, the d2t mapping is used during inference to convert the draft model's predicted token IDs into the verifier model's token ID space for verification. The reverse t2d mapping is needed during training to convert target (ground truth) token IDs into the draft model's vocabulary space for loss computation. The fact that t2d is absent from the safetensors file suggests that either: (a) the speculators training pipeline computes t2d on-the-fly from d2t (by inverting the mapping, though this is lossy if the mapping is not bijective), (b) t2d is saved separately in a file that was not captured in this inspection, or (c) the training pipeline does not require t2d because it operates entirely in the verifier's token ID space.

This ambiguity would need to be resolved before the checkpoint can be deployed. The assistant's decision to inspect the file now—rather than waiting for a vLLM loading failure—demonstrates proactive debugging. The knowledge created by this message is a precise inventory of the checkpoint's contents, which can be compared against vLLM's expected format to identify any gaps.

The Broader Significance: From Training to Production

This message sits at a critical juncture in the session. The assistant has just proven that the EAGLE-3 training pipeline works end-to-end, but the checkpoint is not yet ready for production deployment. The inspection reveals that the checkpoint contains the essential components—model weights, vocabulary mapping, and config—but the format must be validated against vLLM's expectations. The subsequent messages in the session will show the assistant pivoting to generate higher-quality synthetic training data, suggesting that while the pipeline is functional, the data quality needs improvement before a production-quality draft model can be trained.

The message also illustrates a recurring theme in this session: the tension between the speculators library's internal conventions and vLLM's expectations. Earlier messages showed the assistant patching API incompatibilities between the two libraries; this checkpoint inspection is a continuation of that integration work. Each library was developed independently, and their interfaces must be carefully aligned for the combined system to function.

Conclusion: A Small Inspection with Large Implications

Message [msg 2782] is, on its surface, a simple diagnostic command: open a file, list its contents, print the keys. But in the context of the broader session, it represents a critical quality gate. The assistant is not merely checking that training "worked" in the narrow sense of producing output files; they are verifying that the output meets the specific format requirements of the downstream system. This attention to integration detail—understanding not just how to train a model, but how the trained artifact will be consumed—is what separates a successful deployment from a frustrating debugging session.

The message also demonstrates a disciplined engineering workflow: test early, test the artifact independently, and verify assumptions before they become failures. By inspecting the safetensors file immediately after training, the assistant creates a clear record of what the checkpoint contains, enabling informed decisions about next steps—whether that means adjusting the training pipeline, patching vLLM's loader, or generating additional data.