The Checkpoint Conversion: Bridging Speculators and vLLM for EAGLE-3 Inference

Introduction

In the sprawling landscape of large language model deployment, few tasks are as deceptively complex as checkpoint conversion. What appears to be a simple file format transformation often conceals a minefield of tensor naming conventions, architectural assumptions, and framework-specific quirks. Message [msg 2994] captures this moment perfectly: the assistant has just completed a grueling 5-epoch EAGLE-3 training run on 10,000 synthetic samples for the Kimi-K2.5 model, producing a trained drafter checkpoint in the speculators library format. Now, it must prepare that checkpoint for consumption by vLLM, the inference engine that will ultimately serve the model. This single message — a diagnostic inspection of the checkpoint's internal structure — represents the critical bridge between training and inference, between the speculators training framework and the vLLM serving runtime.

The Context: A Pipeline Reaches Its Inflection Point

To understand why this message matters, one must appreciate the journey that led here. The preceding messages in the conversation ([msg 2971] through [msg 2993]) document the completion of a massive synthetic data pipeline. The assistant had generated 10,000 reasoning samples from Kimi-K2.5 itself, extracted hidden states at 3,165 tokens per second (producing 828 GB of training data), and then fine-tuned an EAGLE-3 drafter from the AQ-MedAI checkpoint over five epochs — completing in 2.6 hours with checkpoints saved at each epoch.

The final checkpoint resides at /data/eagle3/output_10k/4/. It contains a model.safetensors file (4.5 GB), a config.json, and optimizer states. But this checkpoint was produced by the speculators training library, which uses its own internal naming conventions and architectural representations. vLLM, the inference engine that will deploy the model with speculative decoding, expects a different format — specifically, a "flat LlamaForCausalLMEagle3" configuration with specific tensor key names and embedded d2t (draft-to-target) and t2d (target-to-draft) vocabulary mapping tensors.

This is the inflection point. The assistant cannot simply point vLLM at the training output and expect it to work. It must first understand exactly what the training produced, compare it against a known-good reference (the AQ-MedAI checkpoint), and then write a conversion script that bridges the two formats.

The Message: A Diagnostic First Step

The message begins with the assistant articulating its understanding of what the final checkpoint requires:

Now I need to prepare the final checkpoint for vLLM. The checkpoint needs: 1. Correct config.json (flat LlamaForCausalLMEagle3 format) — already written 2. model.safetensors — already there, but likely has speculators key names 3. d2t and t2d tensors embedded in the safetensors file

This three-point checklist reveals the assistant's mental model of the conversion task. It has already ensured the configuration file is in the correct format — a step it verified at the end of the training run in [msg 2993]. The remaining unknowns are the tensor key names (point 2) and the presence of the vocabulary mapping tensors (point 3). The assistant suspects the key names are in "speculators key names" — a format used by the speculators library — and needs to confirm this before writing a renaming script.

The diagnostic tool of choice is a Python script executed remotely via SSH that uses the safetensors library's safe_open function to inspect the weight keys of both the trained model and the AQ-MedAI reference checkpoint. This is a deliberate, low-risk first step: before making any changes, the assistant wants to see exactly what it's working with.

The Inspection: What the Output Reveals

The output from the trained model (epoch 4) shows:

=== Trained model (epoch 4) ===
  d2t: torch.Size([32000]) torch.int64
  embed_tokens.weight: torch.Size([163840, 7168]) torch.bfloat16
  fc.weight: torch.Size([7168, 21504]) torch.bfloat16
  layers.0.hidden_norm.weight: torch.Size([7168]) torch.bfloat16
  layers.0.input_layernorm.weight: torch.Size([7168]) torch.bfloat16
  layers.0.mlp.down_proj.weight: torch.Size([7168, 18432]) torch.bfloat16
  layers.0.mlp.gate_proj.weight: torch.Size([18432, 7168]) torch.bfloat16
  layers.0.mlp.up_proj.weight: ...

The output is truncated (indicated by "..."), but the visible keys already tell a revealing story. The d2t tensor is present with shape [32000] and dtype int64 — this is the draft-to-target vocabulary mapping, a critical component for EAGLE-3's two-vocabulary architecture. The embed_tokens.weight has shape [163840, 7168], which corresponds to the full vocabulary size of the DeepSeek V3 / Kimi-K2.5 base model (163,840 tokens) projected into the 7,168-dimensional embedding space of the drafter.

The layer structure — hidden_norm, input_layernorm, and the MLP projections (down_proj, gate_proj, up_proj) — reveals that the EAGLE-3 drafter is architecturally similar to a standard transformer decoder layer, but with specific modifications for the speculative decoding task. The fc.weight tensor with shape [7168, 21504] is the feature connector that bridges the base model's hidden states to the drafter's input space, a signature component of the EAGLE architecture family.

The Reasoning: Why This Approach Makes Sense

The assistant's decision to inspect rather than assume is grounded in sound engineering practice. The speculators library and vLLM are developed by different teams with different internal conventions. While both implement EAGLE-3, they may use different key names for identical tensors. For example, speculators might use layers.0.mlp.down_proj.weight while vLLM might expect model.layers.0.mlp.down_proj.weight or some other prefix convention.

By comparing the trained model against the AQ-MedAI reference, the assistant is performing a differential analysis. The AQ-MedAI checkpoint is known to work with vLLM (it was used as the starting point for fine-tuning), so its key naming convention is the target. Any differences between the trained model's keys and the AQ-MedAI keys represent transformation work that needs to be done.

The assistant also implicitly assumes that the tensor values themselves are correct — the training produced valid weights — and that only the naming and packaging need adjustment. This is a reasonable assumption given that the training completed without errors and the loss curves showed proper convergence (as evidenced by the consistent 6 it/s training speed across epochs).

The Knowledge Boundary: What the Reader Must Understand

To fully appreciate this message, one needs background in several areas. First, the EAGLE-3 architecture itself: it is a speculative decoding framework where a lightweight "drafter" model predicts multiple future tokens in parallel, conditioned on the base model's hidden states. The drafter has its own vocabulary (32,000 tokens in this case) that is mapped to and from the base model's much larger vocabulary (163,840 tokens) via the d2t and t2d tensors.

Second, the speculators library: it provides training infrastructure for EAGLE-3 and related architectures, but uses its own checkpoint format. The library's Eagle3DraftModel architecture (visible in the config.json from [msg 2989]) is not directly compatible with vLLM's LlamaForCausalLMEagle3 architecture, even though they represent the same underlying model.

Third, the vLLM inference engine: it supports EAGLE-3 speculative decoding but requires checkpoints in a specific format with specific key names and configuration fields. The "flat" config format mentioned by the assistant refers to a configuration dictionary where all nested fields are flattened into a single level, as opposed to the nested structure used by Hugging Face transformers.

The Output Knowledge: What This Message Creates

This message produces critical diagnostic information. By the time the output is fully rendered (the truncated portion would show all remaining layer keys), the assistant will know:

  1. Whether the trained checkpoint contains all required tensors
  2. Whether the key names match vLLM's expectations or need renaming
  3. Whether the d2t and t2d tensors are present and correctly shaped
  4. How the trained checkpoint differs from the known-good AQ-MedAI reference This knowledge directly enables the next step: writing a conversion script that renames tensors, restructures the config, and packages everything into a format vLLM can load. Without this diagnostic, any conversion attempt would be guesswork.

The Broader Significance: A Microcosm of ML Engineering

This message, for all its apparent simplicity, encapsulates a fundamental truth about modern ML engineering: the hardest problems are often at the interfaces between systems. Training a model is one challenge; deploying it is another entirely. The gap between the two is filled with countless small, careful diagnostic steps like this one — inspecting tensor shapes, comparing key names, verifying configurations. Each step is unglamorous, but collectively they determine whether a project succeeds or fails.

The assistant's methodical approach — articulate requirements, inspect current state, compare against reference, plan transformation — is a template for any engineer facing a similar integration challenge. It prioritizes understanding over action, diagnosis over intervention. In a domain where a single misplaced tensor can crash an entire inference server, this caution is not merely prudent; it is essential.

Conclusion

Message [msg 2994] marks the transition from training to deployment in a complex EAGLE-3 speculative decoding pipeline. It is a diagnostic message — one that gathers information rather than acting on it — but it is no less critical for being preparatory. The assistant's inspection of the checkpoint's internal structure reveals the shape and naming of the tensors that must be transformed for vLLM compatibility. This knowledge directly enables the conversion work that follows, and it exemplifies the careful, methodical approach required when bridging the gap between training frameworks and inference engines. In the broader narrative of this coding session, this message is the calm before the storm — the moment of assessment before the assistant discovers that even with a correctly formatted checkpoint, vLLM's EAGLE-3 integration with MLA attention will yield only a 15% acceptance rate, triggering a pivot to SGLang and an entirely new set of debugging challenges.