The First Brick: Building the EAGLE-3 Draft Model Configuration

In the long arc of deploying and optimizing a 1-trillion-parameter Kimi-K2.5 model on 8×RTX PRO 6000 Blackwell GPUs, there comes a moment when the team pivots from profiling bottlenecks to building solutions. Message [msg 2523] captures that pivot in its most concrete form: the creation of the very first file in a custom EAGLE-3 speculative decoding training pipeline. It is a short message — a single write tool call producing a JSON configuration file — but it represents the culmination of extensive research, the rejection of dead ends, and the beginning of a new, ambitious engineering effort.

The Road to This Message

To understand why this message exists, one must trace the chain of reasoning that led to it. The session had just completed a deep profiling campaign of Kimi-K2.5 INT4 on 8×Blackwell GPUs, which revealed that AllReduce communication dominated decode time at 51.5% ([chunk 19.0]). The assistant then pivoted to investigate speculative decoding as a software-only optimization path that could bypass the PCIe bottleneck without requiring hardware changes.

The research phase was thorough. The assistant launched parallel subagent tasks to explore three dimensions: the fundamentals of speculative decoding for MoE models, the support in vLLM and SGLang frameworks, candidate draft models, and training feasibility ([msg 2507]). The findings were sobering. N-gram speculative decoding, the simplest approach, was tested empirically on the running vLLM instance and proved to be 9–26% slower than baseline ([msg 2504]). The reason was twofold: reasoning models like Kimi-K2.5 generate novel thinking chains with little repetitive n-gram structure, yielding acceptance rates of only 17–31%; and the MoE expert activation overhead during verification meant that every rejected draft token cost more than it saved. This was exactly the phenomenon predicted by recent MoE-Spec research.

The only viable off-the-shelf draft model was AQ-MedAI/Kimi-K2-Instruct-eagle3, a ~1B-parameter EAGLE-3 head trained for the earlier Kimi-K2, not K2.5. Using it would mean accepting lower acceptance rates due to distribution mismatch between the two model versions. The assistant concluded that the most promising path was the Baseten approach: training a custom EAGLE-3 head specifically for K2.5.

The user then gave a clear directive in [msg 2505]: "Start implementing the training scripts, on our existing machine with lowered numbers, then should be easy to port to much more expensive b300 machine for a simple train run. Write down notes as you go." This is the direct parent of message [msg 2523]. The assistant was told to build, not just research.

The Architecture Decision

Before writing any code, the assistant had to choose between two competing frameworks: SpecForge (from the SGLang project) and Speculators (from the vLLM project). Both support EAGLE-3 training with MoE verifiers. The assistant ran parallel research tasks ([msg 2507]) to examine both codebases at the source level, reading the training scripts, data generation pipelines, and model definitions.

The decision to use Speculators was announced in [msg 2509]: "I'll use Speculators (vLLM project) since we're already on vLLM and it has a cleaner pipeline." This was a pragmatic choice. The team already had vLLM 0.16.0rc2 deployed and running the Kimi-K2.5 model. Using the sibling project within the vLLM ecosystem minimized integration risk — or so it seemed at the time. (As later messages would reveal, speculators v0.3.0 was designed for vLLM ≤0.15, and the API mismatches with 0.16 would cause runtime errors that required extensive patching.)

The assistant then installed speculators into the existing Python environment ([msg 2512]), verified that the core imports worked ([msg 2513], [msg 2514], [msg 2515]), and read the key source files to understand the data format and training infrastructure ([msg 2517], [msg 2518], [msg 2519], [msg 2520]). By message [msg 2521], the assistant had a complete understanding and announced the plan: "Let me now build everything — I'll create a complete training pipeline directory with all the scripts."

What the Message Actually Does

Message [msg 2523] is the execution of that plan's first step. The assistant writes draft_config.json into the newly created eagle3-train/ directory. This JSON file defines the architecture of the EAGLE-3 draft model that will be trained to predict the target model's hidden states.

The configuration is based directly on the architecture of AQ-MedAI/Kimi-K2-Instruct-eagle3, which the assistant had thoroughly examined in a research task ([msg 2507]). That model uses a LlamaForCausalLMEagle3 architecture with a draft_vocab_size of 32,000 tokens and an eagle_config specifying auxiliary hidden states from layers [2, 30, 58] — a sparse set of three layers spread across the 60-layer model. The draft model has its own small transformer with 16 layers, a hidden size of 2048, and 16 attention heads.

The assistant's config adapts this architecture for Kimi-K2.5. The key parameters — draft vocabulary size, hidden dimensions, number of draft layers, the specific layers from which to extract auxiliary hidden states — are all carried over from the K2 EAGLE-3 model. This is a reasonable starting point: the K2 and K2.5 models share the same MoE architecture family, so the draft model architecture that worked for one should be a good prior for the other.

The LSP Diagnostics: A Distraction or a Signal?

The message also includes a block of LSP diagnostics showing errors in an unrelated file: source/server_args_sm120.py. These errors — indentation issues, unresolved imports, unclosed brackets — are pre-existing problems in a SGLang configuration file that was written earlier in the session. They have nothing to do with the draft config being created.

The inclusion of these diagnostics is a quirk of the assistant's tool-use interface: the LSP server reports errors across the entire workspace after every file write, and the assistant dutifully reports them. For the reader, this is a reminder that the workspace contains many experimental files, some of which were written hastily during debugging sessions and never cleaned up. The assistant does not act on these diagnostics, nor should it — they are in a file (server_args_sm120.py) that was part of an earlier SGLang deployment attempt that has since been superseded by the vLLM-based deployment.

Assumptions Embedded in This Message

Every engineering decision carries assumptions, and this message is no exception. The most significant assumption is that the K2 EAGLE-3 architecture is directly transferable to K2.5. The two models share the same underlying MoE structure (both are from the Kimi family), but K2.5 may have differences in layer count, hidden dimensions, or attention mechanisms that would require adjustments to the draft model. The assistant is betting that the architecture is close enough that training will converge.

A second assumption is that the speculators training infrastructure will work with the data generated from vLLM 0.16. The assistant had already noted the version constraint — speculators requires vLLM ≤0.15 for its data generation path — and was planning to work around it by using the offline LLM class with the worker extension. As later messages would show, this assumption was optimistic: the VllmHiddenStatesGenerator would fail at runtime due to API changes in the SchedulerConfig and KV cache utility functions.

A third assumption is that the training can be done on the existing 8×RTX PRO 6000 hardware. The user explicitly requested this ("on our existing machine with lowered numbers"), and the assistant's pipeline is designed to scale down the data generation from 500K–1.4M samples to something feasible on PCIe-connected GPUs. The assumption is that a smaller training run on slower hardware will produce a draft model that is at least directionally correct, and that the hero run can be ported to rented B200/B300 NVL8 machines.

The Knowledge Flow

The input knowledge required to write this message is substantial. The assistant needed to understand: the EAGLE-3 training paradigm (from the research papers and framework codebases), the exact architecture of the existing K2 EAGLE-3 model (from HuggingFace), the speculators library's configuration format (from reading the source), the target model's architecture (Kimi-K2.5 INT4), and the project's directory structure and conventions.

The output knowledge created by this message is the draft_config.json file itself, which serves as the architectural blueprint for the entire training pipeline. Every subsequent script — the data generator, the vocabulary mapper, the training loop — depends on the parameters defined in this config. It is the foundation upon which the rest of the pipeline is built.

The Broader Significance

In the context of the full session, message [msg 2523] is the moment when the speculative decoding effort transitions from research and planning to implementation. The assistant has spent dozens of messages profiling, researching, testing n-gram speculation, evaluating frameworks, and reading source code. Now it begins to build.

The pipeline that starts with this single JSON file will grow to include five scripts (dataset preparation, hidden state extraction, vocabulary mapping, training, and a shell orchestrator), will be tested end-to-end with a small sample of 10 prompts, will encounter and partially overcome API incompatibilities between speculators and vLLM 0.16, and will ultimately be documented in next-steps-eagle.md as a complete, reproducible training plan. But at this moment, in message [msg 2523], there is only one file: a JSON configuration, carefully modeled on proven architecture, representing the first brick of a new building.