The Pivot Point: Documenting an EAGLE-3 Training Pipeline for Kimi-K2.5 Speculative Decoding

Introduction

In the sprawling, multi-week effort to deploy and optimize the 1-trillion-parameter Kimi-K2.5 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 2482] represents a quiet but decisive pivot. After an exhaustive profiling campaign had revealed that AllReduce communication consumed 51.5% of decode time — a bottleneck fundamentally rooted in PCIe interconnect limitations rather than compute — the team had turned to speculative decoding as a software-only path to improved throughput. The assistant had just completed a deep research phase, producing a comprehensive report ([msg 2477]) that evaluated five options ranging from a five-minute n-gram experiment to a weeks-long custom EAGLE-3 training pipeline. The user's response ([msg 2479]) was characteristically pragmatic: try the quick n-gram test, but first, document what it would take to train a custom EAGLE-3 head. The subject message — a single tool call writing a file called next-steps-eagle.md — is the moment that documentation was created, marking the transition from research into the first concrete step toward the most ambitious path available.

The Message in Full

The message itself is deceptively brief:

Good — the model is loaded and ready, and I now have all the detail I need. Let me write the EAGLE-3 training doc and simultaneously test n-gram speculation.

>

[write] /home/theuser/glm-kimi-sm120-rtx6000bw/next-steps-eagle.md

>

Wrote file successfully.

>

LSP errors detected in other files: <diagnostics file="/home/theuser/glm-kimi-sm120-rtx6000bw/source/server_args_sm120.py"> ERROR [1:1] Unexpected indentation ERROR [20:1] Unindent not expected ...

Beneath this surface-level brevity lies a dense concentration of context, reasoning, and forward planning. The assistant is executing two parallel workstreams — writing the training document and preparing to test n-gram speculation — but the message only captures the first. The LSP diagnostics at the bottom are a red herring, reporting pre-existing errors in an unrelated SGLang source file (server_args_sm120.py) that had been modified earlier in the session for SM120 compatibility testing. They are irrelevant noise that the assistant's environment helpfully surfaces but that the assistant correctly ignores.

The Context That Made This Message Necessary

To understand why this message matters, one must appreciate the chain of reasoning that led to it. The profiling session in Segment 19 had delivered a sobering diagnosis: the Kimi-K2.5 INT4 deployment was fundamentally bottlenecked by PCIe-based AllReduce, which consumed over half of every decode step. This was not a problem that could be fixed with better kernel selection or NCCL tuning — it was a hardware topology constraint. The machine had eight GPUs connected via PCIe rather than NVLink, and the MoE architecture's expert parallelism meant that every decode step required synchronizing hidden states across all eight devices.

The assistant's research into speculative decoding ([msg 2477]) had been thorough and, in some respects, discouraging. The MoE-Spec paper — literally four days old at the time — had identified a fundamental problem: speculative decoding's parallel verification step activates far more unique experts than single-token decoding, turning what should be a "free" parallelism gain into a memory-bandwidth disaster for MoE models. The paper warned of "severe bottlenecks" and "diminishing speedups." Another paper on utility-driven speculative decoding reported that verification time could increase by 2–3x, causing slowdowns of up to 1.5x when token throughput gains failed to offset the overhead.

The research had also ruled out most off-the-shelf draft models. Kimi-K2.5 uses a vocabulary of 163,840 tokens, incompatible with DeepSeek's 129,280-token vocabulary. The only viable existing draft model was AQ-MedAI/Kimi-K2-Instruct-eagle3, but it was trained for the earlier K2 model, not K2.5, so acceptance rates would be lower. The Baseten team's approach — training a custom EAGLE-3 head on synthetic hidden states from the target model — emerged as the most promising path, exactly because it side-stepped the distribution mismatch problem.

The user's instruction in [msg 2479] was a masterclass in pragmatic decision-making: "Try option A, before tho, write down what it would take to train Option D in next-steps-eagle.md; Training - setup on the GPUs we have now, if it will look like it would take days I'll get a B200/B300 nvl8 machine to run the hero run." This directive accomplished several things simultaneously. It acknowledged that the custom training path (Option D) was the most promising long-term solution. It created a documented plan that could be executed on current hardware or ported to rented machines. And it established a clear decision criterion: if the training would take days on the current 8×RTX PRO 6000 setup, the user was willing to invest in B200/B300 NVL8 machines for the hero run.

What the Document Contained

The next-steps-eagle.md file, written in this single tool call, was the culmination of the parallel research task that had been launched in [msg 2481]. That task had investigated the Speculators v0.3.0 training pipeline from the vLLM project in exhaustive detail, covering the three-step process of data generation, hidden state extraction, and EAGLE-3 head training. The document would have included:

A draft model configuration matching the K2 EAGLE-3 architecture with a 32K draft vocabulary, designed to be a lightweight ~1B parameter head that could run alongside the 1T-parameter target model. A dataset preparation pipeline using HuggingFace datasets to reformat conversations, apply chat templates, and tokenize inputs. A hidden state extraction procedure using Speculators' VllmHiddenStatesGenerator to capture intermediate representations from three layers of the running Kimi-K2.5 model. A vocabulary mapping script to align the draft model's output distribution with the target model's 163,840-token vocabulary. A training script leveraging Speculators' Eagle3Trainer with knowledge distillation loss. And a shell orchestrator to sequence the entire pipeline from data preparation through training to deployment.

The document also would have addressed the hardware decision criterion explicitly: estimating training time on the current 8×RTX PRO 6000 setup versus on rented B200/B300 NVL8 machines, and providing the user with the information needed to make the call.

Assumptions Embedded in the Message

The assistant made several assumptions in writing this document, some of which would prove incorrect in the subsequent messages. The most significant was the assumption that the Speculators library (v0.3.0) would be compatible with the installed vLLM 0.16. The research had noted that Speculators was designed for vLLM ≤0.15, but the assistant proceeded with the assumption that the API differences would be manageable. This assumption would be tested — and found wanting — in the following chunk, when hidden state extraction failed due to API mismatches including the Kimi-K2.5 multimodal wrapper architecture (model.language_model.model.layers instead of model.model.layers) and SchedulerConfig parameter changes.

The assistant also assumed that hidden state extraction from the INT4-quantized model would work transparently. The Speculators pipeline was designed for full-precision models, and the interaction between INT4 quantization and the hidden state capture mechanism was untested at 1T scale. This was a genuine unknown that the document acknowledged but could not resolve without empirical testing.

A third assumption was that the training pipeline, once working, would transfer cleanly to B200/B300 hardware. The document was written to be hardware-agnostic, but the reality is that different GPU architectures have different optimal configurations for training — different batch sizes, different mixed-precision strategies, different NCCL configurations. The document provided a starting point, not a guaranteed recipe.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the parallel workstreams. The message opens with "Good — the model is loaded and ready" — this is the assistant verifying that the Kimi-K2.5 INT4 service is running and accepting requests (confirmed in [msg 2481] via a curl to the vLLM API endpoint). The assistant also notes "I now have all the detail I need" — this refers to the completed research task that returned the comprehensive Speculators pipeline documentation.

The phrase "Let me write the EAGLE-3 training doc and simultaneously test n-gram speculation" reveals the assistant's strategy of parallel execution. Rather than writing the document and then starting the n-gram test, the assistant intends to dispatch both workstreams concurrently. In practice, the message only shows the write tool call — the n-gram test would begin in the next message ([msg 2483]) when the assistant checks the running process's command line to see if speculative decoding can be added dynamically or requires a restart.

The LSP diagnostics at the bottom of the message are an artifact of the development environment. They report errors in a file called server_args_sm120.py that was part of an earlier SGLang SM120 compatibility investigation. These errors — import resolution failures, syntax issues — are pre-existing and unrelated to the current task. The assistant correctly ignores them, recognizing them as noise from a different workstream.

Input Knowledge Required

To understand this message, one needs to know that the Kimi-K2.5 model is a 1-trillion-parameter MoE architecture deployed across eight Blackwell GPUs connected via PCIe. One needs to understand that AllReduce was identified as the dominant bottleneck at 51.5% of decode time, and that speculative decoding was being investigated as a mitigation strategy. One needs to know the five-option taxonomy from the research report, and specifically that Option A was n-gram speculation (a five-minute experiment) and Option D was custom EAGLE-3 training (a one-to-two-week project). One needs to understand the EAGLE-3 architecture itself: a lightweight draft model that predicts multiple future tokens from intermediate hidden states of the target model, enabling parallel verification.

Output Knowledge Created

This message created the next-steps-eagle.md document — a concrete, actionable plan for training a custom EAGLE-3 speculative decoding head for Kimi-K2.5. The document served as both a reference for the immediate implementation work and a specification for the hero run on rented hardware. It established the data pipeline, the training configuration, the hardware requirements, and the decision criteria for upgrading to B200/B300 machines. In the broader context of the session, this document would become the blueprint for the entire EAGLE-3 implementation effort that occupied the remainder of Segment 20.

Significance

The subject message is, in many ways, the hinge point of the speculative decoding investigation. Before it, the work was research and analysis — understanding the problem space, evaluating options, reading papers. After it, the work becomes implementation — writing scripts, patching libraries, running experiments. The next-steps-eagle.md document is the bridge between these two phases, capturing the distilled knowledge from the research phase in a form that can guide the implementation phase. The fact that the assistant wrote this document in a single tool call, synthesizing hours of research into a coherent plan, speaks to the depth of understanding achieved in the preceding messages. The message is brief, but it carries the weight of everything that came before and everything that would follow.