The Pivot to Synthetic Data: Why EAGLE-3 Needed Kimi-K2.5's Own Reasoning

Introduction

In the course of building an EAGLE-3 speculative decoding pipeline for the 1-trillion-parameter Kimi-K2.5 model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, a critical realization emerged: the training data used to teach the draft model was fundamentally wrong. Message 2841 marks the moment when the assistant—and the user—recognized that capturing the model's own reasoning patterns was essential for effective speculative decoding, and pivoted from a generic dataset approach to a synthetic data generation strategy.

This article examines that single message in depth: the reasoning behind it, the assumptions it made, the knowledge it required, and the decisions it set in motion. It is a study of a turning point in a complex ML engineering session—a moment where technical understanding of how speculative decoding works reshaped the entire data pipeline.

The Message

The subject message (msg id=2841) reads:

You're right — we need to capture Kimi-K2.5's actual reasoning outputs (the thinking tokens), not just the prefill hidden states of the raw dataset. The draft model needs to learn to predict the model's thinking patterns, not just generic conversation patterns.

Let me think about the approach:

1. Feed each question to Kimi-K2.5 independently via vLLM inference 2. Capture the full output (thinking + answer, up to 8K tokens) 3. This becomes the training data — the model's own outputs are what the draft model needs to predict 4. Then extract hidden states from these complete conversations (prompt + model's response)

This means we need a new step 1 that: - Takes questions from open-perfectblend - Runs them through vLLM as inference requests - Saves the complete conversation (user prompt + model's full response including thinking) - Then the existing step 2 extracts hidden states from these complete conversations

Let me plan and build this.

The message also includes a todo update marking the new dataset generation script as "in_progress" and listing pending tasks: starting the vLLM server, running inference on open-perfectblend questions, and adapting the hidden state extraction step.

Context: What Came Before

To understand why this message was written, we need to look at the trajectory of the session. The assistant had just completed an end-to-end EAGLE-3 training pipeline on 1000 samples from the open-perfectblend dataset. This pipeline had three steps:

Tokenization (step 1): Convert raw text into tokenized sequences with loss masks
Hidden state extraction (step 2): Run the full sequences through the verifier model (Kimi-K2.5) and capture intermediate hidden states from layers 2, 30, 58, and 60
Training (step 3): Train the EAGLE-3 draft model to predict these hidden states given the preceding context The pipeline worked. The assistant had successfully extracted 503K tokens across 1000 samples at 2912 tok/s, then trained the draft model for 10 epochs in 27.7 minutes. The output checkpoint was verified to be vLLM-compatible with the correct flat config format and weight shapes matching the AQ-MedAI reference model. But there was a subtle problem—one that the user identified in message 2840:

Dataset: to capture k2.5 thinking (I think we need to) lets do: from open-perfectblend, on just vllm infer every question, capture thinking and output.

The user recognized that the existing pipeline was using the raw dataset text as the training signal. The open-perfectblend dataset contains human-written conversations, not Kimi-K2.5's own outputs. An EAGLE-3 draft model needs to predict the verifier model's hidden state trajectories—but if the training data consists of generic text that the model never actually generated, the hidden states extracted during step 2 correspond to the model's prefill of arbitrary text, not its generation of its own reasoning.

Why This Message Was Written: The Core Reasoning

The assistant's response in message 2841 reveals a deep understanding of how EAGLE-3 speculative decoding works. The key insight is captured in the first paragraph: "The draft model needs to learn to predict the model's thinking patterns, not just generic conversation patterns."

EAGLE-3 works by training a lightweight "draft" model to predict the hidden states of a large "verifier" model at specific intermediate layers. During inference, the draft model generates candidate tokens quickly, and the verifier validates them in parallel—a technique called speculative decoding. The acceptance rate—how often the verifier accepts the draft's tokens—depends critically on how well the draft model can predict the verifier's internal representations.

If the draft model is trained on hidden states extracted from arbitrary text (like human-written conversations), it learns to predict what the verifier's hidden states look like when processing generic language. But during inference, the verifier will be processing its own generated tokens—which have a very different distribution. A reasoning model like Kimi-K2.5 produces distinctive patterns: long chains of thinking tokens (token ID 163606), followed by response tokens (token ID 163607), then the final answer. The hidden state trajectories during self-generated reasoning are fundamentally different from those during prefill of human-written text.

This is why the assistant immediately agrees with the user's intuition and elaborates on it. The message is not just acknowledging the user's suggestion—it's articulating the mechanism by which the data quality affects draft model performance.

How Decisions Were Made

The message lays out a four-step plan, but it's more than a simple to-do list. The assistant is making several architectural decisions:

Decision 1: Feed each question independently. Rather than batching questions or using the existing tokenized dataset, each question from open-perfectblend is sent as a separate inference request to the vLLM server. This ensures that the model's generation isn't influenced by other conversations in the context—each response is a clean, independent sample of the model's reasoning.

Decision 2: Capture the full output up to 8K tokens. The 8K limit is a pragmatic choice. Kimi-K2.5's reasoning can be very verbose—some chains of thought extend for thousands of tokens. Setting a generous limit ensures that the training data captures complete reasoning chains rather than truncated fragments. The assistant later adjusted this to 8K based on the user's specification.

Decision 3: Insert a new step between the existing step 1 and step 2. Rather than rewriting the entire pipeline, the assistant correctly identifies that the existing hidden state extraction (step 2) and training (step 3) code can remain unchanged. The only change is what feeds into step 2: instead of tokenized raw text, it will receive tokenized model-generated conversations. This is a clean architectural decision that minimizes disruption.

Decision 4: The model's own outputs become the training data. This is the most important conceptual shift. The assistant explicitly states: "This becomes the training data — the model's own outputs are what the draft model needs to predict." This reframes the entire purpose of the data generation pipeline: it's not about representing the "truth" of a conversation, but about capturing the verifier's own generative distribution.

Assumptions Made

The message, and the work that followed it, rests on several assumptions:

Assumption 1: The draft model will perform better when trained on the verifier's own outputs. This is the core hypothesis driving the pivot. It's well-supported by the speculative decoding literature—EAGLE-3 papers and Baseten's published results both emphasize training on the target model's generations. But it's still an assumption until validated with empirical acceptance rate measurements.

Assumption 2: The open-perfectblend questions are a reasonable proxy for the kinds of prompts the draft model will encounter in production. The assistant doesn't question whether the dataset's questions are representative—it accepts the user's direction to use them. This is a reasonable starting point, but the user later raises the excellent point that their own usage data might be even better (message 2854).

Assumption 3: 8K tokens is sufficient to capture complete reasoning chains. For most prompts, 8K is generous. But for complex multi-step reasoning tasks, Kimi-K2.5 could theoretically generate longer chains. The assistant implicitly assumes that truncation at 8K will still capture the most important parts of the reasoning (the beginning, where the thinking pattern is established).

Assumption 4: The vLLM server's kimi_k2 reasoning parser correctly extracts the thinking and response segments. The assistant later discovers that the reasoning field is accessed via msg.reasoning (not reasoning_content), and that the special tokens need to be reconstructed manually. This assumption was partially incorrect, as we'll see.

Assumption 5: Running 200 concurrent requests (as the user later specified) won't overwhelm the server or cause quality degradation. The assistant doesn't question the concurrency level, but high concurrency on a PCIe-bound system could increase latency and potentially affect the generation quality if the model starts timing out.

Input Knowledge Required

To understand and act on this message, significant domain knowledge was required:

Knowledge of EAGLE-3 architecture. The assistant needed to understand that EAGLE-3 trains a draft model to predict intermediate hidden states of the verifier, and that the training data must therefore come from the verifier's own generative distribution. This is not obvious—many speculative decoding implementations train on arbitrary text.

Knowledge of Kimi-K2.5's special token structure. The model uses thinking (token 163606) and response (token 163607) to delimit reasoning and answer segments. The assistant later needed to reconstruct these tokens in the output format, which required knowing the exact token IDs.

Knowledge of the vLLM OpenAI-compatible API. The assistant knew how to query the vLLM server, what parameters to set (max_tokens=8192), and how to extract reasoning from the response object.

Knowledge of the existing pipeline structure. The assistant knew that steps 2 and 3 could be reused, and that only a new data generation step was needed. This required familiarity with the 02_extract_hidden_states.py and 04_train.py scripts.

Knowledge of the hardware constraints. The assistant knew that the vLLM server takes ~22 minutes to load the model, that the GPUs are PCIe-connected (limiting throughput), and that disk space on /root/ was limited (later moving output to /data/).

Output Knowledge Created

This message and the subsequent work created several valuable artifacts:

A new data generation script (01b_generate_synthetic.py) that feeds questions from open-perfectblend through the vLLM inference server, captures the full reasoning + answer output, and formats it for the hidden state extraction pipeline.

A corrected understanding of the vLLM response format. The assistant discovered that reasoning is accessed via msg.reasoning (not reasoning_content), and that the special tokens need to be explicitly inserted around the reasoning content. This knowledge was hard-won through debugging.

A timeout fix. The default 60-second client timeout was insufficient for long reasoning generations. The assistant increased it to 1800 seconds (30 minutes), which was necessary for the 8K token generations.

A reconstruction of the full token sequence. The assistant added logic to reconstruct the correct token sequence with thinking (token 163606) and response (token 163607) special tokens wrapping the reasoning content, ensuring the hidden state extraction step receives the exact same token sequence the model generated.

A 10K-sample generation run. The user capped the run at 10K samples and redirected output to the 3TB /data volume, creating a substantial training dataset for the draft model.

The Thinking Process Visible in the Message

The message reveals a structured, methodical thinking process. The assistant doesn't just say "okay, let's do that"—it breaks down the problem:

Identify the core problem: "The draft model needs to learn to predict the model's thinking patterns, not just generic conversation patterns." This reframes the user's intuition into a precise technical requirement.
Enumerate the steps: The numbered list (1-4) shows the assistant thinking through the data flow: question → inference → capture → extract hidden states. Each step builds on the previous one.
Identify what changes: "This means we need a new step 1." The assistant recognizes that the existing pipeline's step 1 (tokenization of raw text) is insufficient, and a new inference-based step must precede it.
Plan the implementation: The todo list shows the assistant thinking ahead: write the script, start the server, run inference, adapt step 2. This is a practical execution plan, not just a conceptual design. The message also shows the assistant's characteristic thoroughness. It doesn't assume the existing pipeline can be reused without modification—it explicitly notes that step 2 needs to be adapted to handle the new data format.

Mistakes and Refinements

While the message itself is sound, the subsequent execution revealed two issues:

Timeout underestimation. The assistant initially used the default 60-second client timeout, which was far too short for 8K token generations from a 1T-parameter model on PCIe-bound GPUs. This was corrected to 1800 seconds.

Reasoning field name. The assistant initially checked for reasoning_content in the response, but the correct attribute was reasoning. This required a code fix after the first test run.

These are minor issues that were quickly corrected. The core insight—that synthetic data from the model's own generations is superior to raw text—was correct and drove the entire subsequent data generation effort.

Conclusion

Message 2841 represents a critical pivot point in the EAGLE-3 training pipeline. It transformed the data generation approach from "tokenize any text" to "capture the model's own reasoning." This shift reflects a deep understanding of how speculative decoding works: the draft model must predict the verifier's hidden states during generation, not during prefill of arbitrary text.

The message is a study in effective technical communication. The assistant acknowledges the user's insight, elaborates on the reasoning behind it, lays out a clear plan, and begins execution. It demonstrates that building effective ML systems requires not just implementing pipelines correctly, but understanding the distributional properties of the data at each stage.

For anyone building speculative decoding systems, this message contains a crucial lesson: the training data for your draft model must come from the target model's own generative distribution. Generic text, no matter how high-quality, cannot teach the draft model to predict the unique hidden state trajectories that arise during self-generated reasoning.