The Pivot: Why Synthetic Data from Model Reasoning Became Essential for EAGLE-3 Training

A Single Message That Redirected the Pipeline

In the middle of a complex machine learning engineering session, a brief user message arrived that fundamentally altered the trajectory of the work. The message read:

Dataset: to capture k2.5 thinking (I think we need to) lets do: from open-perfectblend, on just vllm infer every question, capture thinking and output. Each question should be fed independently to vllm, answer up do 8k tokens.

This was message 2840 in a long conversation spanning dozens of hours of development. To understand its significance, one must appreciate the state of the pipeline at that exact moment. The assistant had just completed an end-to-end EAGLE-3 training pipeline — a speculative decoding system for the massive 1-trillion-parameter Kimi-K2.5 Mixture-of-Experts model. The pipeline had been validated on 10 samples, scaled to 1000 samples, and had produced a working checkpoint. But the user recognized a critical flaw in the training data itself.

The Context: What Had Just Been Accomplished

Moments before this message, the assistant had delivered a comprehensive session summary ([msg 2839]) documenting the completion of the core EAGLE-3 training pipeline. The pipeline consisted of several stages: extracting hidden states from the verifier model (Kimi-K2.5) using the open-perfectblend dataset, then training a small draft model (2.5B parameters) to predict those hidden states. The extraction had processed 1000 samples — 503,000 tokens — in under 3 minutes after a 22.5-minute model load. The training had run 10 epochs across 9500 steps in 27.7 minutes on a single GPU.

The assistant's summary concluded with a "Next Steps" section that offered two paths: a "hero run" on rented B200/B300 hardware (6-18 hours, $180-900) or continuing locally by extracting more samples (10K-50K) for a better local model. The assistant was clearly satisfied with the pipeline's completion and was looking forward to scaling it up.

The Insight Behind the Message

The user's message cut through this momentum with a crucial realization. The training data used so far — the open-perfectblend dataset — consisted of human-written conversations. The hidden states extracted from Kimi-K2.5 during prefill of these conversations captured how the model processes human text, not how it generates its own reasoning. But the entire purpose of an EAGLE-3 draft model is to predict the verifier model's future tokens during autoregressive generation — specifically, to anticipate the model's chain-of-thought reasoning and final answers.

This distinction is subtle but critical. An EAGLE-3 draft model works by taking the verifier's hidden states from the most recent few layers and using them to predict what tokens the verifier would generate next. If the draft model is trained only on prefill hidden states from human-written text, it learns to predict continuations of human dialogue patterns. But what it really needs to learn is how Kimi-K2.5 itself thinks — the characteristic patterns of its reasoning, the structure of its <think> blocks, the way it transitions from analysis to answer.

The user's phrasing — "I think we need to" — reveals this was a reflective insight, not a command from a pre-existing plan. The parenthetical hesitation signals that the user was reasoning through the problem in real time, arriving at the conclusion that the current approach was fundamentally insufficient.

The Decision Architecture

The message makes several implicit decisions that shaped the next phase of work:

First, the decision to use the model's own outputs as training data. Instead of continuing to scale up the existing pipeline (extracting more samples from open-perfectblend), the user chose to create a new dataset by feeding each question from open-perfectblend through the vLLM inference server and capturing the model's actual responses. This is a form of self-distillation or synthetic data generation — using the target model to produce the training examples that the draft model will learn from.

Second, the decision to feed questions independently. The user explicitly specified "Each question should be fed independently to vllm." This is an important design choice. Batch processing would be more efficient but would introduce dependencies between responses. Independent feeding ensures each response is conditioned only on its own prompt, which is how the draft model will be used during inference — it needs to predict continuations of individual prompts, not interleaved conversations.

Third, the decision to capture up to 8K tokens of output. The 8K token limit is generous enough to capture Kimi-K2.5's full reasoning chains, which can be extensive for complex questions. The model is known to produce long <think> blocks before generating its final answer. This limit also matches the max-seq-len of 2048 used in the existing extraction pipeline — though notably 8K is larger, suggesting the user anticipated that the model's reasoning outputs would be longer than the original dataset entries.

Fourth, the decision to capture both "thinking and output." This is the core of the insight. The draft model needs to learn to predict the verifier's reasoning tokens (the <think> content) as well as the final answer. In Kimi-K2.5, reasoning is demarcated by special tokens — thinking (token 163606) and response (token 163607) — and the model's chain-of-thought appears between them. Capturing both parts means the training data will include these reasoning patterns, which is precisely what the draft model must learn to accelerate.

Assumptions Embedded in the Message

The message carries several assumptions, some explicit and some implicit:

The assumption that open-perfectblend contains questions suitable for eliciting reasoning. The dataset was originally chosen for its general conversation quality, but the user assumes its questions will trigger Kimi-K2.5 to produce substantive reasoning chains. This is a reasonable assumption — open-perfectblend is a diverse instruction-following dataset — but it's untested. Some questions might be too simple to trigger extended reasoning, producing short answers that don't provide rich training signal.

The assumption that vLLM inference at scale is practical. The user doesn't specify how many questions to process, but the assistant later interprets this as a 10K-sample run. Running 10,000 inference requests against an 8-GPU vLLM server serving a 1T-parameter model is a significant computational undertaking. Each request could take tens of seconds to generate 8K tokens, meaning the total run could span hours. The user's later instruction to use "up to 200 parallel prompts" ([msg 2849]) addresses this by adding concurrency, but the original message doesn't specify parallelism.

The assumption that the existing pipeline's later stages (hidden state extraction, training) can be reused. The user's message only addresses dataset generation, not the downstream pipeline. The implicit assumption is that once the synthetic data is generated, it can be fed into the existing 02_extract_hidden_states.py and 04_train.py scripts. This turned out to be correct — the assistant later adapted the extraction to use the new data — but it required additional work to reconstruct the token sequences with the correct special tokens.

What the Message Didn't Say

The message is notably concise. It doesn't specify:

The Assistant's Interpretation

The assistant's response ([msg 2841]) demonstrates a deep understanding of the user's intent. It immediately grasps the core insight: "The draft model needs to learn to predict the model's thinking patterns, not just generic conversation patterns." The assistant then outlines a clear plan:

  1. Feed each question to Kimi-K2.5 independently via vLLM inference
  2. Capture the full output (thinking + answer, up to 8K tokens)
  3. This becomes the training data
  4. Then extract hidden states from these complete conversations The assistant also correctly identifies the key technical challenge: reconstructing the full token sequence with the correct special tokens. The existing pipeline expected raw text, but the synthetic data needs to preserve the thinking and response token boundaries so the draft model learns to predict reasoning structure.

The Knowledge Flow

Input knowledge required to understand this message:

The Broader Significance

This message represents a classic moment in machine learning engineering: the realization that the training data distribution must match the inference distribution. The EAGLE-3 draft model would be used during autoregressive generation to predict Kimi-K2.5's next tokens. Training it on prefill hidden states from human text was training on the wrong distribution. The model needed to learn from Kimi-K2.5's own reasoning patterns — the characteristic ways it structures its thoughts, the length of its reasoning chains, the vocabulary it uses in its internal monologue.

This insight is particularly important for speculative decoding because the draft model's accuracy directly determines the speedup. If the draft model predicts tokens that the verifier would not have generated, those predictions are rejected and the speculation is wasted. Training on the verifier's actual outputs maximizes the probability that the draft model's predictions will be accepted.

The message also illustrates the iterative nature of ML engineering. The assistant had just celebrated completing the pipeline, but the user recognized that the pipeline was solving the wrong problem at the data level. This is not a failure — it's a natural progression. The first pass proved the pipeline worked mechanically; the second pass would make it work effectively.

Conclusion

Message 2840 is a turning point in the conversation. It redirects effort from scaling a flawed pipeline to fixing the fundamental data quality issue. The user's concise directive — capture Kimi-K2.5's actual thinking, not just human text — demonstrates a deep understanding of what makes speculative decoding work. The assistant's rapid comprehension and execution of this pivot shows the effectiveness of the human-AI collaboration: the user provides strategic insight, the assistant handles technical implementation. Together, they transformed a working pipeline into one that could actually produce a useful draft model.