The Data Quality Pivot: From Pipeline Completion to Synthetic Reasoning Data in EAGLE-3 Training

Introduction

In the arc of a complex machine learning engineering session, there are moments of quiet triumph—when a pipeline that has been built, debugged, and validated finally runs end-to-end. And then there are moments of strategic redirection—when the user looks at the working pipeline and recognizes that it is solving the wrong problem at the data level. This chunk captures precisely such a redirection.

The assistant had just completed an extraordinary engineering feat: an end-to-end EAGLE-3 speculative decoding training pipeline for the 1-trillion-parameter Kimi-K2.5 Mixture-of-Experts model, validated on 10 samples and scaled to 1000. The hidden state extraction ran at 2912 tok/s, the training completed 10 epochs in 27.7 minutes, and the output checkpoint was verified to be vLLM-compatible with weight shapes matching the AQ-MedAI reference model. The session summary in [msg 2839] documented this milestone with precise metrics and a forward plan for a "hero run" on rented B300 hardware.

But the user saw deeper. In [msg 2840], they redirected the entire approach with a concise directive: "Dataset: to capture k2.5 thinking (I think we need to) lets do: from open-perfectblend, on just vllm infer every question, capture thinking and output." This single message recognized a critical flaw—the training data consisted of hidden states from the model's prefill pass on human-written text, not from the model's own autoregressive reasoning outputs. The draft model needed to learn to predict Kimi-K2.5's actual thinking patterns, not generic conversation continuations.

This chunk covers the pivot from pipeline validation to high-quality synthetic data generation: writing the inference script, fixing timeout and field extraction issues, planning a 25K-sample run, downloading the AQ-MedAI checkpoint for finetuning, updating the training script for finetuning support, and launching the large-scale inference run—only to discover new challenges at scale.## The Completed Pipeline: A Foundation Worth Building On

Before the pivot, the assistant had accomplished what many ML engineering teams would consider a significant milestone. The EAGLE-3 training pipeline, built from scratch using the speculators library (v0.3.0), had been validated end-to-end. The pipeline consisted of four stages: data preparation (01_prepare_data.py), hidden state extraction (02_extract_hidden_states.py), vocabulary mapping (03_build_vocab_mapping.py), and training (04_train.py).

The journey to this point had been arduous. The assistant had explored the speculators library's source code to understand the correct API usage—discovering that Eagle3SpeculatorConfig, Eagle3DraftModel, and the built-in Trainer class were the correct interfaces, not the custom training loop initially written. It had monkey-patched the verifier weight extraction to handle Kimi-K2.5's nested configuration structure (where weights are prefixed with language_model.), a structural incompatibility that would have prevented the pipeline from running at all. It had debugged a bf16 dtype mismatch between the model's float32 weights and the hidden states, fixed an unexpected TransformTensors API difference, and set _attn_implementation to "flex_attention" to satisfy EAGLE-3's BlockMask mechanism.

The 10-sample validation run completed in approximately one minute for 3 epochs, confirming the pipeline was mechanically sound. The 1000-sample scale-up then demonstrated production viability: the 547GB model loaded across 8 GPUs in 22.5 minutes, hidden state extraction processed 502,983 tokens in 2.9 minutes at 2912 tok/s, and training ran 10 epochs across 9500 steps in 27.7 minutes at 6 steps/s on a single GPU. The output checkpoint was verified to have the correct flat config format (LlamaForCausalLMEagle3 architecture), matching weight shapes to the AQ-MedAI reference model, and the correct head_dim=128 parameter.

The assistant's session summary in [msg 2839] captured this achievement with characteristic precision: six numbered items tracing the progression from API exploration through implementation through validation through production-scale execution through documentation. The "Next Steps" section presented two paths—a "hero run" on rented B200/B300 hardware (6-18 hours, $180-900) or continued local work with more training data.

The Insight That Changed Everything

The user's message in [msg 2840] cut through this momentum with a crucial realization. The training data used so far—the open-perfectblend dataset—consisted of human-written conversations. The hidden states extracted from Kimi-K2.5 during prefill of these conversations captured how the model processes human text, not how it generates its own reasoning. But the entire purpose of an EAGLE-3 draft model is to predict the verifier model's future tokens during autoregressive generation—specifically, to anticipate the model's chain-of-thought reasoning and final answers.

This distinction is subtle but critical. An EAGLE-3 draft model works by taking the verifier's hidden states from the most recent few layers and using them to predict what tokens the verifier would generate next. If the draft model is trained only on prefill hidden states from human-written text, it learns to predict continuations of human dialogue patterns. But what it really needs to learn is how Kimi-K2.5 itself thinks—the characteristic patterns of its reasoning, the structure of its <think> blocks, the way it transitions from analysis to answer.

The user's phrasing—"I think we need to"—reveals this was a reflective insight, not a command from a pre-existing plan. The parenthetical hesitation signals that the user was reasoning through the problem in real time, arriving at the conclusion that the current approach was fundamentally insufficient.

The assistant's response in [msg 2841] demonstrates a deep understanding of the user's intent: "You're right—we need to capture Kimi-K2.5's actual reasoning outputs (the <think> tokens), not just the prefill hidden states of the raw dataset. The draft model needs to learn to predict the model's thinking patterns, not just generic conversation patterns." The assistant then outlined a clear plan: feed each question to Kimi-K2.5 independently via vLLM inference, capture the full output (thinking + answer, up to 8K tokens), and use these complete conversations as the training data for hidden state extraction.## Writing the Synthetic Data Generation Script

The assistant's first action was to check the vLLM server configuration and the open-perfectblend dataset format ([msg 2842]). The server was confirmed to be running on port 8000 with OpenAI-compatible API and the kimi_k2 reasoning parser that handles thinking blocks. Then, in [msg 2843], the assistant wrote 01b_generate_synthetic.py—a new script that feeds each question from open-perfectblend independently through the vLLM inference server at high concurrency.

The script's design reflected several key decisions. First, it used the OpenAI Python client library to communicate with the vLLM server, sending each question as a separate request. Second, it captured both the reasoning field (the model's chain-of-thought between thinking and response special tokens) and the content field (the final answer). Third, it reconstructed the full token sequence with the correct special tokens— thinking (token 163606) and response (token 163607)—wrapping the reasoning content, ensuring that the downstream hidden state extraction would preserve the model's reasoning structure.

The script was designed for scale: it supported configurable concurrency (default 128, later increased to 200 per the user's instruction in [msg 2849]), configurable max completion tokens (up to 8K), and configurable max samples. It wrote output as a JSONL file where each line contained the original question, the model's full response (reasoning + content), and the tokenized input_ids with loss_mask, ready for the hidden state extraction step.

Debugging the First Run: Timeout and Field Extraction Issues

When the assistant tested the script, two issues emerged that required immediate fixes. The first was a timeout problem: the default OpenAI client timeout of 60 seconds was far too short for long reasoning generations. Kimi-K2.5's chain-of-thought reasoning can take 30-60 seconds or more to generate 8K tokens, especially under the PCIe-bound AllReduce bottleneck that dominated decode time at 51.5%. The assistant increased the client timeout to 1800 seconds (30 minutes), ensuring that even the longest reasoning chains would have time to complete.

The second issue was more subtle: the script was checking reasoning_content to extract the model's reasoning, but the correct attribute was reasoning. This field name mismatch meant the reasoning content was not being captured at all—the script would get empty reasoning fields and only save the final content. The assistant fixed this by properly extracting reasoning from msg.reasoning instead of reasoning_content.

These two fixes—one obvious (timeout), one subtle (field name)—illustrate the iterative nature of integrating with real inference servers. The OpenAI-compatible API that vLLM exposes has its own conventions, and the reasoning field (used by models like Kimi-K2.5 that support chain-of-thought) is accessed differently than the reasoning_content field used by some other models. The assistant had to discover this through testing.

The 25K Sample Inference Run

With the script fixed, the assistant prepared for a large-scale inference run. The user specified several parameters in [msg 2870]: run locally, use the /data volume (a newly available 3TB disk), target 25K samples, and keep training on a single GPU. The assistant updated its todo list in [msg 2871] and began orchestrating the run.

The first step was verifying that /data was available and had sufficient space. A quick df -h check in [msg 2872] confirmed 2.8 TB free on /dev/rbd0—plenty of room for the hidden states that would be generated later. The vLLM server was checked and confirmed ready in [msg 2873], with the model loaded and responding to API requests.

In parallel with starting the inference run, the assistant also began downloading the AQ-MedAI/Kimi-K2-Instruct-eagle3 checkpoint in [msg 2874]. This was a critical strategic decision: instead of training the draft model from scratch with random weights, the assistant planned to finetune from this existing checkpoint. The AQ-MedAI model was trained for Kimi-K2 (the predecessor to K2.5) with the same architecture, same hidden_size (7168), same auxiliary layers [2, 30, 58], and same weight shapes. Finetuning from this checkpoint would converge faster and likely achieve higher acceptance rates with less data.

The inference run was launched in [msg 2876] with the command:

nohup /root/ml-env/bin/python3 /root/eagle3-train/01b_generate_synthetic.py \
    --model-path /shared/kimi-k2.5-int4 \
    --output-dir /data/eagle3/synth_25k/prepared \
    --dataset mlabonne/open-perfectblend \
    --max-samples 25000 \
    --max-completion-tokens 8192 \
    --max-seq-len 16384 \
    --concurrency 200 \
    --vllm-url http://localhost:8000 \
    --min-completion-tokens 50

The initial monitoring in [msg 2878] showed the GPUs pegged at 100% utilization, with 200 concurrent requests all generating their first completions. The early throughput was low (0.6 req/s) because all 200 requests were still in-flight generating long first responses. The download of the AQ-MedAI checkpoint initially failed because huggingface-cli wasn't in the PATH for the nohup shell—a quick fix in [msg 2879] using the full path /root/ml-env/bin/huggingface-cli resolved this.## Monitoring Progress and Discovering Scale Issues

As the inference run progressed, the assistant monitored its throughput and adjusted expectations. In [msg 2888], the log showed 150/25000 samples completed at 0.9 req/s with an average completion of 648 tokens. The throughput in tokens per second, measured via vLLM's metrics endpoint in [msg 2891], was approximately 1,600 tok/s—close to the previously measured peak of 1,536 tok/s at C=128 concurrency.

The assistant calculated the estimated total time in [msg 2892]: with an average completion of ~650 tokens across 25K samples, that's ~16.25M generation tokens. At 1,600 tok/s, that's approximately 2.8 hours. However, the assistant noted that the average completion might increase as the dataset presented harder questions later, potentially extending the runtime.

The AQ-MedAI checkpoint download completed successfully in [msg 2880], with all four files fetched in 5 seconds. The assistant verified the checkpoint in [msg 2881], listing the weight keys and confirming the midlayer.* naming convention (which would need remapping to layers.0.* for the speculators library's Eagle3DraftModel). The checkpoint was 2.38 GB—a 2.5B-parameter draft model.

With the download complete and inference running, the assistant turned to updating 04_train.py to support finetuning from the AQ-MedAI checkpoint. In [msg 2883], it read the current training script and identified the needed changes: add a --finetune-from argument, load the AQ-MedAI weights into the trainable layers after model creation, and handle the midlayer.* → layers.0.* key remapping. The edits were applied in [msg 2884] through [msg 2886], and the updated script was copied to the remote server in [msg 2887].

The Timeout Problem at Scale

The user's monitoring in [msg 2899] revealed a growing problem. The generation throughput was showing significant variability—dropping from 1,572 tok/s to as low as 183 tok/s at times, with occasional prompt processing spikes. More critically, the inference log showed a pattern of timeout errors:

ERROR sample 17004: Request timed out.
Progress: 2680/25000 (222 errors), 0.5 req/s, avg completion: 1263 tokens, elapsed: 5945s
ERROR sample 7214: Request timed out.
ERROR sample 7216: Request timed out.
Progress: 2690/25000 (224 errors), 0.5 req/s, avg completion: 1262 tokens, elapsed: 5984s

The timeout errors were accumulating—224 errors by the time 2690 samples had been processed. The user's observation was precise: "maybe tune to 128 parallel and increase timeout?" This suggested that the 1800-second timeout, while generous, was still being hit by some requests. The concurrency of 200 might be too aggressive, causing requests to queue up and exceed the timeout window before they even started generating.

This is a classic scaling issue in large-scale inference workloads. At C=200 concurrency, the vLLM server's scheduler has to manage 200 concurrent requests, each potentially generating up to 8K tokens. If the server's generation throughput is ~1,600 tok/s, and each request averages 1,263 tokens of completion, the time to complete 200 requests is approximately 200 × 1,263 / 1,600 = ~158 seconds. But if some requests have much longer completions (approaching the 8K max), they could take 200 × 8,000 / 1,600 = 1,000 seconds—well within the 1800-second timeout but causing other requests to queue behind them.

The 224 errors out of 2690 samples (~8.3% error rate) suggested that a small fraction of requests were either timing out due to the queue depth or failing for other reasons (network issues, server-side errors). The user's suggestion to reduce concurrency to 128 and increase the timeout further was a pragmatic response: lower concurrency reduces queue depth and server load, while a longer timeout ensures that even the longest reasoning chains have time to complete.

The Broader Significance of This Chunk

This chunk captures a fundamental tension in ML engineering: the difference between a pipeline that works and a pipeline that produces good results. The assistant had built a technically impressive pipeline—validated end-to-end, with verified checkpoint compatibility, running at production-relevant speeds. But the user recognized that the pipeline was only as good as its training data, and the training data was fundamentally misaligned with the inference task.

The pivot from prefill hidden states to model-generated reasoning outputs represents a deep understanding of what makes speculative decoding work. The draft model's job is to predict the verifier's future tokens during autoregressive generation. Training it on prefill hidden states from human-written text is training on the wrong distribution. The model needs to learn from Kimi-K2.5's own reasoning patterns—the characteristic ways it structures its thoughts, the length of its reasoning chains, the vocabulary it uses in its internal monologue.

This insight is particularly important for speculative decoding because the draft model's accuracy directly determines the speedup. If the draft model predicts tokens that the verifier would not have generated, those predictions are rejected and the speculation is wasted. Training on the verifier's actual outputs maximizes the probability that the draft model's predictions will be accepted.

The chunk also illustrates the iterative nature of ML engineering at scale. The assistant had just celebrated completing the pipeline, but the user recognized that the pipeline was solving the wrong problem at the data level. This is not a failure—it's a natural progression. The first pass proved the pipeline worked mechanically; the second pass would make it work effectively. The timeout errors at scale are another iteration in this cycle: the script works at small scale, but at 25K samples with 200 concurrent requests, new failure modes emerge that require tuning.

Conclusion

This chunk represents a pivotal transition in the EAGLE-3 training project. The assistant had built a working pipeline, but the user's strategic insight redirected the effort from scaling a flawed approach to fixing the fundamental data quality issue. The synthetic data generation script, the AQ-MedAI checkpoint download, the finetuning support in the training script, and the 25K inference run all represent steps toward a higher-quality training dataset that captures Kimi-K2.5's actual reasoning patterns.

The timeout errors at scale are a reminder that even well-designed systems encounter new failure modes when pushed to production scale. The user's observation—"maybe tune to 128 parallel and increase timeout?"—demonstrates the kind of operational intuition that comes from experience with large-scale inference workloads. The assistant's response to this feedback, and the subsequent tuning of the inference parameters, would determine whether the 25K sample run succeeds or requires another iteration.

In the broader narrative of deploying speculative decoding for a 1T-parameter model, this chunk marks the moment when the team stopped asking "does the pipeline work?" and started asking "does the pipeline produce good results?" That shift in focus—from mechanical correctness to data quality—is what separates a proof-of-concept from a production system.## References

[1] "The Architecture of Awareness: How an AI Assistant Writes Its Own Operating Picture" — Analysis of the assistant's session summary in [msg 2839], examining how the assistant synthesized the completed pipeline work into a structured narrative.

[2] "The Pivot: Why Synthetic Data from Model Reasoning Became Essential for EAGLE-3 Training" — Deep analysis of [msg 2840], the user message that redirected the approach from pipeline scaling to data quality.

[3] "The Moment of Truth: EAGLE-3 Training Completes on a 1T-Parameter Model" — Analysis of [msg 2829], the training completion announcement that preceded the data quality pivot.

[4] "The Moment of Verification: Validating an EAGLE-3 Draft Model After 27 Minutes of Training" — Analysis of [msg 2830], the checkpoint verification that confirmed the pipeline produced vLLM-compatible output.

[5] "The Documentation Pivot: Consolidating EAGLE-3 Training Success into a Forward Plan" — Analysis of [msg 2836], the notes update that bridged pipeline completion to the next phase.

[6] "The Meta-Cognitive Assistant: How a Todo List Becomes a Window into AI Reasoning" — Analysis of [msg 2838], the todo list that tracked task completion across the pipeline build.

[7] "The Quiet Handoff: An SCP Command That Closed the Loop on EAGLE-3 Training" — Analysis of [msg 2837], the file transfer that delivered the updated orchestrator to the production environment.