The EAGLE-3 Awakening: From a Silent Bug to a 10× Data Scaling Pipeline
Introduction
In the high-stakes world of large language model inference optimization, few things are as frustrating as a bug that silently nullifies weeks of work. This is the story of such a bug—a single incorrect flag that rendered an entire speculative decoding system useless—and the sprawling, multi-threaded effort to not only fix it but to address the root cause of the performance gap that remained.
This chunk of an opencode coding session captures two parallel narratives that converge on a single goal: making EAGLE-3 speculative decoding work efficiently for the Kimi-K2.5 model. The first narrative is a detective story, tracing a devastating bug to a one-word flag mismatch and benchmarking the fix. The second is a data scaling odyssey, dispatching ten parallel search agents to scour the internet for training datasets that could close the remaining performance gap. Together, they illustrate the iterative, multi-faceted nature of real-world ML engineering—where the path from a research paper to a production deployment is paved with debugging, benchmarking, and data curation.
The Silent Killer: A One-Flag Bug
The EAGLE-3 architecture is an elegant approach to speculative decoding. Instead of the draft model predicting entire tokens from scratch, it predicts the target model's hidden states at intermediate layers. The target model captures hidden states from layers [2, 30, 58], concatenates them into a 21504-dimensional vector, and passes them to the draft model. The draft model's fusion layer (fc) then processes these concatenated states to predict the next token's hidden state, which the target model can accept or reject.
But for this to work, the server must be started with the correct flag: --speculative-algorithm EAGLE3. The is_eagle3() check in the codebase is strict—only the exact string EAGLE3 triggers the target model to capture and concatenate those intermediate layer hidden states.
The server had been started with --speculative-algorithm EAGLE instead.
The difference is devastating. With EAGLE, the target model passes only the final-layer hidden states (7168 dimensions) to the draft model. The draft model's fc fusion layer, expecting 21504-dimensional input, is silently bypassed. All trained weights become dead code. The draft model's predictions are never accepted—the system achieves an effective acceptance length of exactly 1.0, meaning every token is verified and none are speculated. The draft model is, for all practical purposes, useless.
This is the kind of bug that is both infuriating and instructive. It is not a logic error, not a race condition, not a memory corruption. It is a configuration mismatch—a single word that separates working from broken. And because the system does not crash or produce obvious errors (the draft model still runs, the server still serves tokens), it can persist for days or weeks without detection, silently wasting compute and engineering effort.
Benchmarking the Fix
Once the flag was corrected to EAGLE3, the hidden states began arriving as the expected 21504-dimensional vectors. The draft model's fc layer engaged, and the system started working as designed. The acceptance length jumped from 1.0 to approximately 2.1—meaning the draft model could now predict roughly two tokens ahead before the target model needed to verify.
But the throughput numbers told a more sobering story. The best EAGLE-3 configuration (CUDA graphs enabled, 5 draft tokens) achieved 82.3 tokens per second. The non-speculative baseline—running the target model directly without any draft model—achieved 90 tok/s. The speculative decoding system was still 9% slower than running without speculation.
This is the paradox of speculative decoding: the draft model adds overhead (running the draft model itself, transferring hidden states, managing the speculation buffer), and this overhead must be offset by the acceptance rate. An acceptance length of ~2.1 was insufficient to overcome the overhead. The EAGLE-3 paper's scaling curves suggested that the primary lever for improvement was more training data—specifically, data that closely matches the target model's token distribution.
The team also tested the AQ-MedAI drafter with the correct EAGLE3 flag. It performed worse, at 50.5 tok/s, confirming that the custom K2.5-trained drafter was better but still data-limited. The path forward was clear: scale the training data by an order of magnitude.
The Data Scaling Imperative
The insight that drove the next phase of work was both simple and profound: the draft model was underperforming because it hadn't seen enough examples of what Kimi-K2.5's hidden states look like. The existing training dataset was small—a few thousand examples generated directly from Kimi-K2.5. To reach the acceptance rates promised by the EAGLE-3 paper, the team needed to scale up by roughly 10×.
But generating 10× more data from Kimi-K2.5 directly would take days of inference time on the production server. A more efficient approach was to find existing datasets that were already produced by models with similar token distributions—DeepSeek-R1, DeepSeek-V3, and other MoE reasoning models whose output statistics approximate those of Kimi-K2.5.
This is where the session's focus shifted from debugging and benchmarking to large-scale dataset discovery. The user dispatched a subagent with a precise mission: search for datasets specifically created for or derived from DeepSeek models, Kimi models, or similar MoE reasoning models, with the goal of finding data whose token distribution closely matches what Kimi-K2.5 produces.
The Dataset Search Campaign
The subagent's search unfolded in three carefully orchestrated rounds, each building on the insights of the previous one. This iterative approach—broad exploration, targeted follow-up, and specific verification—mirrors the methodology a skilled human researcher would use, but executed at machine speed with parallel tool calls.
Round One: Mapping the Territory
The first round ([msg 1]) launched five parallel web searches covering the major entry points: "DeepSeek R1 dataset huggingface 2025," "DeepSeek V3 training data huggingface dataset," "Kimi dataset huggingface reasoning model," "DeepSeek distillation dataset synthetic reasoning huggingface," and "reasoning model synthetic dataset huggingface 2025 MoE." These queries were designed for breadth, casting a wide net across the Hugging Face ecosystem to understand what was available.
The results were informative but not directly actionable. The searches returned model pages (deepseek-ai/DeepSeek-R1, deepseek-ai/DeepSeek-V3, moonshotai/Kimi-K2-Thinking) rather than dataset pages, along with a blog post about fine-tuning DeepSeek-R1 and a Hugging Face collection of reasoning datasets. The assistant had found signals, but not the datasets themselves.
Round Two: Following the Leads
The second round ([msg 2]) demonstrated the assistant's ability to read between the lines of search results. The assistant recognized that the first round's results contained references to specific datasets and collections that deserved direct investigation. Six parallel searches targeted these leads: the Open-R1 project's datasets (Mixture-of-Thoughts, OpenR1-Math-220k), the crownelius collection of reasoning datasets, PrimeIntellect's SYNTHETIC-2, simplescaling's s1K, Kimi K2 training data, and specific datasets like Bespoke-Stratos-17k and Sky-T1.
This round was significantly more productive. The searches returned concrete dataset pages: open-r1/Mixture-of-Thoughts (349k examples, DeepSeek-R1 generated), PrimeIntellect/SYNTHETIC-2 (105k verified SFT examples), simplescaling/s1K (1k examples with DeepSeek-R1 thinking trajectories), and bespokelabs/Bespoke-Stratos-17k (16.7k examples with explicit thought markers).
Round Three: Drilling Down
The third round ([msg 3]) continued the pattern of targeted retrieval, searching for specific datasets that had been mentioned or referenced in earlier results: OpenThoughts-114k, the AM-DeepSeek-R1-Distilled dataset (3.34 million examples), and a combined search for Bespoke-Stratos-17k and Sky-T1.
Each query was a bet on a specific lead from the previous rounds. The assistant was not just collecting data—it was validating hypotheses about where relevant datasets might exist.
The Dataset Cartographer's Report
The culmination of this three-round search campaign was a comprehensive report ([msg 4]) that presented twelve candidate datasets in a structured, ranked format. The report was far more than a list—it was a strategic analysis that synthesized raw search results into actionable intelligence.
Each dataset entry followed a consistent template: HuggingFace ID, approximate size, format, content description, availability, generator model, and a "why relevant" justification. This consistency made the report scannable and comparable across multiple dimensions simultaneously.
The datasets were ranked by relevance to Kimi-K2.5's token distribution, with the top three being datasets directly generated by Kimi models:
crownelius/KimiK2.5-2000x(2.1k examples) — Directly generated by Kimi-K2.5 itself, making it the closest token distribution match despite its small size.SWE-Factory/DeepSWE-Agent-Kimi-K2-Trajectories-2.8K(2.8k examples) — Agentic coding trajectories from Kimi-K2, capturing the exact reasoning patterns of the model family.crownelius/Gemini-3-Pro-Opus-4.5-Kimi-K2.5-13000x(13.9k examples) — Mixed-model outputs including Kimi-K2.5 alongside other frontier models. The report's key takeaway was a pragmatic compromise: the Kimi-native datasets are the closest distribution match but are very small. For practical scale, the DeepSeek-R1 distillation datasets (ranked #4–#8) are the best proxies, sharing the same MoE architecture family with similar reasoning token patterns.
From Discovery to Production: The Inference Pipeline
The dataset search was not an academic exercise. The identified datasets fed directly into a production inference pipeline designed to regenerate responses through Kimi-K2.5, ensuring exact token distribution match for training the EAGLE-3 draft model.
Ten datasets were ultimately selected and prepared, totaling 88,088 samples. Of these, 4,800 were already tokenized in Kimi-native format, while 83,288 were prompts that needed inference—requiring the baseline SGLang server to generate responses through Kimi-K2.5.
The inference pipeline was launched on the baseline SGLang server, which was already running the target model. At approximately 830 tokens per second throughput, the pipeline was expected to process all 83K prompts in 24–55 hours. A live progress monitor script was created to track this long-running process, and the full pipeline plan was documented in train_plan_v4.md.
This is the moment where the two narratives converge. The bug fix made the draft model work; the benchmarking revealed it was data-limited; the dataset search identified the sources; and the inference pipeline began generating the training data that would, hopefully, close the remaining 9% performance gap.
Lessons and Implications
Several themes emerge from this chunk that are relevant to anyone working on LLM inference optimization.
First, configuration bugs are the most dangerous kind. The EAGLE vs EAGLE3 flag mismatch produced no errors, no crashes, no obvious signs of malfunction. The system ran, served tokens, and appeared to work—all while silently discarding the draft model's trained weights. This type of bug can persist indefinitely, wasting compute and engineering time, and it highlights the importance of runtime validation that checks not just that components are running, but that they are running correctly.
Second, benchmarking reveals uncomfortable truths. The 82.3 tok/s vs 90 tok/s baseline was a hard number that could not be argued with. The draft model, despite being correctly wired and having trained weights, was making the system slower. This forced the team to confront the data limitation rather than celebrating the bug fix. Good benchmarks are uncomfortable by design.
Third, data is the ultimate lever. The EAGLE-3 paper's scaling curves pointed to training data size as the primary factor in acceptance rate. The team's response—scaling up by 10× through a combination of existing dataset discovery and targeted inference generation—reflects a pragmatic understanding that in ML, the question "what should we train on?" is often the most consequential one.
Fourth, iterative search is a skill. The three-round dataset search campaign demonstrates that effective information retrieval is not about making the most queries, but about making the right queries in the right sequence. The assistant started broad, identified leads, followed them, and synthesized the results into a ranked taxonomy. This is the same pattern a skilled human researcher would follow, but executed with machine speed and parallelism.
Conclusion
This chunk of the opencode session captures a pivotal transition in a complex ML engineering project. The EAGLE-3 bug fix was necessary but insufficient—it made the system work correctly, but not efficiently. The real work lay in recognizing that the draft model was data-limited, formulating a strategy to scale the training data by 10×, and executing that strategy through a combination of web search, dataset curation, and inference pipeline deployment.
The story is a reminder that in machine learning, the gap between a research paper's results and production deployment is bridged by exactly this kind of painstaking, multi-faceted work. The bug fix was a single line change, but the data scaling effort involved ten parallel agents, three rounds of iterative search, twelve candidate datasets, 88,088 samples, and an inference pipeline expected to run for 24–55 hours. The 9% performance gap would not be closed by a clever algorithm tweak or a hyperparameter adjustment—it would be closed by more and better data.
In the end, the most important insight from this chunk is not about EAGLE-3 or speculative decoding or even Kimi-K2.5. It is about the nature of engineering progress: sometimes the biggest breakthroughs come not from building something new, but from recognizing what was already broken, measuring what was already insufficient, and finding the data that was already out there.