Two Tracks to Better Speculative Decoding: Debugging EAGLE-3 and Scaling Training Data

Introduction

In the high-stakes world of large language model deployment, the difference between a working system and a broken one can be as small as a single flag. This chunk of an opencode coding session captures a pivotal moment in the deployment of the GLM-5-NVFP4 model with EAGLE-3 speculative decoding, where the team pursued two parallel tracks simultaneously: diagnosing a subtle but crippling bug in the EAGLE-3 implementation, and scaling up the training dataset by an order of magnitude to improve draft model quality. The interplay between these tracks—one reactive, one proactive—reveals the iterative, multi-threaded nature of real-world ML engineering.

The EAGLE-3 Hidden State Concatenation Bug

The first and most urgent thread in this chunk was the resolution of a critical bug that had been blocking all draft model deployment. The EAGLE-3 speculative decoding architecture works by having the target model (in this case, GLM-5-NVFP4) capture hidden states from intermediate layers—specifically layers [2, 30, 58]—and concatenate them into a rich 21504-dimensional representation that the draft model uses to predict future tokens. This concatenated representation is the key innovation of EAGLE-3, allowing the draft model to access multi-level semantic information rather than just the final layer's output.

The root cause of the bug was traced to a simple but devastating flag mismatch. The server had been started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in the codebase is strict—only the exact string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states. With the EAGLE flag, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states. This mismatch had two catastrophic effects: the fc fusion layer in the draft model was silently bypassed, and all trained weights became effectively useless. The draft model was operating on impoverished representations, unable to leverage the multi-level information that EAGLE-3 was designed to provide.

After restarting the server with the correct EAGLE3 flag, the hidden states began arriving as the expected 21504-dimensional vectors, and the draft model's predictions started being accepted by the target model. The acceptance length (accept_len) jumped from 1.0 (meaning virtually no tokens were accepted) to approximately 2.1, confirming that the fix was working.

Benchmarking the Fix: Progress and Limitations

With the bug fixed, the team benchmarked the EAGLE-3 configuration to measure real-world throughput gains. The best configuration—using CUDA graphs and 5 draft tokens—achieved 82.3 tokens per second. This was an improvement over the broken state, but it still fell short of the 90 tok/s non-speculative baseline. In other words, the speculative decoding was actually making the system slower by approximately 9%.

This counterintuitive result highlights a fundamental challenge in speculative decoding: the overhead of running both the draft model and the target model, combined with the verification step, can outweigh the benefits if the draft model's acceptance rate is not high enough. With an accept_len of approximately 2.1, the system was accepting about two tokens per speculation round, but the cost of generating and verifying those speculations was higher than simply running the target model directly.

The team also tested the AQ-MedAI drafter with the correct EAGLE3 flag, which performed even worse at 50.5 tok/s. This confirmed that the custom K2.5-trained drafter was better than the alternative, but both were fundamentally data-limited. The EAGLE-3 paper's scaling curve suggests that more training data is the primary lever for improving acceptance rates—a finding that directly motivated the second major thread of this chunk.

Scaling the Training Dataset by 10×

While the bug fix and benchmarking were underway, a massive parallel effort was launched to scale up the training dataset. The hypothesis was straightforward: if the draft model's acceptance rate was limited by the quality and diversity of its training data, then providing more data—specifically, data that better matches the target model's token distribution—should improve performance.

The team dispatched ten parallel subagents to search for large synthetic instruction datasets across multiple categories: agentic coding, reasoning, and general chat. Each subagent was given a specific mission to find datasets on HuggingFace, the dominant repository for open ML datasets. The search was systematic and iterative, covering major dataset families including Magpie, OpenHermes, Infinity Instruct, SlimOrca, Capybara, and others.

The subagent session that produced the dataset catalog is documented in detail across several articles. In [1], the initial search strategy is analyzed, showing how the assistant launched parallel web searches to maximize coverage. The assistant's decision to search across multiple queries simultaneously—rather than sequentially—reflects an understanding of the opencode session model's synchronous round structure, where all tool calls in a round are dispatched together and results arrive in the next round.

In [2], the second wave of searches targeted the remaining dataset families that the first wave had not covered, including SlimOrca, Capybara, and a compound query for WildChat, Tulu, UltraChat, and SmolTalk. This iterative refinement strategy—searching, reviewing results, and searching again—is characteristic of effective AI-assisted research.

The third round of searches, analyzed in [3], focused on specific high-value datasets that had been missed: Magpie-Pro-1M variants, Cosmopedia (31M samples), OpenCodeInstruct (5M samples), and the Tulu 3 SFT mixture. The assistant's decision to launch this additional round rather than immediately compiling results shows a sophisticated awareness of its own coverage gaps.

The final synthesis, examined in [4], organized twelve datasets into a tiered structure based on size: mega-scale (5M+ samples), large-scale (1M-5M), and large (500K-1M). The report included detailed metadata for each dataset—HuggingFace ID, size, format, source model, and license—transforming raw search results into actionable intelligence for the training pipeline.

From Dataset Discovery to Pipeline Execution

The dataset search yielded ten datasets totaling 88,088 samples. Of these, 4,800 were already tokenized in the Kimi-native format and could be used immediately. The remaining 83,288 prompts required inference—they needed to be processed through the Kimi-K2.5 model to regenerate responses that match the target model's token distribution. This is a critical step: for speculative decoding training, the draft model needs to learn from the target model's own outputs, not from generic instruction-response pairs.

An inference pipeline was launched on the baseline SGLang server, which was already serving the GLM-5-NVFP4 model at approximately 830 tokens per second throughput. Processing all 83,288 prompts through this pipeline was estimated to take between 24 and 55 hours, depending on response lengths and system load. A live progress monitor script was created to track the pipeline's status, and the full pipeline plan was documented in train_plan_v4.md.

The Broader Significance

This chunk represents a microcosm of the challenges in modern ML engineering. The EAGLE-3 bug fix demonstrates how a single misconfigured flag can silently break an entire system—the draft model appeared to run, but it was operating on fundamentally wrong inputs. The benchmarking results show that even a correctly configured speculative decoding system may not provide speedups if the draft model is undertrained. And the dataset scaling effort illustrates the data-centric nature of modern ML: when model architecture and inference infrastructure are optimized, the next bottleneck is almost always data.

The parallel structure of this work is also noteworthy. The bug fix and dataset scaling were pursued simultaneously, with different sub-teams (or subagents) working on different aspects of the problem. This mirrors the way real engineering teams operate: while one team diagnoses a production issue, another team works on the next improvement. The opencode session's ability to spawn subagents and run them in parallel enabled this multi-threaded workflow.

Conclusion

This chunk of the coding session captures a turning point in the GLM-5-NVFP4 deployment. The EAGLE-3 hidden state concatenation bug was identified and fixed, restoring the draft model to functional operation. Benchmarking revealed that while the fix worked, the draft model was still data-limited, achieving only 82.3 tok/s against a 90 tok/s baseline. The response was to scale up the training dataset by 10×, launching a systematic search for large synthetic instruction datasets and building an inference pipeline to regenerate responses through the target model. The results of this work—both the bug fix and the dataset expansion—position the team to make a second attempt at achieving meaningful speedups from speculative decoding, this time with a properly trained draft model operating on correct hidden state representations.## The Architecture of the Dataset Search

The dataset search effort deserves closer examination because it reveals the structured, methodical approach that characterizes effective AI-assisted research. The user's initial request ([msg 0]) was remarkably precise: it specified six dataset families to investigate, provided seven suggested search queries, and defined a reporting format with five required metadata fields. This level of specificity is not micromanagement—it is prompt engineering at its finest, providing enough structure to guide the assistant while leaving room for autonomous execution.

The assistant's response was equally methodical. Rather than attempting to search for everything at once, it organized the work into three waves of parallel searches. The first wave ([msg 1]) covered the broadest queries and the most prominent dataset families: a catch-all query for large synthetic datasets, plus targeted searches for Magpie, OpenHermes, and Infinity Instruct. The second wave ([msg 2]) covered the remaining named families: SlimOrca, Capybara, and a compound query for WildChat, Tulu, UltraChat, and SmolTalk. The third wave ([msg 3]) drilled into specific high-value datasets that had been missed: Magpie-Pro-1M, Cosmopedia, OpenCodeInstruct, and the Tulu 3 collection.

This three-wave strategy is a textbook example of iterative refinement. Each wave built on the results of the previous ones, filling in gaps as they became apparent. The assistant demonstrated meta-cognitive awareness of its own coverage—it recognized that certain prominent datasets were underrepresented and launched additional searches to find them. This is the kind of sophisticated research behavior that distinguishes effective AI assistance from simple information retrieval.

The Final Dataset Report

The culmination of the search effort was a comprehensive report ([msg 4]) that organized twelve datasets into three tiers. The report's structure reveals the assistant's understanding of how practitioners think about data: by scale, by source, and by license. The tiered organization helps readers quickly identify datasets in their target size range, while the summary table at the end provides a quick-reference cheat sheet.

The top recommendations were:

1. Infinity-Instruct (7.45M samples) — the largest general-purpose instruction set
2. OpenCodeInstruct (5M samples) — for code capability
3. Magpie 1M variants — high-quality synthetic data with quality metadata; combining multiple variants yields 4M+ total
4. OpenHermes-2.5 (1M samples) — battle-tested, diverse, GPT-4 distillation
5. Tulu 3 SFT mixture (939K samples) — carefully curated by AI2, state-of-the-art recipe These recommendations directly fed into the training pipeline, with ten datasets selected and prepared totaling 88,088 samples. The pipeline's goal was to regenerate responses through Kimi-K2.5, ensuring the training data matched the target model's token distribution—a critical requirement for effective speculative decoding training.## References [1] "The Data Hunt: Analyzing a Subagent's Mission to Find Large Synthetic Instruction Datasets" — Analyzes the initial search strategy and the user's precise request structure. [2] "The Second Wave: Parallel Search for Synthetic Instruction Datasets" — Examines the second round of searches covering SlimOrca, Capybara, and compound queries. [3] "The Third Round: Iterative Dataset Discovery in an AI-Assisted Research Session" — Details the third round of targeted searches for high-value datasets. [4] "The Map of Synthetic Instruction Data: How One AI Assistant Catalogued the Landscape of LLM Training Datasets" — Analyzes the final comprehensive dataset report and its tiered structure. [5] "The Art of Parallel Search: How an AI Agent Systematically Mapped the Landscape of Large Synthetic Instruction Datasets" — Examines the parallel search strategy and the reasoning behind it.