The EAGLE-3 Bug and the 10× Data Rescue: A Tale of Speculative Decoding Debugging and Dataset Scaling

Introduction

In the high-stakes world of large language model deployment, few optimizations offer as tantalizing a promise as speculative decoding. The idea is elegant: use a small, fast "draft" model to generate candidate tokens that a larger, slower target model verifies in parallel, achieving substantial speedups without sacrificing output quality. But as this opencode session reveals, the gap between theory and practice is filled with subtle bugs, flag mismatches, and the relentless pursuit of better training data. This article chronicles a pivotal chunk of work in a coding session dedicated to deploying the GLM-5-NVFP4 model using SGLang on a machine with eight RTX PRO 6000 Blackwell GPUs. It covers the discovery and resolution of a critical EAGLE-3 hidden state concatenation bug, the sobering benchmarking results that followed, and the massive parallel effort to scale up training data by 10× through a coordinated subagent campaign.

The Bug That Wasn't a Bug: A Single Flag Change

For days, the team had been struggling with a perplexing problem. The EAGLE-3 draft model — trained with considerable effort on custom data — was producing predictions that were being rejected at an alarming rate. The acceptance length hovered around 1.0, meaning the target model was accepting only the first draft token before falling back to its own generation. At that rate, speculative decoding was not just failing to accelerate inference; it was actively slowing it down, since every rejected token represented wasted computation.

The root cause, when finally discovered, was almost absurdly simple. The server had been started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in the codebase is strict — only the string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only hidden states instead of the expected 21504-dimensional concatenated states. This caused the fc fusion layer — a critical component that merges the target model's hidden states with the draft model's features — to be silently bypassed. All the trained weights in the draft model were effectively useless, because the input they were designed to process never arrived.

The fix was a single command-line parameter change. After restarting the server with --speculative-algorithm EAGLE3, the hidden states correctly arrived as 21504-dimensional vectors, and the draft model's predictions began being accepted at a meaningful rate. The acceptance length jumped from 1.0 to approximately 2.1 — a 2× improvement that validated the entire EAGLE-3 approach. The draft model was finally doing what it was trained to do.

The Sobering Benchmark: 82.3 tok/s vs. 90 tok/s

With the bug fixed, the team turned to benchmarking. The best EAGLE-3 configuration — using CUDA graphs and 5 draft tokens — achieved 82.3 tokens per second. The non-speculative baseline, running the target model alone, achieved approximately 90 tok/s. Despite the fix, speculative decoding was still 9% slower than the baseline.

This result was deeply counterintuitive. Speculative decoding is supposed to accelerate inference, not decelerate it. The root cause was clear from the acceptance length data: at ~2.1 accepted tokens per draft run, the overhead of running the draft model (which itself requires forward passes through a transformer) outweighed the savings from parallel verification. The EAGLE-3 paper's scaling curves suggested that the primary lever for improving acceptance length was more training data — specifically, data that more closely matched the target model's output distribution.

The team also tested the AQ-MedAI drafter with the correct EAGLE3 flag, which performed worse at 50.5 tok/s. This confirmed that the custom K2.5-trained drafter was better, but still data-limited. The path forward was clear: the training dataset needed to be scaled up dramatically.

The 10× Data Rescue: A Parallel Subagent Campaign

Scaling up the training dataset by 10× became the team's top priority. The existing dataset — approximately 4,800 tokenized Kimi-native samples — was clearly insufficient. The draft model needed to see a much broader distribution of code, reasoning, and conversational data to accurately predict the target model's outputs.

The approach was ambitious: launch ten parallel subagent sessions, each tasked with searching for high-quality datasets in a specific category. The categories included agentic coding datasets, reasoning datasets, general chat datasets, and code instruction datasets. Each subagent was given a detailed brief specifying what to search for, where to search (primarily HuggingFace), and how to report results.

The subagent campaign that followed is documented across five articles in this session. The first subagent message ([msg 0]) tasked a subagent with finding "high-quality coding instruction/conversation datasets on HuggingFace that would be good for training a speculative decoding draft model" [1]. The subagent was given specific search queries and asked to return at least 10 candidate datasets with detailed metadata including dataset name, size, format, language, and license status.

The subagent's response was methodical and multi-round. In the first round ([msg 1]), it launched three parallel web searches targeting general coding instruction datasets, CodeAlpaca/CodeFeedback specifically, and programming conversation datasets [4]. This "opening salvo" revealed the assistant's sophisticated understanding of search strategy — by dispatching independent queries simultaneously, it maximized parallelism and minimized latency.

In the second round ([msg 2]), the assistant pivoted to more targeted searches for large-scale datasets like OpenCodeInstruct, Magicoder, and Glaive Code Assistant [3]. This "iterative search" demonstrated a refinement strategy: broad exploration first, then targeted discovery. The assistant also searched for speculative decoding-specific resources, finding the FastDraft paper (arXiv:2411.11055) which provided research grounding for the dataset selection decisions.

The third round ([msg 3]) continued the refinement, searching for specific missing datasets including TokenBender's code_instructions_122k, Dolphin Coder, and NVIDIA's OpenCodeInstruct [2]. This "third search" showed the assistant performing gap analysis — comparing the datasets it expected to find against what the search results actually returned, and filling the gaps with highly targeted queries.

The final synthesis ([msg 4]) organized 18 candidate datasets into five categories: large-scale code instruction datasets, medium-scale Alpaca-style datasets, conversational/multi-turn code datasets, commit/edit-based datasets, and large composite/OpenCoder-style datasets [5]. The assistant provided a recommended blend — OpenCodeInstruct (subsampled), Code-Feedback (conversational), Magicoder-Evol-Instruct-110K (high-quality instruction), and Dolphin-Coder (system-prompt-aware) — with specific rationale for each choice.

From Dataset Discovery to Inference Pipeline

The dataset search campaign was just the first phase. From the candidate pool, ten datasets were selected and prepared, totaling 88,088 samples. Of these, 4,800 were already tokenized Kimi-native samples that could be used immediately. The remaining 83,288 prompts required inference — they needed to be processed through the target model (Kimi-K2.5) to regenerate responses that matched the target model's token distribution. This is a critical step: for speculative decoding, the draft model must learn to predict the target model's outputs, not just any plausible continuation. On-policy data generation, where the target model itself generates the responses, is essential for alignment.

An inference pipeline was launched on the baseline SGLang server, which was already serving the GLM-5-NVFP4 model. The pipeline achieved approximately 830 tok/s throughput — impressive but still requiring 24-55 hours to process all 83K prompts. A live progress monitor script was created to track the pipeline's status, and the full pipeline plan was documented in train_plan_v4.md.

The Broader Significance

This chunk of work illustrates several profound lessons about speculative decoding in practice. First, flag mismatches can silently break complex systems. The difference between EAGLE and EAGLE3 is a single character, but it caused the entire draft model to be effectively disabled. The hidden state concatenation logic was working correctly — it was just never triggered. This is a reminder that configuration validation is not just a best practice but a critical safety net for complex ML systems.

Second, benchmarking reveals uncomfortable truths. The 82.3 tok/s result, while disappointing, was invaluable data. It confirmed that the draft model was working (accept_len ~2.1) but that the acceptance rate was insufficient to overcome speculation overhead. Without this measurement, the team might have assumed the approach was working and moved on. Instead, the data drove a focused investment in data scaling.

Third, dataset curation for speculative decoding is fundamentally different from dataset curation for general fine-tuning. The draft model needs data that mirrors the target model's output distribution, not just any high-quality code instructions. This is why the inference pipeline — generating responses through Kimi-K2.5 — was necessary. The 83K prompts would become training examples only after being processed through the target model, ensuring the draft model learns the right distribution.

Fourth, parallel subagent campaigns are a powerful paradigm for data discovery. By launching ten agents simultaneously, each focused on a different category, the team covered far more ground than any single researcher could. The subagents' methodical approach — broad search, targeted refinement, gap analysis, and structured synthesis — produced a comprehensive dataset inventory that would have taken a human days or weeks to compile.

Conclusion

The EAGLE-3 bug and the 10× data rescue represent two sides of the same coin: the relentless iteration required to make speculative decoding work in practice. The bug fix was a one-character change that unlocked the entire system. The benchmarking was a reality check that pointed to data as the primary bottleneck. And the subagent campaign was a massive, coordinated effort to scale up training data by an order of magnitude.

As the inference pipeline processes its 83K prompts at 830 tok/s, the team waits to see whether the expanded dataset will push acceptance length high enough to beat the 90 tok/s baseline. The EAGLE-3 paper's scaling curves suggest it should — but as this session has repeatedly demonstrated, theory and practice rarely align perfectly. The next chunk of work will reveal whether the 10× data rescue was enough, or whether further iteration is needed.

For now, the story stands as a testament to the complexity of deploying state-of-the-art ML systems. It is not enough to train a good model. One must also configure it correctly, benchmark it honestly, identify its weaknesses, and feed it the right data. And sometimes, the difference between success and failure is a single flag.## References

[1] "The Dataset Hunter: How a Subagent Was Tasked to Find Code Instruction Data for Speculative Decoding" — Analyzes the initial subagent task message that launched the dataset search campaign, revealing how the parent agent decomposed the research goal and delegated it to a specialized subagent.

[2] "The Third Search: How a Methodical Agent Built a Comprehensive Dataset Inventory for Speculative Decoding" — Examines the third round of web searches, where the assistant performed gap analysis and targeted specific missing datasets like TokenBender's code_instructions_122k and NVIDIA's OpenCodeInstruct.

[3] "The Iterative Search: How an AI Assistant Refines Its Dataset Discovery Strategy" — Covers the second wave of searches, where the assistant pivoted from broad exploration to targeted discovery of large-scale datasets like OpenCodeInstruct and Magicoder.

[4] "The Opening Salvo: Launching Parallel Searches for Code Instruction Datasets" — Documents the first round of parallel web searches, revealing the assistant's sophisticated search strategy and understanding of the tool-calling architecture.

[5] "The Art of Dataset Synthesis: How One AI Assistant Mapped the Landscape of Coding Instruction Data for Speculative Decoding" — Analyzes the final synthesis message, where 18 candidate datasets were organized into five categories with actionable recommendations for draft model training.