The EAGLE-3 Hidden State Bug and the 10× Dataset Scaling Campaign

Introduction

This article synthesizes the work captured across multiple message articles from this chunk. The dataset discovery process is documented in detail in [1] (the subagent's initial mission), [4] (the parallel search strategy), [5] (the information foraging loop), [2] (the third round of targeted searches), and [3] (the final curated report). Together, these articles paint a complete picture of how an AI assistant navigated the complex landscape of HuggingFace datasets to support a critical ML deployment.

In the high-stakes world of speculative decoding deployment, the difference between a working system and a broken one can be as small as a single flag. This is the story of how an opencode session uncovered and resolved a critical EAGLE-3 hidden state concatenation bug—a bug that had silently rendered an entire draft model's trained weights useless—and then pivoted to a massive parallel data acquisition campaign to address the root cause: insufficient training data. The chunk covers a dramatic arc from debugging to benchmarking to dataset scaling, showcasing the iterative, multi-threaded nature of modern ML engineering.

The Bug: A Single Flag Mismatch

The centerpiece of this chunk is the resolution of a bug that had been blocking all draft model deployment for the GLM-5-NVFP4 speculative decoding pipeline. The root cause was deceptively simple: the SGLang server had been started with --speculative-algorithm EAGLE instead of EAGLE3. This one-character difference in a command-line flag had cascading consequences throughout the entire system.

The is_eagle3() check in the SGLang codebase is strict—it performs an exact string match. Only the literal string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58] of the GLM-5 architecture. With the incorrect EAGLE flag, the draft model received 7168-dimensional final-layer-only hidden states instead of the expected 21504-dimensional concatenated states. This meant the draft model's fc fusion layer—the very layer that was supposed to process the rich concatenated representation—was silently bypassed. All the trained weights that had been carefully optimized for the EAGLE-3 architecture were effectively useless, because the input they were designed to process never arrived in the right form.

The fix was as simple as correcting the flag, but the debugging process to identify this issue was anything but trivial. The symptom—poor draft model performance—could have been caused by any number of factors: bad training data, incorrect model architecture, learning rate issues, or a bug in the inference engine. The team had to trace through the entire pipeline, examining hidden state dimensions, verifying layer outputs, and ultimately discovering that the flag mismatch was the culprit.

Benchmarking the Fix: Progress, But Not Enough

After restarting the server with the correct EAGLE3 flag, the hidden states correctly arrived as 21504-dimensional vectors, and the draft model's predictions began being accepted at a meaningful rate. The acceptance length (accept_len) rose from a baseline of 1.0 (meaning the target model was rejecting virtually all draft predictions) to approximately 2.1. This was a significant improvement—the draft model was now contributing useful speculative tokens.

However, benchmarking revealed a sobering reality. The best EAGLE-3 configuration—using CUDA graphs and 5 draft tokens—achieved 82.3 tokens per second. Against the non-speculative baseline of 90 tok/s, this was still roughly 9% slower. In other words, the speculative decoding system, which was supposed to accelerate inference, was actually making it slower.

The root cause was clear: an acceptance length of ~2.1 was insufficient to overcome the overhead of speculation. Each round of speculative decoding involves running the draft model forward (which has its own computational cost) and then verifying the draft tokens with the target model. If the draft model's predictions are accepted too infrequently, the overhead of running the draft model outweighs the savings from parallel verification. The EAGLE-3 paper's scaling curve suggested that the primary lever for improving acceptance rates was more training data—the draft model simply hadn't seen enough examples to make high-quality predictions across the diverse range of inputs it was encountering.

The team also tested the AQ-MedAI drafter with the correct EAGLE3 flag. It performed slightly worse at 50.5 tok/s, confirming that the custom K2.5-trained drafter was the better option, but still data-limited.

The Data Scaling Campaign: Ten Parallel Agents

With the diagnosis clear—the draft model needed more training data—the team pivoted to a massive parallel data acquisition campaign. The goal was to scale up the training dataset by a factor of 10×, from roughly 8,800 samples to 88,000 samples. This was not a simple matter of finding more data; it required a coordinated search across multiple data domains, followed by a complex inference pipeline to regenerate responses in a format compatible with the target model.

Ten parallel subagent sessions were dispatched simultaneously, each tasked with searching for datasets in a specific domain. The domains included agentic coding datasets, reasoning datasets, general chat datasets, and—most critically for the draft model's multi-language code capabilities—multi-programming-language coding datasets. The subagents searched HuggingFace systematically, using engineered web search queries to discover datasets across five categories: multi-language code instruction datasets, competitive programming datasets, code translation datasets, polyglot code datasets, and LeetCode/coding challenge datasets.

The search process itself was a masterclass in information foraging. The subagents began with broad parallel searches covering all five categories, then progressively refined their queries in subsequent rounds. When initial searches returned incomplete results—missing critical datasets like The Stack v2 (the massive 600+ language code corpus) and MultiPL-E (the multi-language benchmark)—the subagents launched targeted follow-up searches to fill the gaps. By the third and fourth rounds, the raw search results had been synthesized into a structured, curated report of ten datasets with consistent metadata: HuggingFace ID, approximate size, languages covered, format, and availability.

The Selected Datasets

The ten datasets ultimately selected represented a diverse portfolio of code-related training data:

  1. The Stack v2 (bigcode/the-stack-v2): The premier multi-language code pretraining corpus with ~4 trillion tokens across 600+ programming languages. Gated access requiring agreement with SoftwareHeritage terms.
  2. CodeSearchNet (sentence-transformers/codesearchnet): A classic code search dataset with 1M-10M rows covering Go, Java, JavaScript, Python, Ruby, and PHP, in Parquet format.
  3. McEval-Instruct (mceval/mceval-instruct): A 40-language instruction-tuning dataset derived from the McEval benchmark, openly available.
  4. MultiPL-E (multi-pl-e/multi-pl-e): A multi-language benchmark extending HumanEval to 18+ programming languages, suitable for both evaluation and fine-tuning.
  5. Codeforces (open-r1/codeforces): A competitive programming dataset with problem-solution pairs from Codeforces contests, covering multiple languages.
  6. Verifiable Coding Problems (PrimeIntellect/verifiable-coding-problems): 144K verifiable coding problems with test cases, useful for reasoning and verification training.
  7. LeetCode Problem Solutions (kaysss/leetcode-problem-solutions): A dataset of LeetCode problem solutions with multiple language implementations.
  8. HumanEval-X (zai-org/humaneval-x): A multi-language benchmark with 820 samples across 5 languages (Python, Java, JavaScript, Go, Rust), providing parallel implementations.
  9. HumanEval-XL (humaneval-xl/humaneval-xl): An extended version of HumanEval covering more languages with cross-lingual generation tasks.
  10. SWE-PolyBench (swe-polybench/swe-polybench): A real-world software engineering benchmark with multi-language tasks.

The Inference Pipeline

Of the 88,088 total samples collected, 4,800 were already in the Kimi-native tokenized format and could be used directly. The remaining 83,288 samples were prompts that needed to be processed through an inference pipeline to regenerate responses matching the target model's token distribution. This was a critical step: the draft model needed to learn from the target model's own output distribution, not from arbitrary responses collected from the internet.

The inference pipeline was launched on the baseline SGLang server, which was serving the GLM-5-NVFP4 model at approximately 830 tok/s throughput. Processing all 83,288 prompts through this pipeline was estimated to take between 24 and 55 hours—a significant time investment, but necessary for generating high-quality training data.

To monitor the pipeline's progress, a live progress monitor script was created. This script provided real-time visibility into the inference job's status, allowing the team to track throughput, estimate completion time, and detect any failures early. The full pipeline plan was documented in train_plan_v4.md, which served as the authoritative reference for the data generation workflow.

The Parallel Search Architecture

One of the most impressive aspects of this chunk is the parallel search architecture used for dataset discovery. The ten subagent sessions were dispatched simultaneously, each operating independently but contributing to a shared goal. Within each subagent session, the assistant used the opencode tool architecture to issue multiple web search queries in parallel within a single round, minimizing round-trip latency.

This parallel approach was not just about speed—it was about coverage. Different search queries returned different types of results. Broad queries like "multi language code dataset huggingface" returned general-purpose datasets, while specific queries like "bigcode/the-stack-v2 huggingface multilingual code dataset" targeted known canonical datasets. By running both types of queries in parallel, the subagents could simultaneously discover new datasets and verify the existence of known ones.

The search strategy followed a deliberate funnel: broad exploratory searches in the first round, targeted follow-ups on promising leads in the second round, gap-filling searches in the third round, and finally a synthesis phase where all results were compiled into a structured report. This funnel approach ensured comprehensive coverage while avoiding the trap of getting lost in an endless search loop.

From Infrastructure to Application

This chunk also represents a significant shift in the overall project trajectory. Earlier segments of the opencode session had focused on infrastructure setup: installing NVIDIA drivers, configuring CUDA Toolkit 13.1, creating Python virtual environments, and wrestling with flash-attn compilation issues. The environment had been stabilized with a compatible stack of PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1, and the machine had been upgraded to 8 RTX PRO 6000 Blackwell GPUs.

With the infrastructure solid, the focus shifted to application deployment and optimization. The EAGLE-3 bug fix and benchmarking represented the first major application-level challenge. The data scaling campaign represented the response to that challenge. This arc—from infrastructure to debugging to data acquisition—is characteristic of real-world ML engineering, where the boundaries between system administration, software engineering, and data science blur continuously.

Lessons and Implications

Several lessons emerge from this chunk. First, the importance of exact flag matching in complex ML systems cannot be overstated. A single character difference (EAGLE vs EAGLE3) silently broke an entire speculative decoding pipeline, and the bug was invisible at the system level—no errors, no crashes, just subtly wrong behavior. This underscores the need for rigorous configuration validation in production ML systems.

Second, the relationship between training data quantity and speculative decoding performance is not linear. The EAGLE-3 paper's scaling curve shows that acceptance rates improve with more data, but the improvement rate depends on the quality and diversity of that data. The team's decision to scale up by 10× was based on this understanding, but the actual improvement from the new data remained to be measured.

Third, parallel data acquisition is a powerful pattern for ML engineering. By dispatching multiple subagents to search for datasets simultaneously, the team could cover more ground in less time than any sequential approach. The structured reporting format—consistent metadata fields, categorization by use case, and actionable recommendations—enabled rapid decision-making about which datasets to include.

Finally, the chunk demonstrates that ML deployment is never truly "done." Even after fixing the EAGLE-3 bug, benchmarking the fix, and launching the data pipeline, the team was still waiting for results. The inference pipeline would take 24-55 hours to complete, and only then could the new training data be used to improve the draft model. This iterative cycle—debug, benchmark, gather data, train, benchmark again—is the rhythm of ML engineering.

Conclusion

The chunk spanning segment 27 of this opencode session captures a pivotal moment in the deployment of the GLM-5-NVFP4 model with EAGLE-3 speculative decoding. From the discovery and fix of a critical hidden state concatenation bug, through the sobering benchmarking results that revealed the draft model was still underperforming, to the massive parallel data acquisition campaign that aimed to address the root cause—this chunk tells a story of systematic problem-solving at scale. The ten parallel subagent sessions, the curated selection of ten multi-language coding datasets, and the launch of a 24-55 hour inference pipeline all represent coordinated responses to a single diagnosis: the draft model needed more diverse training data. Whether this intervention will be sufficient remains to be seen, but the methodology—parallel search, structured curation, and systematic pipeline execution—provides a template for similar challenges in ML deployment.## References

[1] "The Dataset Scout: A Subagent's Mission to Find Multi-Language Coding Data" — Analyzes the initial subagent dispatch message, the strategic context behind the dataset search, and the assumptions encoded in the search specification.

[2] "Drilling Deeper: The Third Round of Multi-Language Coding Dataset Discovery" — Examines the third wave of the search expedition, focusing on McEval and CodeSearchNet as targeted gap-filling queries.

[3] "The Dataset Hunter: How an AI Assistant Curated Multi-Language Coding Resources for Draft Model Training" — Covers the final consolidated report of ten datasets, the reasoning behind selection criteria, and the summary table as a decision support tool.

[4] "The Art of the Parallel Search: Uncovering Multi-Language Coding Datasets for Draft Model Training" — Analyzes the first round of parallel web searches, the architecture of the message, and the trade-offs between speed and thoroughness.

[5] "The Information Foraging Loop: A Deep Dive into Dataset Discovery on HuggingFace" — Examines the second round of targeted searches, the information foraging theory behind the assistant's decisions, and the iterative refinement of search strategy.