The Great Dataset Hunt: Orchestrating Ten Parallel Agents to Scale EAGLE-3 Training Data

In the sprawling, multi-day odyssey to deploy speculative decoding on the Kimi-K2.5 language model, few individual messages represent a more decisive pivot than message 3654. After days of debugging, benchmarking, and incremental tuning—culminating in the discovery that a single misnamed flag (--speculative-algorithm EAGLE instead of EAGLE3) had rendered an entire training run worthless—the project stood at a crossroads. The EAGLE-3 draft model was finally producing correct hidden states, but its acceptance rate hovered around 2.1 tokens per speculation step, yielding 82.3 tok/s against a 90 tok/s non-speculative baseline. The draft model was not merely underperforming; it was actively slowing the system down.

The root cause of this underperformance was clear from the research literature: the EAGLE-3 paper had trained on approximately 530,000 samples, and the production SpecBundle system used over a million. The project's current draft model had been trained on only 10,000 samples. The scaling curve in the paper was unambiguous—more data was the primary lever for improving acceptance rates. When the user issued the instruction to "start 10 agents to look for more matching agentic-coding-related datasets" and "aim for 10x size," the assistant responded with a message that would define the next phase of the project: a massively parallel, ten-agent search across the HuggingFace ecosystem for datasets spanning coding, reasoning, agentic tool use, and general chat.

The Context: A Hard-Earned Fix and a Sobering Benchmark

To understand why message 3654 represents such a critical inflection point, one must appreciate the journey that preceded it. The preceding messages in segment 27 document a painful debugging saga. The EAGLE-3 draft model, trained over many hours on 10,000 samples extracted from Kimi-K2.5, was being deployed with SGLang's speculative decoding infrastructure. The server logs showed that the draft model's predictions were being accepted at a rate barely above random chance—an accept_len of approximately 1.0, meaning the system was essentially rejecting every draft token and falling back to the target model.

The culprit, eventually traced through careful log inspection and code reading, was a single command-line flag. The server had been started with --speculative-algorithm EAGLE rather than --speculative-algorithm EAGLE3. The is_eagle3() check in SGLang's codebase is strict: only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states from layers 2, 30, and 58. With the wrong flag, the draft model received 7168-dimensional final-layer-only hidden states instead of the expected 21504-dimensional concatenated states. The fusion layer (fc) was silently bypassed, and all trained weights were effectively useless.

After correcting the flag and restarting the server, the hidden states arrived at the correct dimensionality, and the draft model's predictions began being accepted at a meaningful rate—approximately 2.1 tokens per speculation step. But the benchmarking results were sobering. The best EAGLE-3 configuration (CUDA graphs enabled, 5 draft tokens, 3 speculation steps) achieved 82.3 tok/s on average, still roughly 9% slower than the 90 tok/s non-speculative baseline. Even with the fix, speculative decoding was a net negative.

The assistant's summary in message 3652 laid out the options plainly: train on more data, switch back to the baseline, try longer training on the existing data, or keep experimenting. The user's response in message 3653 was decisive: "Start 10 agents to look for more matching agentic-coding-related datasets from which we'll generate a kimi k2.5 dataset. We'll aim for 10x size." This directive set the stage for message 3654, which would operationalize that ambition through a carefully orchestrated parallel search.

The Architecture of the Search: Ten Agents, Ten Categories

Message 3654 is structurally unusual. It begins with a brief framing paragraph—"Good plan. We need to find datasets that will produce good agentic/coding/reasoning training data for the EAGLE-3 drafter—diverse, high-quality text that matches the kind of outputs Kimi-K2.5 will generate in production. Let me search broadly in parallel"—and then launches ten simultaneous task tool calls. Each task spawns a subagent session that runs to completion independently, searching HuggingFace for datasets in a specific category.

This architectural choice is significant. The assistant could have searched sequentially, one category at a time, but that would have taken ten times as long. By launching all ten agents in parallel, the assistant maximizes throughput, leveraging the fact that each search is independent and the results can be synthesized afterward. The task tool in the opencode framework spawns a full subagent conversation that blocks the parent session until completion, but multiple task calls in the same round run their subagents concurrently. The parent session waits for all of them to finish before proceeding to the next round.

The ten search categories reveal a sophisticated understanding of what makes good training data for a speculative decoding draft model:

1. Coding instruction datasets — The first agent searches for high-quality coding instruction and conversation datasets. This category is essential because Kimi-K2.5 is being deployed in agentic coding scenarios, and the draft model needs to learn the token distribution of code-heavy responses. The agent searches for datasets like nvidia/OpenCodeInstruct and ise-uiuc/Magicoder-Evol-Instruct-110K, which contain structured code generation tasks.

2. Agentic/tool-use datasets — The second agent targets agentic coding and tool-use datasets, where an AI agent uses tools, writes code, debugs, and interacts with environments. This is the most directly relevant category for the project's use case. The agent finds datasets like nebius/SWE-agent-trajectories (80,000 rows of full SWE-agent trajectories) and SWE-Factory/DeepSWE-Agent-Kimi-K2-Trajectories-2.8K (trajectories generated by Kimi-K2 itself, providing an exact token distribution match).

3. Reasoning/math datasets — Kimi-K2.5 is a reasoning model that generates <think> blocks and long chain-of-thought reasoning traces. The third agent searches for reasoning, math, and chain-of-thought datasets to help the draft model learn these distinctive reasoning patterns. It finds datasets like open-thoughts/OpenThoughts-114k and open-r1/Mixture-of-Thoughts.

4. General chat/instruction datasets — The fourth agent searches for large general-purpose chat and instruction-following datasets. These provide diverse conversational patterns that help the draft model generalize beyond coding-specific scenarios. The agent finds datasets like anon8231489123/ShareGPT_Vicuna_unfiltered and HuggingFaceH4/ultrachat_200k.

5. EAGLE-3/SpecForge training data — The fifth agent performs a meta-search: it looks for what datasets the EAGLE-3 paper and SpecForge (SGLang's official EAGLE trainer) actually used for training. This is a critical piece of research because it grounds the project in established best practices. The agent discovers that the EAGLE-3 paper used ShareGPT (68K samples) and UltraChat-200K, while the production SpecBundle system uses mlabonne/open-perfectblend (1.4M samples) and significantly outperforms the paper's original results.

6. Code review/debugging datasets — The sixth agent searches for code review, debugging, and software engineering datasets. These are particularly relevant for agentic coding use cases where the model needs to understand code quality, identify bugs, and suggest fixes.

7. Long-form writing/analysis datasets — The seventh agent searches for long-form writing, analysis, and explanation datasets. These provide diverse token patterns that help the draft model handle extended generation tasks, which are common in production use.

8. DeepSeek/Kimi specific datasets — The eighth agent searches for datasets specifically created for or derived from DeepSeek models, Kimi models, or similar MoE (Mixture of Experts) reasoning models. These have token distributions closest to what Kimi-K2.5 produces, making them particularly valuable for training a draft model that needs to predict the target model's outputs.

9. Multi-language coding datasets — The ninth agent searches for multi-programming-language coding datasets. These help the draft model handle different code syntax patterns across Python, JavaScript, Rust, C++, and other languages, ensuring it doesn't overfit to a single language's patterns.

10. Large synthetic instruction sets — The tenth agent searches for very large (100K+ samples) synthetic instruction datasets. These provide bulk volume for training and include datasets like BAAI/Infinity-Instruct (7.45M samples) and Magpie-derived datasets.## The Thinking Behind the Categories: Assumptions and Design Decisions

The selection of these ten categories reveals several important assumptions made by the assistant. First and foremost, the assistant assumes that diverse training data is better than homogeneous data for speculative decoding draft models. This is a non-trivial assumption. One could argue that a draft model should be trained exclusively on data that matches the exact production distribution—in this case, agentic coding trajectories. The assistant instead chooses a broad mix spanning coding, reasoning, general chat, and long-form writing, implicitly assuming that diversity helps the draft model generalize and handle edge cases.

Second, the assistant assumes that response regeneration through the target model is essential. The search is explicitly for prompts and instructions, not for complete conversations. The plan is to take the prompts from these datasets and run them through Kimi-K2.5 to generate responses, ensuring the training data matches the target model's actual token distribution. This assumption is grounded in the EAGLE-3 paper's methodology and the SpecBundle system's approach, both of which regenerate responses rather than using original dataset responses. The reasoning is sound: a draft model trained on GPT-4's token distribution will not effectively predict Kimi-K2.5's outputs, even if the prompts are identical.

Third, the assistant assumes that the EAGLE-3 paper's scaling curve applies to this specific use case. The paper demonstrates that acceptance rate improves with more training data, following a clear scaling law. The assistant bets that increasing the dataset from 10K to ~100K samples will push the acceptance rate from ~2.1 to the break-even point of ~2.5 or higher. This is a testable hypothesis, but it's not guaranteed—the scaling curve might be different for Kimi-K2.5's architecture, the reduced 32K vocabulary, or the specific training hyperparameters being used.

Fourth, the assistant assumes that ten categories provide sufficient coverage without being redundant. There's overlap between categories—for example, code instruction datasets and multi-language coding datasets both contain code, and reasoning datasets and long-form writing datasets both contain extended text. The assistant implicitly assumes that the marginal benefit of covering multiple overlapping categories outweighs the cost of processing redundant data.

The Input Knowledge Required

To fully understand message 3654, one needs substantial background knowledge across several domains. The reader must understand what EAGLE-3 is—a speculative decoding architecture that uses a lightweight draft model to predict multiple tokens ahead, which the target model then verifies in parallel. They must understand the role of hidden states in this architecture: the draft model conditions its predictions on the target model's intermediate layer representations, which is why the concatenation bug (receiving 7168-dim instead of 21504-dim states) was so devastating.

The reader must also understand the HuggingFace ecosystem and the landscape of available datasets. The assistant's search terms reference specific dataset IDs (nvidia/OpenCodeInstruct, nebius/SWE-agent-trajectories, open-r1/Mixture-of-Thoughts) and formats (ShareGPT format with from/value fields, conversational format with messages lists, Parquet format). Without familiarity with these conventions, the significance of the search results would be lost.

Knowledge of the project's infrastructure is also essential. The assistant references the existing SGLang server, the hidden state extraction pipeline, the training script (04_train.py), and the vocabulary mapping files (t2d.pt, d2t.pt). The disk space constraint (1.8TB free on /data) and the subsequent resolution (disk resized to 11TB) are critical context for understanding why the assistant can now pursue a 10× scale-up.

Finally, the reader must understand the EAGLE-3 paper's findings and the SpecBundle system's approach. The assistant's search for "EAGLE-3/SpecForge training data" is not just a data collection exercise—it's a research effort to validate the scaling strategy against established benchmarks. The discovery that SpecBundle uses mlabonne/open-perfectblend (1.4M samples) provides both validation and a potential shortcut.

The Output Knowledge Created

Message 3654 produces several forms of output knowledge. Most immediately, it produces ten comprehensive dataset reports covering 15+ candidate datasets per category, each with size estimates, format descriptions, content assessments, and license information. These reports represent a significant knowledge base for anyone planning to train a speculative decoding draft model on agentic coding data.

The message also produces a validated dataset strategy. The ten agents collectively confirm that there are sufficient high-quality datasets available to construct a ~100K sample training set with the desired agentic-coding-heavy composition. This is not trivial—the assistant could have discovered that suitable datasets don't exist at sufficient scale, forcing a strategy revision.

More subtly, the message produces negative knowledge: the confirmation that certain dataset categories (code review, multi-language coding) are less useful than others for this specific task. The code review agent returns an empty result, suggesting that dedicated code review datasets are either not available on HuggingFace or not suitable for this purpose. This negative finding is valuable because it prevents wasted effort on fruitless searches.

The message also creates architectural knowledge about how to structure parallel dataset research. The ten-category taxonomy, the search terms used, and the evaluation criteria (size, format, content, license, relevance) form a reusable methodology that could be applied to future data collection efforts.

The Thinking Process: Strategy and Trade-offs

The reasoning visible in message 3654 reveals a sophisticated strategic mind at work. The assistant's opening statement—"We need to find datasets that will produce good agentic/coding/reasoning training data for the EAGLE-3 drafter—diverse, high-quality text that matches the kind of outputs Kimi-K2.5 will generate in production"—encapsulates three key criteria: diversity, quality, and distribution matching. These criteria are in tension with each other. Diversity pulls toward including many different types of data, while distribution matching pulls toward data that closely resembles Kimi-K2.5's outputs. The assistant implicitly prioritizes diversity at the category level (covering coding, reasoning, chat, writing) while ensuring distribution matching through the response regeneration step.

The decision to launch ten parallel agents rather than searching sequentially reflects a throughput-maximization mindset. The assistant knows that each search will take time—subagent sessions can run for many rounds as they explore HuggingFace, read dataset cards, and compile results. By launching all ten at once, the assistant effectively parallelizes the most time-consuming part of the process: the research phase. This is only possible because the searches are independent; the results of one search don't inform the others.

The assistant also demonstrates awareness of the project's constraints. The disk space issue (1.8TB free vs. ~9.2TB needed for 100K samples) is acknowledged in the subsequent synthesis (message 3655), where the assistant calculates the storage requirements and proposes solutions (streaming, online training, or disk expansion). This constraint awareness prevents the dataset search from producing an infeasible plan.

Mistakes and Incorrect Assumptions

While message 3654 is well-reasoned, several assumptions warrant scrutiny. The most significant is the assumption that 10× more data will produce a meaningful improvement in acceptance rate. The EAGLE-3 paper's scaling curve shows diminishing returns—the improvement from 10K to 100K samples might be smaller than the improvement from 1K to 10K samples. The assistant implicitly assumes a roughly logarithmic scaling relationship, but the actual curve for this specific model, architecture, and data distribution is unknown.

The assistant also assumes that the ten selected categories are complementary rather than redundant. In practice, there may be significant overlap between, say, code instruction datasets and multi-language coding datasets, or between reasoning datasets and long-form writing datasets. This redundancy means the effective diversity of the 100K sample set might be lower than the nominal diversity.

Another potential issue is the assumption that all datasets can be processed through the same pipeline. Different datasets have different formats, different license restrictions, and different quality characteristics. The assistant's plan to "extract prompts/instructions only" and regenerate responses through Kimi-K2.5 glosses over the significant engineering effort required to handle each dataset's idiosyncrasies. Some datasets might have malformed entries, missing fields, or encoding issues that require special handling.

The assistant also assumes that the existing vocabulary mapping (32K reduced vocab) will work for all datasets. The 32K vocabulary was created from the original 10K sample set and covers 98.3% of tokens. With new data from diverse sources, the coverage might drop, meaning some tokens in the new training data would be unmappable. This could introduce noise into the training process.

Finally, the assistant assumes that the training infrastructure can scale linearly with data volume. Training on 10K samples took approximately 1-2 hours. Training on 100K samples with 3-5 epochs might take 30-100 hours, depending on batch size and hardware utilization. The assistant's estimate of ~12 hours for training might be optimistic, especially if the training script wasn't designed for datasets of this size.

The Broader Significance: A Pivot Point in the Project

Message 3654 represents a decisive pivot in the project's trajectory. Before this message, the focus was on debugging the existing pipeline—fixing the flag, enabling CUDA graphs, tuning speculation parameters. After this message, the focus shifts to data scaling—finding, preparing, and processing a 10× larger training set. This pivot acknowledges that the current draft model is fundamentally data-limited and that no amount of hyperparameter tuning will overcome this limitation.

The message also demonstrates a key principle of machine learning engineering: when your model is underperforming, look at the data first. The assistant could have continued tweaking speculation parameters, trying different tree structures, or adjusting learning rates. Instead, it correctly identified that the primary bottleneck was training data volume and addressed it at the root.

The parallel search architecture itself is a model for how to approach data collection at scale. Rather than manually searching for datasets one at a time, the assistant leverages the task tool to spawn independent agents that explore different categories simultaneously. This approach could be generalized to any data collection task where the search space is large and the categories are independent.

Conclusion

Message 3654 is a masterclass in strategic pivoting and parallel research orchestration. Coming at a critical juncture—immediately after fixing a bug that had rendered the entire EAGLE-3 deployment non-functional, and facing the sobering reality that even with the fix, speculative decoding was slower than the baseline—the assistant makes a decisive bet on data scaling. The ten-agent parallel search covers the full spectrum of relevant data types, from agentic coding trajectories to general chat conversations, and produces a comprehensive map of the available dataset landscape.

The assumptions underlying this search are reasonable but not guaranteed: that more data will improve acceptance rates, that diversity helps generalization, that response regeneration is essential, and that the existing infrastructure can scale to 10× the data volume. The output knowledge—ten detailed dataset reports, a validated strategy, and a roadmap for the next phase—provides the foundation for the project's continued evolution.

In the broader narrative of the opencode session, message 3654 marks the transition from debugging to scaling. The hidden state concatenation bug is fixed, the benchmark numbers are known, and the path forward is clear: find more data, generate more responses, train a better draft model. The ten agents dispatched in this message will return with the raw materials for that effort, and the project will move from a frustrating debugging phase into a productive data generation and training phase. Whether the bet on data scaling pays off—whether 100K samples will push the acceptance rate past the break-even point—remains to be seen, but the strategy is sound, the execution is thorough, and the foundation is laid.