The Data Scaling Pivot: How Ten Parallel Agents Forged a 100K-Sample Training Strategy for EAGLE-3

Introduction

In the sprawling, multi-day effort to deploy speculative decoding for the Kimi-K2.5 language model, there comes a moment where the trajectory of the entire project pivots decisively. That moment is message [msg 3655] in this opencode session — a message that is simultaneously a synthesis, a strategy document, a negotiation with physical constraints, and a request for human judgment. It is the point at which the assistant, having just resolved a critical bug that had rendered all prior EAGLE-3 draft model training useless, confronts the deeper truth: the bug was fixed, but the model still wasn't fast enough. The bottleneck had shifted from a software configuration error to a fundamental data scarcity problem.

This article examines that single message in depth. We will explore why it was written, the reasoning that shaped its content, the assumptions baked into its recommendations, and the knowledge it both consumed and produced. The message is a masterclass in how an AI assistant operating at scale can decompose a complex problem, delegate research to parallel subagents, synthesize their findings, and present a coherent plan — all while navigating hard constraints of disk space, compute budget, and model architecture.

The Context That Made This Message Necessary

To understand message [msg 3655], we must first understand what came immediately before it. The session had been engaged in a protracted battle with EAGLE-3, a speculative decoding technique that uses a lightweight "draft" model to predict multiple future tokens, which the target model then verifies in parallel. The promise of EAGLE-3 is throughput acceleration — if the draft model is accurate enough, the speculation overhead is more than compensated by generating multiple tokens per forward pass.

But for days, every attempt to deploy EAGLE-3 had failed. The acceptance rate — the fraction of draft tokens accepted by the target model — hovered near zero. The trained draft model was essentially useless. The root cause, discovered and fixed just before this message ([msg 3651]), was a single flag: the server had been started with --speculative-algorithm EAGLE instead of EAGLE3. This seemingly minor difference was devastating. The is_eagle3() check in SGLang's code is strict — only the exact string EAGLE3 triggers the target model to capture and concatenate intermediate hidden states from layers [2, 30, 58]. With EAGLE, the draft model received 7168-dim final-layer-only states instead of the expected 21504-dim concatenated states. The trained fc fusion layer was silently bypassed, and all the weights from 10K samples of training were wasted.

After fixing the flag, the assistant benchmarked the corrected configuration. The best result was 82.3 tokens per second with CUDA graphs and 5 draft tokens — still 9% slower than the non-speculative baseline of 90 tok/s. The acceptance length of ~2.1 tokens was insufficient. The EAGLE-3 paper's scaling curve suggested that more training data was the primary lever, not architectural changes.

This is the critical juncture. The assistant had just demonstrated that the draft model could work — the hidden states were now correct, the acceptance rate was non-trivial — but it wasn't good enough. The user's response ([msg 3653]) was decisive: "Start 10 agents to look for more matching agentic-coding-related datasets from which we'll generate a kimi k2.5 dataset. We'll aim for 10x size."

Message [msg 3655] is the assistant's response to that directive. It is the synthesis of ten parallel subagent research missions, each tasked with exploring a different category of training data on HuggingFace.## The Architecture of the Ten-Agent Research Sweep

The message begins with a simple line: "Excellent research. Now let me synthesize all 10 agents' findings into a concrete plan." This opening reveals the underlying architecture of the assistant's approach to the problem. Rather than conducting a single sequential search, the assistant dispatched ten parallel task tool calls in the preceding round ([msg 3654]). Each task spawned a subagent — a fully independent AI session that conducted its own multi-turn research conversation, searched HuggingFace, evaluated datasets, and returned a structured report.

This is a fascinating example of hierarchical decomposition. The problem — "find more training data for EAGLE-3" — was too broad for a single agent to handle efficiently. The assistant decomposed it into ten orthogonal dimensions:

Coding instruction datasets (e.g., nvidia/OpenCodeInstruct)
Agentic/tool-use datasets (e.g., nebius/SWE-agent-trajectories)
Reasoning/math datasets (e.g., open-thoughts/OpenThoughts-114k)
General chat/instruction datasets (e.g., ShareGPT_Vicuna_unfiltered)
EAGLE-3/SpecForge training data (researching what the papers actually used)
Code review/debugging datasets
Long-form writing/analysis datasets
DeepSeek/Kimi specific datasets
Multi-language coding datasets
Large synthetic instruction sets (e.g., BAAI/Infinity-Instruct) Each subagent returned a detailed report, visible in the context messages preceding [msg 3655]. The reports varied in length and depth — some were comprehensive multi-page analyses, others (like the code review/debugging search) returned empty results. The assistant's task was to read all ten reports, identify the most promising candidates, and weave them into a coherent strategy. This parallel research pattern is notable for several reasons. First, it demonstrates the assistant's ability to manage concurrency — all ten agents ran simultaneously, and the assistant waited for all of them to complete before proceeding. Second, it shows a sophisticated understanding of dataset taxonomy: the categories were chosen to cover different aspects of the target model's expected usage (agentic coding, reasoning, general chat). Third, it reveals the assistant's awareness of the EAGLE-3 literature — the dedicated agent researching what the EAGLE-3 paper and SpecForge actually used was a meta-research move, ensuring the strategy was grounded in proven approaches rather than guesswork.

Synthesizing the Findings: The Recommended Dataset Mix

The centerpiece of message [msg 3655] is a table proposing a dataset mix targeting approximately 100K samples — a 10× scale-up from the current 10K. The table is carefully constructed, with each dataset accompanied by a rationale explaining why it was chosen.

The proposed mix reveals a deliberate weighting strategy. The assistant prioritized agentic coding datasets above all else, allocating ~22.8K samples to this category (SWE-agent trajectories, DeepSWE-Agent-Kimi-K2-Trajectories, and glaive-function-calling). This reflects the user's stated use case — the model is being deployed in a coding agent context. The inclusion of SWE-Factory/DeepSWE-Agent-Kimi-K2-Trajectories-2.8K is particularly shrewd: these trajectories were generated by Kimi-K2 itself, meaning the token distribution should closely match what the target model produces. This is the closest thing to "free" aligned training data.

The next tier is code instruction datasets (25K samples from OpenCodeInstruct and Magicoder-Evol-Instruct). These provide broad coverage of code generation patterns across multiple programming languages. Then reasoning datasets (25K from Mixture-of-Thoughts and OpenThoughts-114k), which are critical because Kimi-K2.5 is a reasoning model that generates <think> blocks — the draft model needs to learn to predict these structured reasoning traces. Finally, general chat datasets (25K from UltraChat and ShareGPT) provide conversational diversity, following the EAGLE-3 paper's own recipe.

The inclusion of crownelius/KimiK2.5-2000x (2K samples generated by Kimi-K2.5 itself) is a small but strategically important addition — it provides a "calibration" dataset that exactly matches the target model's output distribution.

What's notable is what the assistant chose not to include. The massive synthetic datasets like BAAI/Infinity-Instruct (7.45M samples) and mlabonne/open-perfectblend (1.4M samples) were considered but ultimately not recommended for the primary mix. The assistant explicitly mentions the alternative of using open-perfectblend directly — "This is what SpecBundle uses. It's a curated 1.4M-sample blend already designed for EAGLE training. We could just subsample 100K from it and regenerate responses. Simpler but less agentic-coding focused." This is a deliberate trade-off: the custom mix requires more preparation work but is more targeted to the use case.## The Critical Design Decision: Response Regeneration

Perhaps the most important analytical insight in message [msg 3655] is the assistant's recognition of the need for response regeneration. The assistant writes:

"The EAGLE-3 paper and SpecBundle both regenerate responses through the target model rather than using original dataset responses. This is critical — the draft model needs to predict Kimi-K2.5's token distribution, not GPT-4's or DeepSeek's."

This is a subtle but crucial point. The draft model is trained to predict the hidden states of the target model (Kimi-K2.5). If the training data consists of responses written by GPT-4, DeepSeek, or human annotators, the token distribution will differ from what Kimi-K2.5 would generate. The draft model would learn to predict hidden states for tokens that Kimi-K2.5 would never produce, and conversely would be unfamiliar with the specific token patterns that Kimi-K2.5 favors.

The implication is profound: the datasets being collected are only useful for their prompts, not their responses. The actual responses must be generated by running inference on Kimi-K2.5 itself. This transforms the problem from a simple data collection exercise into a massive inference pipeline — every prompt in the 100K dataset needs to be fed through the target model, and the hidden states from those forward passes need to be captured.

This decision has cascading consequences that the assistant immediately identifies. The hidden state extraction budget becomes the binding constraint.

Navigating Physical Constraints: The Disk Space Analysis

The assistant's analysis of disk space constraints is a masterclass in practical engineering reasoning. The message includes a detailed breakdown:

- Current HS data: 924GB for 10K samples at avg 2103 tok/sample - 100K × same avg = ~9.2TB — way over budget - At max_seq_len=1024: ~4.6TB — still too much - At max_seq_len=512: ~2.3TB — barely fits - Streaming/online training is likely required

This analysis reveals the assistant's deep understanding of the system's physical limitations. It knows the exact size of the current hidden state data (924GB for 10K samples), the average token count per sample (2103), and the available disk space (1.8TB on /data). It then extrapolates linearly to calculate the requirements for 100K samples, exploring the trade-off between sequence length and storage.

The numbers are stark: even at max_seq_len=512 (a 4× reduction from the current average), the storage requirement of ~2.3TB still exceeds the available 1.8TB. The assistant's initial conclusion is that "streaming/online training is likely required" — a reference to SpecForge's online mode, which extracts hidden states and trains simultaneously without storing them to disk.

However, the user's response to the embedded question resolves this constraint: "Disk resized to 11T now." The assistant had anticipated this possibility and built a question into the message to let the user decide. This interactive decision-making — presenting options, explaining trade-offs, and letting the human choose — is a hallmark of effective human-AI collaboration.

The Question Mechanism: Interactive Decision-Making

Message [msg 3655] includes an embedded question block — a structured decision point that the assistant presents to the user. The question has two parts:

Dataset strategy: Custom agentic-heavy mix vs. open-perfectblend subsample vs. pure ShareGPT+UltraChat
Disk space handling: Generate-then-train vs. streaming/online training The user's responses are shown at the bottom of the message: "1. agentic-heavy mix + kimi traces" and "generate -> disk -> train; Disk resized to 11T now". This question mechanism is notable because it demonstrates the assistant's ability to recognize its own knowledge boundaries. It has done extensive research, synthesized findings, and formed a recommendation, but it cannot make the final decision — the user's preferences and infrastructure capabilities (like whether disk can be resized) are external knowledge that must be obtained interactively. The questions are carefully framed. Each option includes a brief description of its trade-offs, and the assistant marks one as "Recommended" to guide the user. This is not a neutral presentation — the assistant has a clear opinion (the custom agentic-heavy mix is preferred) but leaves the final call to the human.## Assumptions Embedded in the Strategy The assistant's recommendations in message [msg 3655] rest on several assumptions, some explicit and some implicit. Understanding these assumptions is critical for evaluating the soundness of the proposed plan. Assumption 1: More data will improve acceptance rate. The assistant explicitly references the EAGLE-3 paper's scaling curve, which shows that draft model quality improves with more training data. However, this assumes that the relationship between data quantity and acceptance rate is monotonic and that the current 10K sample model is on the steep part of the curve. The assistant's own benchmarks show an accept_len of ~2.1 with 10K samples; the paper's curve suggests ~3.0+ would be needed for a speedup. Whether 100K samples is sufficient to bridge this gap is uncertain — the paper used ~530K samples for their best models. Assumption 2: The prompt distribution matters more than the response distribution. By selecting prompts from diverse datasets and regenerating responses through Kimi-K2.5, the assistant assumes that the prompt distribution is the primary determinant of the training data's usefulness. This is reasonable — the draft model needs to learn to predict hidden states for the kinds of inputs it will encounter in production. But it also assumes that the response regeneration process (running Kimi-K2.5 on each prompt) produces sufficiently diverse outputs to train a generalizable draft model. Assumption 3: The 32K reduced vocabulary is not a fatal limitation. The assistant notes that the reduced vocabulary (32K tokens vs. the full model's vocabulary) has 98.3% coverage, meaning ~1.7% of tokens can never be predicted. This is acknowledged as a limitation but not treated as a showstopper. The assumption is that the draft model can still be useful despite this blind spot. Assumption 4: Disk can be resized. The assistant's disk analysis assumes the current 1.8TB free space is a hard constraint, but the embedded question allows the user to override this. The user's response ("Disk resized to 11T now") validates this assumption — the assistant correctly anticipated that infrastructure constraints might be negotiable. Assumption 5: The ten dataset categories are comprehensive and non-overlapping. The assistant's decomposition of the search space into ten categories assumes that these categories cover the relevant dataset landscape without significant redundancy. In practice, there is overlap — for example, open-thoughts/OpenThoughts-114k contains code, math, and reasoning content that overlaps with the code instruction and reasoning categories. The assistant doesn't address how duplicate or overlapping content will be handled during the final dataset construction.

Input Knowledge Required to Understand This Message

Message [msg 3655] is dense with domain-specific knowledge. To fully understand it, a reader would need:

Understanding of speculative decoding and EAGLE-3: Knowledge of how draft models predict hidden states, how acceptance rates work, and why hidden state concatenation from intermediate layers matters. Without this, the distinction between EAGLE and EAGLE3 flags and the 7168-dim vs. 21504-dim hidden state issue would be incomprehensible.
Familiarity with the HuggingFace dataset ecosystem: The message references over a dozen specific dataset identifiers (e.g., nebius/SWE-agent-trajectories, open-r1/Mixture-of-Thoughts, mlabonne/open-perfectblend). Understanding the significance of each requires knowledge of what these datasets contain, their quality, and their provenance.
Knowledge of the EAGLE-3 and SpecForge literature: The assistant references findings from the EAGLE-3 paper and SpecBundle (SGLang's production EAGLE implementation). The reader needs to know that SpecBundle uses open-perfectblend and outperforms the paper's original recipe.
Understanding of the training pipeline: The distinction between using original dataset responses vs. regenerating responses through the target model, the concept of hidden state extraction, and the relationship between sequence length and storage requirements all require familiarity with the EAGLE-3 training process.
Infrastructure awareness: The disk space calculations (924GB for 10K samples, 9.2TB projected for 100K) require understanding how hidden states are stored, the format and size of each sample, and the relationship between token count and storage.
The history of the session: The message builds on the entire prior conversation — the bug fix, the benchmarking results showing 82.3 tok/s vs. 90 tok/s baseline, the acceptance rate measurements, and the user's directive to scale up 10×. Without this context, the message's urgency and direction would seem unmotivated.

Output Knowledge Created by This Message

Message [msg 3655] produces several forms of output knowledge that shape the subsequent trajectory of the project:

A concrete dataset selection: The ten datasets listed in the table become the target set for data collection. This selection constrains all future work — the inference pipeline that follows will process prompts from exactly these datasets.
A decision about response regeneration: The message establishes that responses must be regenerated through Kimi-K2.5, not used from the original datasets. This decision drives the massive inference pipeline that is launched in subsequent messages.
A disk space resolution: The user's response confirms that disk will be resized to 11TB, enabling the generate-to-disk approach rather than streaming training. This removes the primary infrastructure bottleneck.
A prioritized category weighting: The agentic coding focus is explicitly chosen over the simpler open-perfectblend alternative. This weighting reflects the user's use case priorities.
A scaling target: 100K samples is established as the target, representing 10× scale-up from the current 10K. This becomes the organizing metric for all subsequent work.
A documented trade-off analysis: The message serves as a record of why certain decisions were made, which is valuable for future debugging. If the 100K-trained model still underperforms, the team can look back at this message and see the assumptions and constraints that shaped the approach.## The Thinking Process: What the Reasoning Reveals While message [msg 3655] does not contain explicit chain-of-thought reasoning (the assistant's thinking is embedded in the structure and content of the message itself), we can infer the cognitive process that produced it. The assistant faced a complex synthesis problem: ten subagent reports, each containing multiple dataset recommendations with varying levels of detail and quality. The first step was categorization — grouping datasets by type (agentic coding, code instruction, reasoning, general chat) to create a coherent taxonomy. This is visible in the table's category labels. The second step was prioritization. Not all datasets are equally valuable. The assistant ranked them based on several criteria: relevance to the use case (agentic coding > general chat), token distribution alignment (Kimi-generated data > other model's data), dataset quality (curated datasets > raw crawls), and diversity (multi-language > single-language). The ranking is implicit in the sample counts — SWE-agent trajectories get 10K, while KimiK2.5-2000x gets only 2K. The third step was constraint analysis. The assistant calculated the disk space requirements and realized the plan was infeasible under current infrastructure. Rather than abandoning the plan, it explored alternatives (shorter sequence lengths, streaming training) and then escalated to the user via the question mechanism. The fourth step was meta-research validation. The dedicated agent researching EAGLE-3/SpecForge training data provided a crucial insight: SpecBundle uses open-perfectblend and outperforms the paper's original recipe. This gave the assistant confidence that the general approach (curated dataset blend + response regeneration) was sound. Finally, the assistant recognized its own knowledge boundaries. It could not know whether the user could resize the disk or which dataset strategy they preferred. Rather than guessing, it structured these as explicit decision points.

The Broader Significance

Message [msg 3655] represents a pivotal moment in the opencode session. Before this message, the project was in a reactive, debugging mode — fixing bugs, benchmarking configurations, and trying to make a broken pipeline work. After this message, the project shifts into a proactive, scaling mode — building a 100K-sample training pipeline, launching massive inference jobs, and systematically improving the draft model.

The message is also a demonstration of how AI assistants can operate at scale. The ten-agent parallel research sweep is a pattern that could be applied to many domains: literature reviews, competitive analysis, codebase audits, or any problem that benefits from exploring multiple dimensions simultaneously. The assistant's ability to synthesize diverse findings into a coherent plan, identify constraints, and present structured decisions to the user is a template for effective human-AI collaboration.

The disk space analysis is particularly instructive. It shows the assistant reasoning about physical constraints with concrete numbers, exploring trade-offs, and escalating decisions appropriately. This is not generic advice — it is grounded in the specific infrastructure of this session (1.8TB free on /data, 924GB for 10K samples, average 2103 tok/sample). The assistant knows the system it is working on.

Conclusion

Message [msg 3655] is far more than a simple response to a user request. It is a strategic document that synthesizes ten parallel research missions, navigates hard physical constraints, makes explicit design decisions about data composition and training methodology, and presents interactive decision points to the human operator. It demonstrates the assistant's ability to decompose complex problems, manage parallel subagent workflows, reason about infrastructure limitations, and communicate trade-offs clearly.

The message's impact is immediate and lasting. The dataset selections made here will determine the quality of the next draft model. The decision to regenerate responses through Kimi-K2.5 shapes the entire inference pipeline. The resolution of the disk space constraint enables the generate-to-disk approach. And the explicit documentation of assumptions and trade-offs provides a reference point for future debugging.

In the broader narrative of the opencode session, this is the message where the project pivots from debugging to scaling. The bug that had blocked all progress is fixed. The draft model is working, if slowly. The path forward is clear: more data, better data, and a systematic pipeline for generating it. The ten agents have done their research, the strategy is set, and the work of building the 100K-sample training dataset can begin.