The Two Fronts of ML Engineering: Debugging EAGLE-3 and Scaling Training Data in Parallel

Introduction

Modern machine learning engineering often demands working on multiple fronts simultaneously. One team might be debugging a subtle inference bug while another scales up the training pipeline — and both efforts must converge for the project to succeed. This chunk of an opencode coding session captures exactly that dynamic: a single session where the same engineer (with AI assistance) resolved a critical speculative decoding bug, benchmarked the fix, and simultaneously orchestrated a massive 10× scale-up of the training dataset through parallel subagent searches and an inference pipeline.

The chunk spans two parallel tracks of work. The first track is the resolution of an EAGLE-3 hidden state concatenation bug — a deceptively simple flag mismatch that rendered the entire draft model useless. The second track is a coordinated data acquisition campaign: ten parallel subagents scoured HuggingFace for agentic coding, reasoning, and chat datasets, an inference pipeline was launched to regenerate 83,000 prompts through the target model, and a live progress monitor was created to track the multi-day generation process. This article synthesizes both tracks, showing how they connect and what they reveal about the state of ML engineering workflows in 2025.

The EAGLE-3 Bug: A Single Flag, Devastating Consequences

The centerpiece of this chunk's debugging effort was a bug in the EAGLE-3 speculative decoding setup. EAGLE-3 is a state-of-the-art speculative decoding framework that uses a lightweight draft model to predict multiple tokens ahead, which are then verified by the target model in parallel. When working correctly, this can dramatically accelerate inference — but only if the draft model receives the right hidden states from the target model.

The root cause of the bug was traced to a single command-line flag mismatch. The server had been started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. The is_eagle3() check in the codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states from specific layers (layers 2, 30, and 58). With the incorrect EAGLE flag, the target model silently fell back to a default behavior: it passed only the final-layer hidden states (7168 dimensions) to the draft model, instead of the expected concatenated states (21504 dimensions).

This mismatch had cascading effects. The draft model's architecture was designed to receive 21504-dimensional input — three sets of 7168-dimensional states concatenated together. When it received only 7168 dimensions, the fc fusion layer (which fuses the concatenated states) was silently bypassed. All the trained weights in the draft model became useless because the input shape didn't match what the model expected. The draft model was essentially operating on corrupted input, unable to produce meaningful predictions.

The fix was trivial once the root cause was identified: restart the server with the correct flag. After the restart, hidden states correctly arrived as 21504-dimensional vectors, and the draft model's predictions began to be accepted by the target model's verification step. The acceptance length (accept_len) jumped from approximately 1.0 (meaning almost no draft tokens were accepted) to approximately 2.1 (meaning on average, two draft tokens were accepted per speculation step).

Benchmarking the Fix: Progress, But Not Enough

With the bug fixed, the next step was to benchmark the corrected EAGLE-3 configuration and compare it against the non-speculative baseline. The results were instructive — and sobering.

The best EAGLE-3 configuration achieved 82.3 tokens per second when using CUDA graphs and 5 draft tokens. This was an improvement over the buggy configuration, but it still fell short of the 90 tok/s non-speculative baseline. In other words, even with the bug fixed, speculative decoding was making inference slower rather than faster — a roughly 9% performance penalty.

The root cause was the acceptance length. The EAGLE-3 paper's analysis shows that speculative decoding only provides speedups when the acceptance length is sufficiently high to overcome the overhead of running the draft model. With an accept_len of approximately 2.1, the overhead of generating and verifying draft tokens exceeded the savings from parallel verification. The draft model simply wasn't accurate enough to predict tokens that the target model would accept.

The team also tested the AQ-MedAI drafter with the correct EAGLE3 flag, and it performed even worse at 50.5 tok/s. This confirmed that the custom K2.5-trained drafter was the better option, but it was still data-limited. The EAGLE-3 paper's scaling curve pointed to a clear remedy: more training data. The draft model needed to be trained on a larger and more diverse dataset to improve its prediction accuracy and increase the acceptance length.

This diagnosis set the stage for the second major track of work in this chunk: scaling up the training dataset by 10×.

The Data Scaling Campaign: Ten Parallel Subagents

With the diagnosis clear — more data is the primary lever — the team launched a coordinated campaign to expand the training dataset. The goal was ambitious: scale the dataset by a factor of 10, from approximately 8,800 samples to 88,000 samples.

The approach was to dispatch ten parallel subagents, each tasked with searching for specific types of datasets on HuggingFace. The subagents covered a range of categories relevant to agentic coding:

1. Code review datasets — containing pull requests, code suggestions, and human feedback
2. Bug fixing / debugging datasets — containing buggy code and corresponding fixes
3. Code explanation datasets — containing code snippets with natural language explanations
4. Commit message / diff datasets — containing git diffs and commit messages
5. Pull request datasets — containing structured PR data
6. Agentic coding datasets — containing multi-step coding agent trajectories
7. Reasoning datasets — containing chain-of-thought and logical reasoning examples
8. General chat datasets — containing conversational interactions Three of these subagent sessions are documented in detail in the accompanying articles for this chunk. The first article, "The Dataset Hunter" [1], examines the task assignment message that launched one of these subagents. The second article, "The Data Hunt" [2], analyzes the subagent's parallel search execution — how it issued five simultaneous web searches to cover different dataset categories. The third article, "The Empty Message" [3], explores the subagent's silent final message and what it reveals about the system architecture. The parallel subagent approach was strategically sound. Rather than having a single agent sequentially search for datasets across categories, the team decomposed the problem into independent sub-tasks and executed them concurrently. This minimized wall-clock time and allowed each subagent to focus deeply on its assigned category. The trade-off was coordination complexity — the parent session had to wait for all ten subagents to complete before aggregating results, and any single subagent failure (like the empty message analyzed in [3]) could leave gaps in coverage.

Dataset Selection and Preparation

After the subagents completed their searches, the team selected ten datasets for inclusion in the expanded training corpus. These ten datasets totaled 88,088 samples, broken down into two categories:

• 4,800 tokenized Kimi-native samples — these were already in the correct format and could be directly used for training.
• 83,288 prompts needing inference — these were raw prompts that needed to be processed through the target model (Kimi-K2.5) to generate responses that matched the target model's token distribution. The decision to regenerate responses through the target model is a critical detail. In speculative decoding, the draft model needs to learn the target model's token distribution — it needs to predict what tokens the target model would generate. Using raw prompts from external datasets without regenerating responses would introduce a distribution mismatch: the draft model would learn to predict tokens from the original dataset's response distribution, not from the target model's distribution. By running inference on all 83,288 prompts through the baseline SGLang server, the team ensured that the training data reflected the actual target model's behavior.

The Inference Pipeline: Processing 83K Prompts

Generating responses for 83,288 prompts is a significant computational undertaking. The team launched an inference pipeline on the baseline SGLang server, which was running the non-speculative configuration (the 90 tok/s baseline). The pipeline achieved a throughput of approximately 830 tokens per second — impressive throughput enabled by the 8-GPU setup and SGLang's optimized serving infrastructure.

At 830 tok/s, the pipeline was expected to complete the full generation run in approximately 24 to 55 hours. The wide range reflects uncertainty about the average response length per prompt — longer responses require more tokens and thus more time. The team created a live progress monitor script to track the pipeline's status in real time, providing visibility into completion percentage, tokens generated, and estimated time remaining.

The full pipeline plan was documented in a file called train_plan_v4.md, which served as the authoritative reference for the data generation and training workflow. This documentation included:

• The list of selected datasets and their sources
• The inference pipeline configuration and parameters
• The expected throughput and completion timeline
• The training configuration for the draft model after data generation
• The evaluation plan to measure acceptance length improvements

Connecting the Tracks: How Bug Fixing and Data Scaling Converge

The two tracks of work in this chunk — debugging EAGLE-3 and scaling training data — are not independent. They are deeply connected by a causal chain:

1. The EAGLE-3 bug was fixed, enabling correct speculative decoding.
2. Benchmarking revealed that even with correct decoding, the draft model's acceptance length was too low.
3. The EAGLE-3 paper's analysis indicated that more training data is the primary lever for improving acceptance length.
4. This motivated the 10× data scaling campaign.
5. The inference pipeline generates responses that match the target model's distribution.
6. The expanded dataset will be used to retrain the draft model.
7. A better-trained draft model should achieve higher acceptance length.
8. Higher acceptance length will make speculative decoding faster than the baseline. This chain illustrates a fundamental principle of ML engineering: diagnosis drives data strategy. The team didn't decide to scale data arbitrarily — they did so because empirical benchmarking pointed to data quantity as the bottleneck. The debugging effort (Track 1) provided the diagnosis; the data scaling effort (Track 2) provided the remedy.

Lessons for ML Engineering Workflows

Several lessons emerge from this chunk that are broadly applicable to ML engineering:

1. Flag mismatches are devastating and hard to catch. The EAGLE vs EAGLE3 bug is a cautionary tale about configuration management. A single character difference in a command-line flag caused the entire draft model to operate on corrupted input. This argues for stronger validation — the server should have logged a warning or error when the flag didn't match the expected values, or the is_eagle3() check should have been more permissive.

2. Benchmark before optimizing. The team didn't assume the fix would solve the problem — they benchmarked and found that the acceptance length was still insufficient. This empirical approach prevented wasted effort on optimizing the wrong thing.

3. Data quality requires distribution matching. The decision to regenerate responses through the target model (rather than using raw external dataset responses) shows sophisticated understanding of the training dynamics. In speculative decoding, the draft model must learn the target model's distribution, not the distribution of arbitrary external datasets.

4. Parallel subagent decomposition is effective but fragile. The ten parallel subagents enabled rapid dataset discovery, but the empty message artifact [3] shows that parallel execution can produce silent failures. Robust systems need monitoring and verification that each subagent completed its deliverable.

5. Documentation keeps the pipeline coherent. The train_plan_v4.md file served as a central reference point, ensuring that the data selection, inference pipeline, and training configuration were all aligned. In a multi-track effort like this, documentation is essential for maintaining coherence.

Conclusion

This chunk captures a pivotal moment in an ML engineering project: the transition from debugging to data scaling. The EAGLE-3 hidden state concatenation bug was identified and fixed, revealing that the real bottleneck was training data quantity. In response, the team launched a coordinated campaign to scale the dataset by 10×, using parallel subagents for dataset discovery, an inference pipeline for response generation, and comprehensive documentation to keep everything aligned.

The work is not complete — the inference pipeline is expected to run for 24-55 hours, and the draft model retraining will follow. But the foundation has been laid. The bug is fixed, the data pipeline is running, and the path to improved speculative decoding performance is clear. This chunk demonstrates that effective ML engineering requires both deep technical debugging skills and strategic data operations — and that these two skill sets must work in concert for a project to succeed.