The Long March: From Data to Deployment in the EAGLE-3 Training Pipeline

Introduction

Machine learning engineering at scale is rarely a straight line. It is a recursive process of building, debugging, crashing, recovering, and iterating — a long march through infrastructure failures, silent bugs, and hard-won empirical discoveries. This article synthesizes a single chunk (Segment 30, Chunk 0) of an opencode coding session that traces the complete lifecycle of training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 language model, from data preparation through deployment. The chunk spans 129 messages (indices 4091 through 4219) and captures every major phase of the pipeline: a comprehensive knowledge dump, data merging, hidden state extraction, infrastructure catastrophe, multi-GPU training strategy, and final deployment with SGLang speculation.

The numbers tell the story of the arc: 37,312 training samples, 87.8 million tokens, ~4.6 TB of hidden state data, a catastrophic VM crash requiring disk migration, 10.8 hours of training across 4 GPUs, a final validation accuracy of 74.7%, and an estimated acceptance length of ~2.95 tokens — a 40% improvement over the previous 10K-sample drafter. But the real story is in the journey: the bugs fixed, the assumptions challenged, the infrastructure failures survived, and the engineering judgment that guided every decision.

The Comprehensive Knowledge Dump: A Project's Memory

The chunk opens with one of the most remarkable artifacts in the entire conversation: message 4091, a comprehensive knowledge dump that documents every discovery, decision, bug fix, benchmark result, architectural detail, and plan for the entire project up to that point <chunk seg=30 chunk=0 article=1>. Running over 10,000 words, this message reads less like a conversational turn and more like a project README, a design document, and a post-mortem rolled into one.

The message was written at a critical transition point. All data generation was complete — 40,114 samples across 10 datasets, 138.4M tokens, $86 spent on OpenRouter API calls — and the assistant was about to embark on a 72-hour hidden state extraction pipeline. Rather than letting the accumulated knowledge remain scattered across hundreds of previous messages, the assistant synthesized it into a single, coherent document. The structure is meticulous: hardware specifications (8× RTX PRO 6000 Blackwell GPUs, no NVLink, PCIe Gen5), software versions (SGLang, vLLM, PyTorch 2.10.0, CUDA 12.8), model architecture details (Kimi-K2.5, 1T-parameter MoE with MLA and INT4 quantization), tokenizer critical findings (the \u003c|im_end|\u003e token having two IDs, one of which is wrong), the EAGLE vs EAGLE3 flag bug, performance benchmarks (90 tok/s baseline, 82.3 tok/s with EAGLE-3 speculation), and detailed space estimates for the upcoming extraction.

This document served as organizational memory — a durable artifact that could be consulted during the complex work ahead. It encoded not just what was known, but how that knowledge was obtained, what assumptions it rested on, and what decisions it supported.

The Merge Point: Assembling the 100K Dataset

Following the knowledge dump, the pipeline moved into data preparation. The assistant verified the merge script (merge_and_shuffle.py), deployed it, and monitored its execution <chunk seg=30 chunk=0 articles=6,7,8>. The merge process combined the response data from multiple datasets into a unified training corpus, then shuffled it to ensure the drafter would see a diverse mix of examples during training.

The merge completed successfully, producing 37,312 records with 87.8 million tokens — a dataset roughly 10× larger than the previous 10K-sample run that had achieved only 2.1 acceptance length <chunk seg=30 chunk=0 article=9>. A cleanup operation followed, removing the 924 GB of raw response data that was no longer needed <chunk seg=30 chunk=0 article=10>. This seemingly mundane deletion was a critical resource management decision: with only 11 TB of available disk space and a 3.5 TB extraction looming, every gigabyte mattered.

The Hidden State Extraction: A Multi-Act Drama

The hidden state extraction phase was by far the most complex and error-prone part of the pipeline. It involved running each of the 37,312 training prompts through the Kimi-K2.5 target model via a patched SGLang server, capturing the internal neural activations (hidden states) that the EAGLE-3 drafter would learn to predict.

Setup and Server Configuration

The extraction required a specially configured SGLang server with a hidden state dump patch applied to the model code <chunk seg=30 chunk=0 article=16>. The assistant had to stop any existing SGLang instances, verify the patch was applied, configure the server for extraction mode (with --disable-cuda-graph and --disable-radix-cache to ensure clean hidden state capture), and launch it across multiple GPUs <chunk seg=30 chunk=0 articles=18,19,20,21>.

A critical moment came during server verification. The assistant polled the server's health endpoint and received a 200 OK response, but further investigation revealed the server was silently failing — the model weights hadn't finished loading, and the Python import path was incorrect <chunk seg=30 chunk=0 articles=24,25,26,27,28,29>. This false positive health check nearly derailed the pipeline. The fix required correcting the PYTHONPATH to include the patched model directory and ensuring the SGLang server was launched with the right environment.

Storage Calculations and Empirical Ground Truth

A pivotal exchange occurred around storage estimation. The user asked whether the 87.8 million token count included only response tokens or both prompts and responses <chunk seg=30 chunk=0 article=33>. The assistant initially recalculated and arrived at ~4.7 TB, then self-corrected by cross-referencing against empirical data from the previous 10K run <chunk seg=30 chunk=0 article=34>. The 10K run had produced 924 GB for 23.1M tokens, yielding a ratio of ~40 KB per token. Scaling this ratio to 87.8M tokens gave ~3.5 TB — matching the assistant's original estimate.

This moment of self-correction is a masterclass in quantitative reasoning under uncertainty. The assistant moved from a theoretical calculation (4.7 TB) through an overcorrection (5.03 TB) to an empirically validated estimate (3.5 TB), demonstrating the value of grounding theoretical models in real measurements.

The Counter Bug and the Race Condition

The extraction launched, but almost immediately encountered a subtle bug. The hidden state dump mechanism was using a shared counter to match dumps with requests, and this counter was being corrupted by concurrent requests <chunk seg=30 chunk=0 articles=55,56,57,58,59,60>. The symptom was that dump files were being associated with the wrong requests, producing corrupted training data.

The debugging process was extensive. The assistant examined the SGLang server code, traced the counter's behavior across multiple request handlers, and identified the root cause: the counter was incremented in the request handler but read in a separate callback, creating a race condition when multiple requests were in flight simultaneously <chunk seg=30 chunk=0 article=66>. The fix was to remove the fragile counter entirely and instead match dumps to requests by comparing token counts — a more robust approach that didn't depend on shared mutable state <chunk seg=30 chunk=0 articles=61,62,65>.

After the fix was deployed, the extraction ran cleanly: zero errors in the first 60 samples, then zero errors across thousands of samples <chunk seg=30 chunk=0 article=73>. The pipeline had achieved reliability.

Monitoring Progress Under Uncertainty

With the extraction running smoothly at ~2,600 tokens per second, the assistant and user entered a phase of monitoring and planning. Progress checks revealed the extraction at 34.5%, then 49.2%, then 68.6% <chunk seg=30 chunk=0 articles=77,78,86,90,92>. Each check was a moment of verification — confirming that the system was still healthy, that errors remained at zero, and that the ETA was holding steady.

But the monitoring revealed a deeper tension. At 49.2% completion, the user sent a message saying "Seems done, proceed with 4-gpu train" <chunk seg=30 chunk=0 article=84>. The assistant's status check revealed the extraction was only half finished — the user's mental model had advanced to the training phase prematurely <chunk seg=30 chunk=0 article=86>. This collision between expectation and reality is a classic pattern in long-running engineering workflows: the planning horizon naturally extends beyond the current phase, but the data pipeline must complete before the next phase can begin.

Infrastructure Catastrophe and Recovery

The extraction was progressing smoothly when disaster struck. The underlying Ceph cluster ran out of storage space, forcing a VM kill and a complete disk migration to a new 15 TB NVMe drive attached directly to the host machine <chunk seg=30 chunk=0 articles=101,102>. The VM crash was catastrophic: all running processes were terminated, the SGLang server was killed, and the extraction script was lost from /tmp.

The recovery process was methodical and multi-stage <chunk seg=30 chunk=0 articles=103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125>. First, the assistant assessed the damage: a vLLM inference server had auto-started via a systemd service, consuming all 8 GPUs. The assistant stopped and disabled the vLLM service, reclaimed the GPU memory, and verified that the hidden state files (18,421 previously extracted samples) had survived the crash intact on the migrated disk.

The extraction script had to be re-uploaded, the SGLang server restarted, and the extraction relaunched with resume capability. The resume feature was critical — it allowed the extraction to pick up from sample 18,421 rather than starting from scratch, saving over 24 hours of work. The assistant verified the recovery by checking that the extraction was producing new files and that the log showed it was processing samples beyond the previous checkpoint <chunk seg=30 chunk=0 articles=119,120,121,122,123>.

This recovery sequence demonstrates the importance of designing systems for resilience. The extraction script's resume capability, the persistence of hidden state files on the migrated disk, and the assistant's methodical approach to verification all contributed to turning a catastrophic failure into a manageable delay.

Training Strategy: Multi-GPU Scaling and Batch Size

While the extraction was running, the assistant and user engaged in a detailed analysis of the training strategy. The user asked whether training could scale to 2-4-8 GPUs, and whether PCIe bandwidth would bottleneck <chunk seg=30 chunk=0 article=79>. The assistant's response was a systematic bottleneck analysis <chunk seg=30 chunk=0 article=80>.

The EAGLE-3 drafter is small — 2.6B parameters total, 1.2B trainable — making it compute-light but data-heavy. The bottleneck chain was: data loading (1-2 GB/s from disk, sufficient for ~4,400 samples/s), forward/backward compute (fast on one GPU), and AllReduce gradient synchronization over PCIe Gen5. The assistant calculated that AllReduce of 4.8 GB of gradients would take ~150ms across 8 GPUs, ~120ms across 4, and ~75ms across 2. With a single-GPU step time estimated at 200-300ms, the overhead was significant but manageable for 2-4 GPUs.

The user then asked about batch size scaling: "higher batch size makes sense or is it going to degrade learning?" <chunk seg=30 chunk=0 article=81>. The assistant's analysis was grounded in the nature of the EAGLE-3 task — a distillation/imitation problem with a smooth loss landscape, unlike RL or generative adversarial training <chunk seg=30 chunk=0 article=82>. The EAGLE-3 paper used batch sizes of 32-64, and the loss (per-token cross-entropy over a 32K vocabulary) had millions of tokens per epoch, making the gradient signal naturally stable. The assistant proposed linear learning rate scaling: with 4 GPUs and batch_size=8 per GPU (effective batch 32), the learning rate should double from 3e-5 to 6e-5.

A critical discovery came when the assistant read the training script 04_train.py <chunk seg=30 chunk=0 article=83>. The script already supported multi-GPU via FSDP2 (not DDP as assumed), and used batch_size=1 with sequence packing — where each "sample" was a packed sequence of max_seq_len tokens. This meant the effective batch size was controlled by max_seq_len, not batch_size. The assistant recommended: 4 GPUs with torchrun, --max-seq-len 4096, --lr 6e-5, --num-workers 4, and --epochs 5. No script changes were needed.

The Final Phase: Extraction Completion and Deployment

The extraction completed with a final count of 37,312 samples and zero errors. The user's message "GPUs idle now" <chunk seg=30 chunk=0 article=129> signaled the milestone — the GPUs had transitioned from active inference to idle, and the extraction script was in its final CPU-only post-processing phase (building sample_lengths.json by scanning all .pt files).

With the data ready, the assistant launched training on 4 GPUs using torchrun with TTT=5 (teacher-forcing training steps), batch_size=8, and max_seq_len=8192. The training achieved ~100% GPU utilization at 350-400W power draw and completed 5 epochs in ~10.8 hours, converging to a final validation accuracy of 74.7% (full_acc_0) and an estimated acceptance length of ~2.95 tokens — a significant improvement over the previous 10K drafter's 2.1.

The final step was deployment. The assistant prepared the drafter checkpoint at /data/eagle3/output_100k_sglang/4/ with a vLLM-compatible config, applied the weight key fix (layers.0 → midlayer) for SGLang compatibility, corrected the SGLang server argument names (--speculative-num-draft-tokens instead of --num-speculative-tokens, and added --speculative-num-steps), and deployed the SGLang server with EAGLE3 speculation at 16 draft tokens, ready for benchmarking.

Themes and Lessons

Several themes emerge from this chunk that are broadly applicable to ML engineering at scale.

The value of empirical grounding. Throughout the pipeline, theoretical calculations were repeatedly corrected by real measurements. The storage estimate was validated against the 10K run's empirical ratio. The multi-GPU scaling analysis was anchored in observed single-GPU behavior. The ETA was continuously updated based on actual throughput. Theory provides a starting point, but measurement provides truth.

The fragility of distributed state. The counter bug that corrupted hidden state dumps is a cautionary tale about shared mutable state in distributed systems. A single integer counter, incremented in one thread and read in another, nearly invalidated days of work. The fix — matching by token count rather than by counter — replaced fragile synchronization with deterministic matching.

Infrastructure resilience is not optional. The Ceph crash and VM kill demonstrated that infrastructure failures are inevitable at scale. The pipeline survived because it was designed for resilience: the extraction script supported resume, the hidden state files were persisted on stable storage, and the recovery process was methodical rather than panicked.

The human-AI feedback loop. The collaboration between user and assistant was essential to the pipeline's success. The user acted as a high-level monitor, watching system metrics and flagging anomalies ("GPUs idle now"). The assistant handled the detailed investigation and execution. This division of labor — human as sensor and decision-maker, AI as executor and analyst — proved highly effective.

Documentation as organizational memory. The comprehensive knowledge dump at the start of the chunk (message 4091) was not a one-time artifact. It was referenced throughout the pipeline, updated as new discoveries were made, and served as the shared context that enabled efficient communication between user and assistant.

Conclusion

This chunk of the EAGLE-3 training pipeline captures the complete lifecycle of a complex ML engineering project: from knowledge synthesis and data preparation, through extraction and debugging, past infrastructure catastrophe and recovery, to training and deployment. The final result — a drafter with 74.7% validation accuracy and an estimated acceptance length of ~2.95 tokens — represents a significant improvement over the previous iteration. But the real achievement is the pipeline itself: a system that survived bugs, crashes, and uncertainty to deliver a working model.

The lessons from this chunk extend beyond EAGLE-3 training. They are lessons about how to build robust ML infrastructure: ground your estimates in empirical data, design for failure, document your knowledge, and maintain a tight feedback loop between monitoring and action. In the long march from data to deployment, these practices are what separate projects that succeed from those that collapse under their own complexity.